Configure Your Own Jellyfish Fryer All-Flash NAS

19

Bonus: Setting System Tunables

Again, this is another area where it should not be this complex, yet here we are. ZFS was primarily designed as hard disk plus ultra exotic (back then) solid-state storage running over (back then) exotic 10GbE or Infiniband. Today, we have inexpensive SSDs, and inexpensive 10GbE networking. FreeNAS was primarily designed for hard drives and 1GbE connectivity. The heritage means that we can get appreciably more performance by setting a handful of tunable values. The below will let you get at least 40GB/s from a system like the Jellyfish Fryer.

To enter them, go to System -> Tunables. There you want to insert the following with Type “sysctl”:

FreeNAS Tunable Example
FreeNAS Tunable Example

You should end up with something like this. We are taking the opportunity to plug 45drives here.

FreeNAS Tunables
FreeNAS Tunables

To make it easier for copying and pasting:

  • kern.ipc.maxsockbuf = 16777216
  • net.inet.ip.intr_queue_maxlen = 2048
  • net.inet.tcp.recvbuf_inc = 524288
  • net.inet.tcp.recvbuf_max = 16777216
  • net.inet.tcp.recvspace = 4194304
  • net.inet.tcp.sendbuf_inc = 32768
  • net.inet.tcp.sendbuf_max = 16777216
  • net.inet.tcp.sendspace = 2097152
  • net.route.netisr_maxqlen = 2048

Being fair, the iXsystems / FreeNAS teams are adding tuning for high-speed networking into the next version. This is long overdue. Just the above usually can get you anywhere from 100-200MB/s better performance on a 10GbE connection.

These need to be either defaulted or there needs to be dropdowns/ checkboxes for things like “Tune for all-flash” or “Tune for 10GbE”. There are a lot of different tunables in FreeNAS. This is a starting configuration we use which works okay. If you want to delve into this world, there is a lot you can do.

Final Words

Or you can just buy a Lumaforge Jellyfish, or a FreeNAS Centurion and have someone else set this up for you. Perhaps the 45drives Stornado team will help set this up in the future. Frankly, buying storage makes a lot of sense. These companies certainly provide value and you want them to make a margin. When economic downturns hit, the 3-6% margin resellers tend to be the ones that often close their doors and you do not want to have “support” from a company that is out of business. With that said, the Jellyfish Fryer project is set to show you how the other half lives.

45drives Stornado
45drives Stornado

If you are a tech geek, that also needs video storage, there is a good chance you probably already use FreeNAS. If so, the incremental parts of this guide will likely take you under 15 minutes to do. Indeed, something that many creators in wealthy countries forget is that not everyone has access to $25,000 or $45,000 for storage. Not everyone can afford expensive hardware and premiums. Some people and organizations that can afford this storage are just plain thrifty.

For those tech geeks, thrifty spenders, and those who do not have access to the latest technology, the above guide will help you get to bare minimum costs for a solution like this. The guide, including writing, took only about four hours to write at around 3,500 words and with installations, and screenshots. It may seem like a lot to read, or watch, but in reality, if you save $5,000 in four hours, that is a fairly good hourly rate.

There is a case to be made that learning storage for a few hours has an awesome payback. Once you do it, when you need more storage in 2-3 years, you can replicate the process in an hour or less. Many of the frustrations we hear from STH readers around FreeNAS the team is working on. Wizards and more opinionated setup can cut this time in half. The iXsystems team actually has a script in the lab that does most of the FreeNAS steps for you so this could be something we see become a few clicks in the not-too-distant future.

I just wanted to close this piece by saying thank you again to Linus and the entire LTT team. This project was a lot of fun and the LTT team was extremely kind during a day of filming.

19 COMMENTS

  1. >Is FreeNAS any different from a zfs on linux setup?

    A little more appliancy. Personally ZFSOL has legs. Have a look at what Jim Salter is doing. Next level ZFS.

  2. Tony, they use the same OpenZFS base. At an OS-level, FreeNAS is built on FreeBSD which is different than Linux (some good, some bad.) FreeNAS also has a Web GUI and an active team developing it.

    With that said, I personally use a lot of ZoL. More recently our lab has been using a lot of Ceph storage which I am contemplating using at home as well.

  3. Speaking of Ceph, i would really like it if iX implemented Ceph into FreeNAS/TrueNAS, that way FreeNAS could act as a controller for Ceph and at the same time act as a gateway for Samba.

    45drives uses Ceph alot, couldn’t they develop a Ceph plugin for FreeNAS, or work together with iX to implement it.

    Currently there is very few options to scale a NAS other than UP, would be nice to have an option with horisontal scaling too. Now, Ceph isn’t really ment for NAS:es but hey, atleast it would be an option.

    Will probably never happen but there is a huge gap in the market between these single enclosure NAS systems and for example a multi-deployed Isilon “cluster”

  4. It’s worth pointing out that the Stornado can also come with FreeNAS pre-installed. It also comes with support and a warranty packaged into the price.

  5. Interesting video showing the two sides (DIY vs paying an OEM to do it all for you). The SSD storage in the DIY solution sure seems like a no brainer decision though, especially compared to HDDs in the jelly. But MKBHD gave good argument for those who choose the jelly.

    Also was nice seeing Patrick on LTT, quite a different audience for STH.

  6. If only SAMBA supported SMBDIRECT…

    Incredible the performance of SMBdirect with 2 direct connection over 40GB Mellanox CX3.
    Benchmarching a ramdisk exposed as SMB share, nearly 6GBYTES performance on E5 2640 v0 server, IOPS thru the roof…

  7. Jellyfish fryer…
    somehow, this brought to my memory that Video Toaster thing for the good old Commodore Amiga

  8. LOVE this series! I am in the middle of planning a DIY all-flash server for video editing, VFX, and color-grading myself, so am totally the target audience for this. This series couldn’t have come at a better time for me, personally. A few questions I hope to get some input on:
    1. I was planning to go with 8 x Micron 9300 Max 12.8TB U.2 NVMe SSDs attached to 2 x HighPoint SSD7120 X16 PCIe HBAs. These are capable of 3.5GBps individually, so massive performance. I would probably do Z+1 and have a good backup strategy. Any comments on this as a good idea / pitfalls / tweaks?
    2. Would there be a good Windows Server-based solution to doing this sort of build, or is there no equivalent/good file system + software RAID solution there compared to ZFS? It’s just that I’m much more familiar with Windows as a platform and opens up some additional services / features to me on the host (i.e. run some transcodes on it, etc.).
    3. Is there any way to make a Thunderbolt port behave as a direct-connect network so as to negate the need for a Thunderbolt 10G Ethernet adapter if wanting to connect a laptop?
    4. Anyone know a good way to determine what level of CPU is needed to run these high-performance NAS systems? I mean, is the CPU a bottleneck so it pays to have dual high-spec Xeons, or when do we hit the point of diminishing returns? I want max performance, and haven’t figured out a good way to avoid over/under speccing the CPU(s).

  9. How can the freenas centurion be cheaper then the stornado?
    It has 20Tb less and 6 cores less but the freenas has more ram, more 10gbe ports and a warrenty.

  10. This might be the most commented STH article of all time on here. Looks like a bunch of people just found out what we’ve known – Patrick’s the Server Jesus these days. It was great seeing him on LTT. I hope they’re planning to do more.

    G the Centurion uses a standard off the shelf server so that’s cheaper. They’re probably getting a better deal because all the storage is included while Linus and Patrick used retail pricing. You can get lots of 3.84tb drives under $500 now. They didn’t cover Centurion noise but if it’s saving you $15k then that buys a lot of sound deadening. Maybe they’ve tuned fans like on the mini xl+?

  11. This sentence from the article seems…odd. (I read it five times, and am still unsure of the original intent, even if a single word was perhaps skipped, etc…)

    “Many of the frustrations we hear from STH readers around FreeNAS the team is working on. ”

    Seems like it needs an extra word or phrase to clarify meaning…

  12. I’ve been hearing that using USB flash drives has fallen from favor, and that most flash drives will fail within a month or three…

    Yet, the installer still seems to condone/recommend it…or does it?

    Is there an official verdict from the FreeNAS/TrueNAS folks?

  13. [quote]This sentence from the article seems…odd. (I read it five times, and am still unsure of the original intent, even if a single word was perhaps skipped, etc…)

    “Many of the frustrations we hear from STH readers around FreeNAS the team is working on. ”

    Seems like it needs an extra word or phrase to clarify meaning…[/quote]

    Insert comma after FreeNAS. ‘Tis a poor sentence though.

    Good to see Patrick on LTT 🙂

  14. I think I’ve gotten more anxiety than fun from watching Linus play like a child with networking. He really needs to read STH more often and learn a thing or two.

    Glad you could represent Patrick! Also glad you could do a ‘back-to-roots’ STH post like this. Love me some tutorials and thrifty data management advice.

LEAVE A REPLY

Please enter your comment!
Please enter your name here