Now Reading
Install Illumian on ESXi 5.0 for All-in-One ZFS Installation

Install Illumian on ESXi 5.0 for All-in-One ZFS Installation

by Patrick KennedyJuly 25, 2012

One topic I get a lot of questions about is creating an All-in-One ESXi 5.0 machine with ZFS storage based on Illumian, Nexenta, and Solaris 11 Express all alongside guest operating systems. For those not familiar with the concept, the idea is simple, utilize what is arguably the best open storage subsystem very similar to what Oracle and NetApp use in their high-end storage inside a virtual machine. Then, export that storage to other virtual machines running the services you want such as a Windows Home Server 2011 or Linux based LAMP setup. Since the interconnects between virtual machines are fast, assuming you have sufficient CPU to drive the traffic, this method both utilizes underutilized hardware in the ESXi machine while at the same time saving costs on high-performance networking gear. With that in mind, I am going to embark on doing a series of articles walking through the basic setup of the storage subsystem. Today, we are going to get Illumian, an Illumos distribution that serves as the basis for the Nexenta family of products. Of note, Illumian itself is not overly user-friendly and one does not get the slick Nexenta interface, plugins or support, but it does provide a good foundation for those looking to play with an All-in-One ZFS test machine.

Test Configuration

Again it is time to fire up my ESXi 5.0 test bed. If you want to read about the inexpensive, yet very powerful build, you can learn about the AMD Opteron 6128 ESXi server here.

  1. CPUs: 2x AMD Opteron 6128
  2. Motherboard: Supermicro H8DG6-F
  3. Memory: 8x 4GB Kingston unbuffered ECC 1333MHz DIMMs
  4. SSD: OCZ Vertex 3 120GB
  5. Power Supply: Corsair AX650 650w 80 Plus Gold
  6. Chassis: Norco RPC-450B
  7. Cooling: 2x Dynatron A1 G34 Coolers
  8. VMware ESXi 5.0
This server has several LSI cards in it to help test various configurations at this point. Most likely we will be using ESXi VMDirectpath to pass-through controllers to the ZFS virtual machines as that is a leading practice in this setup.

Getting Illumian

The first step in this process is downloading the Illumian ISO ready for the virtual machine. Go to and there is a link to download the ISO. Once you have the ISO downloaded, the next step is to get it onto the ESXi 5.0 machine. I use the following method just because I like keeping active ISOs there in the event I need them. First, select the data store on the server for the ISO.

ESXi 5 Select Data Store

ESXi 5 Select Data Store

Then upload the ISO to the VMware ESXi 5.0 datastore. This is a very simple process and the benefit is that if you ever have to re-install the server, everything will be in one place.

ESXi 5 Upload ISO

ESXi 5 Upload ISO

Prepare the Virtual Machine

The next step is to create a virtual machine for the VM. I already have a guide, albeit with ESXi 4.1 on how to create a Virtual Machine for NexentaStore. Follow that guide and you should be in good shape but some minor differences may present themselves such as now having the ability to use a version 8 ESXi virtual machine.

VMware ESXi Add Memory to VM - VM Properties Memory

VMware ESXi Add Memory to VM – VM Properties Memory

The next step is to mount the Illumian ISO by clicking the wrench and optical disk icon then navigating to the ISO you stored previously on the VMware ESXi server.

ESXi 5 Mount ISO Local Datastore

ESXi 5 Mount ISO Local Datastore

Once this is done, you can choose the Power On command pictured above. One thing I would highly suggest is to first hit Edit Settings and go to Options and change the Boot Options to Boot to BIOS. This will set the virtual machine to boot to BIOS so you can do minor tweaks before everything is setup. One thing that I generally do is move down the PXE network boot to the lowest priority since this is going to be my storage virtual machine.

ESXi 5 BIOS Boot Options

ESXi 5 BIOS Boot Options

Once this is done, hit F10 in the console and Save and Exit. The virtual machine will reboot, and you will see Illumian’s installer. A tip here is to skip installing the 32-bit version.

ESXi 5 Illumian Installer Menu

ESXi 5 Illumian Installer Menu

Now that this is done, you are ready to install Illumian. After I completed the installation, I decided that the installation file was going to be huge. As I am re-launching the YouTube channel in the next week or so, I decided to just make that a quick video. If you are anxious to create an All-in-One ZFS server with Illumian and VMware ESXi 5.0, then good news, the installation is very simple.

About The Author
Patrick Kennedy
Patrick has been running ServeTheHome since 2009 and covers a wide variety of home and small business IT topics. For his day job, Patrick is a management consultant focused in the technology industry and has worked with numerous large hardware and storage vendors in the Silicon Valley. The goal of STH is simply to help users find some information about basic server building blocks. If you have any helpful information please feel free to post on the forums.
  • Max Paolozzi
    July 25, 2012 at 10:46 am

    Does the VMXNET3 VNIC work on Illumian?

  • July 25, 2012 at 1:18 pm

    It is not working out of the box, just like NexentaCE 3.x.x

  • Steve
    July 26, 2012 at 3:13 am

    @Patrick: the link to the AMD Opteron build is wrong. It’s actually linking to a Xeon build.

  • MiniKnight
    July 26, 2012 at 9:35 am

    Good performance. I think OpenIndiana supports VMXNET3. New NexentaCE 4 might. Patrick heard from those guys on that? Or an estimated release date?

  • July 30, 2012 at 12:18 pm

    NexentaStor 4 is due out by the end of the year.

  • TS
    June 12, 2013 at 4:33 am

    Hey, Patrick:
    Nice website, and great articles.

    Just to let you know from experience, of all the illumos variants, OmniOS is the best. Stable, optionally supported. Unlike OpenIndiana and illumian.

    OmniOS+Nappit rocks. No 18TB limits, no limits on licensing. Only thing missing is RSF-1 clustering. But I doubt home users will need RSF-1.

  • cip
    January 15, 2014 at 7:43 am

    Does anyone have experience with All-in-One installations using Nas4Free? There is a new version based on FreeBSD 9.2 with out of the box support for VMXNET3 published just yesterday!

Leave a Response