In 1785, the Scottish Poet Robert Burns, in his poem “To A Mouse,” wrote:
“The best laid schemes of mice and men
Go often awry,”

That sums up how my week went.

Last week, I had the pleasure of taking a week off to de-stress and recuperate before I move into a new role. My plan was to make the best use of that time in my home lab making updates, diving deep into VMware View, and testing out some new items that caught my eye at VMworld. Things obviously didn’t go quite the way I had hoped.

But in the process, I’ve put my home lab on a more solid footing and even added new capabilities that I wouldn’t have otherwise had.

The problems

The problems I faced in my home lab centered entirely around storage. As part of my lab upgrade process, I planned to upgrade Nexenta to the latest version and add additional solid state drives to the HP xW4400 . I was able to add the drives and do the upgrade, but the ZFS pool that I added them to showed up as being in a degraded state with an error on the SSD drives. After doing a little digging, I realized that this issue was because that machine wasn’t set to use ACHI in the BIOS.

Changing the BIOS on the storage box would require a reinstall, and this is where my problems truely began. When I reinstall Nexenta using the latest version, it started to experience boot loops. It would boot to where the kernel would load, display an error, and then quickly reboot. Reverting the BIOS settings did not resolve the issue, and I was able to boot FreeNAS, a FreeBSD-based appliance OS with ZFS support, from a USB drive.

I figured that there was some odd hardware driver issue that was causing the error. Nexenta is an OS derived from OpenSolaris and it’s descendant Illumos, and while it has fairly broad hardware support, it isn’t as all-encompassing as Windows or Linux, so I decided to try a different machine.

The old lab server in my environment was a Dell PowerEdge T310. This server had been set aside to host a few services like my domain controller after I purchased a used PowerEdge T110 Mk II. Since I wasn’t using the full capabilites of this box, I figured I would move Nexenta and my ZFS pools over to it.

I was surprised when I started experiencing the same issues on this server as I did on my HP workstation. The server would start to boot, display an error, and then quickly reboot after installing the latest version. To make matters worse, the Nexenta Community forums were down for maintenance, which made troubleshooting this issue harder.

After doing a little Googling, I was able to find some information on how to halt the system on an error instead of rebooting. And I was surprised to see that the problem that I was having was caused by my USB keyboard. I switched to a different USB keyboard – same issue. Booted without a keyboard and an attached USB hard drive caused the same error.

Something in the Nexenta USB driver stack was causing the issue, and without the community support forums, I wouldn’t be able to dig deeper into it. I’m not a Unix and Linux expert. That area is way outside of my comfort zone, and I wanted to get it running sooner rather than later.

After digging through my stack of unlabeled CDs, I was able to find a Nexenta disk for an earlier version of 3.1. I was able to install this without any issue and then upgrade to the latest version. This time, everything booted without issue.

Improvements to my lab

Despite the issues with the storage that took most of my week to resolve, I was able to make several significant improvements to my home lab. Those improvements were:

1. Adding additional gigbabit ports with a 48-port gigabit switch
2. Improving the overall performance of my storage server by moving it to from a dual-core machine with 2GB of RAM to a quad-core machine with 8GB of RAM
3. Added two SSDs to my ESXi box
4. Added an additional SSD to my ZFS pool
5. Added Fibre Channel between my ESXi box and my Nexenta box

I start a new job this week, and any posts that I would have scheduled are backlogged right now due to the issues I faced. But I have been digging into a new VMware Fling called Web Commander which allows PowerShell scripts to be run from a web interface, and I hope to start having some things ready by the end of the week.