Home Lab Update

Back in October of 2014, I wrote a post about the (then) current state of my home lab.  My lab has grown a lot since then, and I’ve started building a strategy around my lab to cover technologies that I wanted to learn and the capabilities I would need to accomplish those learning goals.

I’ve also had some rather spectacular failures in the last year.  Some of these failures have been actual lab failures that have impacted the rest of the home network.  Others have been buying failures – equipment that appeared to meet my needs and was extremely cheap but ended up having extra costs that made it unsuitable in the long run.

Home Lab 1.0

I’ve never really had a strategy when it comes to my home lab.  Purchasing new hardware happened when I either outgrew something and needed capacity or to replace broken equipment.  If I could repurpose it, an older device would be “promoted” from running an actual workload to providing storage or some other dedicated service.

But this became unsustainable when I switched over to a consulting role.  There were too many things I needed, or wanted, to learn and try out that would require additional capacity.  My lab also had a mishmash of equipment, and I wanted to standardize on specific models.  This has two benefits – I can easily ensure that I have a standard set of capabilities across all components of the lab and it simplifies both upgrades and management.

The other challenge I wanted to address as I developed a strategy was separating out the “home network” from the lab.  While there would still be some overlap, such as wireless and Internet access,  it was possible to take down my entire network when I had issues in my home lab.  This actually happened on one occassion last August when the vDS in my lab corrupted itself and brought everything down.

The key technologies that I wanted to focus on with my lab are:

  1. End-User Computing:  I already use my lab for the VMware Horizon Suite.  I want to expand my VDI knowledge to include Citrix. I also want to spend time on persona management and application layering technologies like Liquidware Labs, Norskale, and Unidesk.
  2. Automation: I want to extend my skillset to include automation.  Although I have vRO deployed in my lab, I have never touched things like vRealize Automation and Puppet.  I also want to spend more time on PowerShell DSC and integrating it into vRO/vRA.  Another area I want to dive back into is automating Horizon environments – I haven’t really touched this subject since 2013.
  3. Containers: I want to learn more about Docker and the technologies surrounding it including Kubernetes, Swarm, and other technology in this stack.  This is the future of IT.
  4. Nutanix: Nutanix has a community edition that provides their hyperconverged storage technology along with the Acropolis Hypervisor.  I want to have a single-node Nutanix CE cluster up and running so I can dive deeper into their APIs and experiment with their upcoming Citrix integration.  At some point, I will probably expand that cluster to three node and use it for a home “private cloud” that my kids can deploy Minecraft servers into.

There are also a couple of key capabilities that I want in my lab.  These are:

  1. Remote Power Management:  This is the most important factor when it comes to my compute nodes.  I don’t want to have them running 24×7.  But at the same time, I don’t want to have to call up my wife and have her turn things on when I’m traveling.  Servers that I buy need to have some sort of remote management capability that does not require an external IP KVM or Wake-on-LAN.   The compute nodes I use need to have some sort of integrated remote management, preferably one with an API.
  2. Redundancy: I’m trying to avoid single-points of failure whenever possible.  Since much of my equipment is off-lease or used, I want to make sure that a single failure doesn’t take everything down.  I don’t have redundancy on all components – my storage, for instance, is a single Synology device due to budget constraints.  Network and Compute, however, are redundant.  Future lab roadmaps will address storage redundancy through hyperconverged offerings like ScaleIO and Nutanix CE.
  3. Flexibility: My lab needs to be able to shift between a number of different technologies.  I need to be able to jump from EUC to Cloud to containers without having to tear things down and rebuild them.  While my lab is virtualized, I will need to have the capacity to build and maintain these environments in a powered-off state.
  4. Segregation: A failure in the lab should not impact key home network services such as wireless and Internet access.

What’s in Home Lab 1.0

The components of my lab are:

Compute

Aside from one exception, I’ve standardized my compute tier on Dell 11th Generation servers.  I went with these particular servers because there are a number of off-lease boxes on eBay, and you can usually find a good deals on servers that come with large amounts of RAM.  RAM prices are also fairly low, and other components like iDRACs are readily available.

I have also standardized on the following components in each server:

  • iDRAC Enterprise for Remote Management
  • Broadcom 5709 Dual-Port Gigabit Ethernet
  • vSphere 6 Update 1 with the Host Client and Synology NFS Plugin installed

I have three vSphere clusters in my lab.  These clusters are:

  • Management Cluster
  • Workload Cluster
  • vGPU Cluster

The Management cluster consists of two PowerEdge R310s.  These servers have a single Xeon X3430 processor and 24GB of RAM.  This cluster is not built yet because I’ve had some trouble locating compatible RAM – the fairly common 2Rx4 DIMMs do not work with this server.  I think I’ve found some 2Rx8 or 4Rx8 DIMMs that should work.  The management cluster uses standard switches, and each host has a standard switch for Storage and a standard switch for all other traffic.

The Workload cluster consists of two PowerEdge R710s.  These servers have a pair of Xeon E5520 processors and 96GB of RAM.   My original plan was to upgrade each host to 72GB of RAM, but I had a bunch of 8GB DIMMs from my failed R310 upgrades, and I didn’t want to pay return shipping or restocking fees.  The Workload cluster is configured with a virtual distributed switch for storage, a vDS for VM traffic, and a standard switch for management and vMotion traffic.

The vGPU cluster is the only cluster that doesn’t follow the hardware standards.  The server is a Dell PowerEdge R730 with 32GB of RAM.  The server is configured with the Dell GPU enablement kit and currently has an NVIDIA GRID K1 card installed.

My Nutanix CE box is a PowerEdge R610 with 32GB of RAM.

Storage

The storage tier of my lab consists of a single Synology Diskstation 1515+.  It has four 2 TB WD Red drives in a RAID 10 and a single SSD acting as a read cache.  A single 2TB datastore is presented to my ESXi hosts using NFS.  The Synology also has a couple of CIFS shares for things like user profiles and network file shares.

Network

The network tier consists of a Juniper SRX100 firewall and a pair of Linksys SRW2048 switches.  The switches are not stacked but have similar configurations for redundancy.  Each server and the Synology are connected into both fabrics.

I have multiple VLANs on my network to segregate different types of traffic.  Storage, vMotion, and management traffic are all on their own VLANs.  Other VLANs are dedicated to different types of VM traffic.

That’s the overall high-level view of the current state of my home lab.  One component I haven’t spent much time on so far is my Horizon design.  I will cover that indepth in an upcoming post.

Home Lab Expansions #VDM30in30

Over the last two weeks, I’ve made some significant changes to my home lab.  The changes were brought about by a steadily increasing electric bill that had been increasing significantly over the last few months.

I picked up two new servers, a PowerEdge R710 and a PowerEdge R610,  on eBay that will replace the 3-node, 2U Dell DCS6005 that I had been using for my lab.  Both servers come with dual quad-core Xeon processor and 24 GB of RAM.   The R610 will be for server workloads, and the R710 will be for testing out VDI related software.

Although I end up with fewer cores and less RAM for running virtual machines, the two new servers have a few features which make them attractive for home lab use.  The include onboard power monitoring to track electricity usage, and I easily view this within the iDRAC.  The baseboard management on the DCS6005 nodes never worked right, so the new servers had iDRAC6 Enterprise modules added for improved remote management.  The new servers are far quieter than the DCS6005, and I can barely hear them once they are running. They also have more expansion slot, which will allow me to start testing GPUs with Horizon

Home Lab Updates

Back in December, I wrote about my home lab setup.  In the last few months, I’ve made a lot of changes to my lab as I’ve added new hardware and capabilities and shuffled some other equipment around to new roles.  It’s also changed locations as my wife and I moved into a new house last month.

The “newest” hardware in my lab is all used gear that I picked up off of eBay.  There were some great deals on custom Dell-built cloud servers from a large cloud provider that more than tripled the overall capacity of my lab.  The server run older AMD 2419EE processors, so they trade a lot of performance for power efficiency, and come with 48GB of RAM each.  They have their own set of quirks that need to be worked around, but that said, they do run vSphere 5.5 well.

The Dell servers can be rather finicky with the network switches that they work with, and the Linksys that I got late last year was swapped out for a Juniper EX4200 that I was able to get on loan from my employer.

The PowerEdge T110 II that I purchased last year has been moved from being a computer node to my primary storage node, and the storage OS has been migrated from OmniOS to Nexenta 4.0.  The PowerEdge T310 that was previously hosting storage has been retired, and I am selling it to a co-worker who is looking to start his own lab up.

I’ve also expanded the Fibre Channel footprint of my lab, and all my servers are connected to storage via Fibre Channel.  I recently picked up a Silkworm 3250 on eBay for $30.  I was also able to pick up a few 4GB QLogic 2450 cards for about $8 a piece. 

There is still some work that needs to be done on the lab.  I need to run some new electrical circuits into my makeshift server room as well as add some venting to take care of the excess heat.  I also plan on running Ethernet throughout the new house, and that will all terminate at my lab as well.

Looking for New Home Lab Storage

I’ve been a fan of Nexenta for a long time.  I’m not sure if it was Sun’s ZFS file system, the easy-to-use web interface, or how Nexenta was able to keep up with my changing needs as my lab grew and acquired more advanced gear.  Or it was support for VAAI.  Whatever the reason, or combination of reasons, Nexenta was a core component in my lab.

That changed a few months ago when I started a series of upgrades that culminated in my storage moving to a new server.  During those upgrades, I came across a few issues that forced me to change to OmniOS and NAPP-IT as a short-term solution while waiting to see if a new version of Nexenta was released.

Nexenta is no longer viable as a storage platform in my lab because:

  • Version 3.1.3.5 doesn’t play nicely with the Broadcom NICs in the Dell PowerEdge T310 that I use for storage due to a line being commented out in the driver.  Even when I fix this, it’s not quite right.
  • Version 3.1.5 didn’t work period when I had USB devices plugged in – which makes it hard to use when you have USB hard drives and a USB keyboard.
  • Version 4 is vaporware.

The OmniOS/Napp-IT combination works, but it doesn’t meet one of my core requirements – VAAI support.

It doesn’t seem like a new version of Nexenta Community Edition will be coming anytime soon.  A beta was supposed to be released early in January, but that hasn’t materialized, and it’s time to move onto a new platform.

My requirements are fairly simple.  My requirements are:

  1. Spouse Approval Factor – My wife is 7 months pregnant and wants to buy a house.  Any solution must be either open-source or extremely cheap.  The less I spend, the better.
  2. Support for Fibre Channel – I’ve started putting 4GB Fibre Channel in as my storage network.  The solution must have support for using Fibre Channel as I would prefer to keep using it for my storage network.
  3. VMware APIs for Array Integration – My home lab is almost entirely virtualized, so any solution must support VAAI.

ZFS isn’t a requirement for a new system, and I’m not worried about performance right now.  A web interface is preferred but not required.

If you have any recommendations, please leave it in the comments.

Three Tips for Starting Your Home Lab

Home labs have been the topic de jure lately, and I covered my lab in my last post.  Virtualization makes it much easier to test new products and run an IT environment at home.  As Chris Wahl said, “Having a lab is a superb way to get your hands dirty with hardware and troubleshooting that just can’t be experience in a “cloud” environment.”

But where, and how, do you get started?  Here are three tips that will help you get started without breaking the bank.

Tip 1: Start Small

A good home lab takes time and money to build up.  You won’t be able to go out and buy a few servers, shared storage, and decent networking gear to run a miniature enterprise environment in your basement.  If you’re just starting out or branching into a new area, you might not need systems that can do a lot of heavy lifting.  An older desktop, or server, might not be on the hardware compatibility list or even offer great performance, but it could be the starter environment that you use to get your feet wet on a platform.

Your lab doesn’t need to run on separate hardware either.  VMware Workstation (Windows/Linux)/Fusion (Mac) and Virtualbox are two virtualization products that allow you to run virtual machines on your desktop or laptop.  GNS3 can run Cisco IOS without having to buy actual Cisco hardware.  Performance won’t be the greatest, and it is very easy to bog down your machine if you aren’t careful, but it can be one of the fastest ways to start getting hands-on without a significant investment.

Tip 2: Look for Deals

The enterprise-grade equipment that you’d find in an office or data center is expensive, and it is priced outside of what most people would be willing to pay for hardware if it was purchased new.  But as you start working on more sophisticated things, you will want to get better equipment. 

Build Your Own Server

There are three good ways to go about doing this.  The first is to build your own servers.  Chris Wahl has a nice list of whitebox servers that members of the community have built.  The nice thing about this is that you can control exactly what components are in the system, and many of the designs listed have a sub-$1000 bill of materials before sales at Amazon or NewEgg.

Buy a New Server

If assembling a server isn’t something that you have the time or inclination for, then you can buy lower-end retail hardware.  ESXi runs on a surprising number of platforms, and the HCL includes inexpensive options like HP Microservers, Dell PowerEdge T110 II, and even the Mac Mini.  Even a low-end server or Mac Mini maxed out with RAM can easily cross the $1000 barrier, but you get the peace-of-mind of having a manufacturer warranty for at least part of the machine’s life.

Pre-Owned Equipment

Off-Lease.  Refurbished.  Pre-Owned.  Whatever you call it, it’s buying used equipment.  And like a day-old loaf of bread, it’s a little stale but still usable and much cheaper.

There is still a lot of life left in the three to five year old equipment that you can pick up.  Many of these servers will show up on the VMware HCL and run vSphere 5.1 or 5.5 without any problem.  Depending on where you get them from, you may get a warranty.

A few months ago, Scott Lowe took this route when building up his lab for OpenStack.  He picked up two off-lease Dell C6100 servers that provided him with 8 blades, 16 processors, and 192 GB of RAM.

Another possible source purchasing used equipment is your employer.  Many employers, especially larger ones, are constantly refreshing equipment in their datacenters.  Purchased equipment needs to be retained or disposed of, and your company may allow you to purchase some of that equipment if their policies allow.

eBay, Craigslist, and local computer recyclers may also be good sources of equipment, and you can often get very good deals on items that they collected from a business.

Caveat emptor applies whenever you buy used equipment.  Although most local businesses and eBayers have reputations to protect, you may not have any recourse if the server you bought turns out to be a rather large and expensive paperweight. 

All of the Above

As you build up your lab, you’ll probably end up with an odd mixture of equipment.  My lab has my PowerEdge T310 that I purchased new over four years ago and a T110 II from Dell Outlet utilizing used QLogic Fibre Channel HBAs that I picked up from a friend who runs a computer recycling business.

Tip 3: Utilize Free/Open Source/Not-for-Resale

The untimely death of MIcrosoft’s TechNet program hurt hobbyists and IT professionals by taking away a source of legitimate software that could be used almost perpetually in a home lab.  That’s been replaced with 120-day trials.  I don’t know about you, but I don’t want to be rebuilding a domain controller/DHCP/DNS infrastructure three times per year.  I pick on Microsoft here because many of the workloads I want to run in my home lab are Microsoft-based, and I find it to be a bigger pain to rebuild an Active Directory infrastructure than a virtual infrastructure.

VMware hasn’t had a TechNet equivalent for many years.  There have been murmurings in the community that it might be coming back, but that doesn’t seem likely at this point.  VMware’s trials only last 60 days on most products, although some, such as Workstation and Fusion, only have 30 day trials.  Although VMware has the free ESXi Hypervisor, the 5.5 version is crippled in that the vSphere client cannot manage machines with the latest vm hardware compatibility levels. 

If there are parts of your lab that you don’t want to rebuild on a regular basis, you will need to look to free and/or open source products beyond the Linux, MySQL, and LibreOffice that people normally associate with those categories.  Some vendors also offer Not-For-Resale licenses, although some of those offers may only be available if you possess a Microsoft or VMware Certification.

The list below does not include everything out there in the community that you can try out, but here are a few products that offer free or not-for-resale versions:

Bonus Tip: Be Creative

If you’ve ever read one of these types of lists on LinkedIn or the Huffington Post, you knew this was coming. 

If you look out in the community, you’ll see some very creative solutions to the problems that home labs can pose.  I’ve posted two of the best ideas below:

Frank Denneman built a rack for his servers using two Lack tables from Ikea.

Greg Rouche’s VSAN/Infiniband environment is built sans server cases on a wood bookshelf.

My Home Lab

The topic of home labs is a popular one lately with several people talking about it on their blogs or twitter.  They are one of the best learning tools that you can have, especially if you are a hands-on learner or just want to try out the latest technology.  I’ve had some form of a home lab since I graduated from college in 2005 and has ranged in size from an old desktop running a Windows Server domain controller to the multi-device environment that I run today. 

It’s taken me a few years to build up to where I am today, and most of my equipment is good enough to keep my wife happy that I’m not spending too much money.

The most recent upgrades to my lab were adding 4GB Fibre Channel and switching from Nexenta to OmniOS.  I have also been slowly swapping out hard drives in my storage box to bring everything up to 1TB drives.  The last one should be arriving by the end of the week.

I use both Fibre Channel and iSCSI in my lab.  Fibre Channel is used to connect the storage to the compute node, and iSCSI is used to connect to the backup server.

Compute

  • Dell PowerEdge T110 II
  • Xeon E3-1240v2
  • 32GB RAM
  • ESXi 5.5
  • 2x 50GB OCZ Vertex 2 SSD
  • 1 Bootable 2GB USB Flash drive
  • 3x gigabit NICs (2 Single Port Intel NICs, 1 Broadcom Onboard NIC)
  • 1 QLogic 2460 4GB Fibre Channel HBA

Storage

  • Dell PowerEdge T310
  • Xeon X3430
  • 8GB RAM
  • OmniOS w/ NAPP-IT Management Interface
  • ZFS (2 mirrored pairs w/ SSD Cache)
  • 3x 7200 RPM 1TB Hard Drives (1x WD Blue, 1x WD Red, 1 Seagate Constellation)
  • 1x 7200 RPM 500 Hard Drive (soon to be upgraded to a 1 TB WD Red)
  • 1 60GB SSD
  • 2x 60GB USB Hard Drives for OmniOS
  • 4 gigbait NICs (2 Onboard Broadcom NICs, 1 Dual-Port Intel NIC)
  • 1 QLogic 2460 4GB Fibre Channel HBA in Target Mode

Backup

  • HP WX4400 Workstation
  • Intel Core 2 Duo 4300
  • 4GB RAM
  • Windows Server 2008R2
  • Veeam 7
  • 80GB OS Drive
  • 2x WD Blue 500GB Hard Drives in software RAID1
  • 3 gigabit NICs (1 Onboard Broadcom NIC, 1 Dual-Port Broadcom NIC)

Network

  • Firewall/Router – Juniper SRX100
  • Switch – Linksys 48-port gigabit switch