Featured

Three Tips for Starting Your Home Lab

/ seanpmassey

Home labs have been the topic de jure lately, and I covered my lab in my last post.  Virtualization makes it much easier to test new products and run an IT environment at home.  As Chris Wahl said, “Having a lab is a superb way to get your hands dirty with hardware and troubleshooting that just can’t be experience in a “cloud” environment.”

But where, and how, do you get started?  Here are three tips that will help you get started without breaking the bank.

Tip 1: Start Small

A good home lab takes time and money to build up.  You won’t be able to go out and buy a few servers, shared storage, and decent networking gear to run a miniature enterprise environment in your basement.  If you’re just starting out or branching into a new area, you might not need systems that can do a lot of heavy lifting.  An older desktop, or server, might not be on the hardware compatibility list or even offer great performance, but it could be the starter environment that you use to get your feet wet on a platform.

Your lab doesn’t need to run on separate hardware either.  VMware Workstation (Windows/Linux)/Fusion (Mac) and Virtualbox are two virtualization products that allow you to run virtual machines on your desktop or laptop.  GNS3 can run Cisco IOS without having to buy actual Cisco hardware.  Performance won’t be the greatest, and it is very easy to bog down your machine if you aren’t careful, but it can be one of the fastest ways to start getting hands-on without a significant investment.

Tip 2: Look for Deals

The enterprise-grade equipment that you’d find in an office or data center is expensive, and it is priced outside of what most people would be willing to pay for hardware if it was purchased new.  But as you start working on more sophisticated things, you will want to get better equipment. 

Build Your Own Server

There are three good ways to go about doing this.  The first is to build your own servers.  Chris Wahl has a nice list of whitebox servers that members of the community have built.  The nice thing about this is that you can control exactly what components are in the system, and many of the designs listed have a sub-$1000 bill of materials before sales at Amazon or NewEgg.

Buy a New Server

If assembling a server isn’t something that you have the time or inclination for, then you can buy lower-end retail hardware.  ESXi runs on a surprising number of platforms, and the HCL includes inexpensive options like HP Microservers, Dell PowerEdge T110 II, and even the Mac Mini.  Even a low-end server or Mac Mini maxed out with RAM can easily cross the $1000 barrier, but you get the peace-of-mind of having a manufacturer warranty for at least part of the machine’s life.

Pre-Owned Equipment

Off-Lease.  Refurbished.  Pre-Owned.  Whatever you call it, it’s buying used equipment.  And like a day-old loaf of bread, it’s a little stale but still usable and much cheaper.

There is still a lot of life left in the three to five year old equipment that you can pick up.  Many of these servers will show up on the VMware HCL and run vSphere 5.1 or 5.5 without any problem.  Depending on where you get them from, you may get a warranty.

A few months ago, Scott Lowe took this route when building up his lab for OpenStack.  He picked up two off-lease Dell C6100 servers that provided him with 8 blades, 16 processors, and 192 GB of RAM.

Another possible source purchasing used equipment is your employer.  Many employers, especially larger ones, are constantly refreshing equipment in their datacenters.  Purchased equipment needs to be retained or disposed of, and your company may allow you to purchase some of that equipment if their policies allow.

eBay, Craigslist, and local computer recyclers may also be good sources of equipment, and you can often get very good deals on items that they collected from a business.

Caveat emptor applies whenever you buy used equipment.  Although most local businesses and eBayers have reputations to protect, you may not have any recourse if the server you bought turns out to be a rather large and expensive paperweight. 

All of the Above

As you build up your lab, you’ll probably end up with an odd mixture of equipment.  My lab has my PowerEdge T310 that I purchased new over four years ago and a T110 II from Dell Outlet utilizing used QLogic Fibre Channel HBAs that I picked up from a friend who runs a computer recycling business.

Tip 3: Utilize Free/Open Source/Not-for-Resale

The untimely death of MIcrosoft’s TechNet program hurt hobbyists and IT professionals by taking away a source of legitimate software that could be used almost perpetually in a home lab.  That’s been replaced with 120-day trials.  I don’t know about you, but I don’t want to be rebuilding a domain controller/DHCP/DNS infrastructure three times per year.  I pick on Microsoft here because many of the workloads I want to run in my home lab are Microsoft-based, and I find it to be a bigger pain to rebuild an Active Directory infrastructure than a virtual infrastructure.

VMware hasn’t had a TechNet equivalent for many years.  There have been murmurings in the community that it might be coming back, but that doesn’t seem likely at this point.  VMware’s trials only last 60 days on most products, although some, such as Workstation and Fusion, only have 30 day trials.  Although VMware has the free ESXi Hypervisor, the 5.5 version is crippled in that the vSphere client cannot manage machines with the latest vm hardware compatibility levels. 

If there are parts of your lab that you don’t want to rebuild on a regular basis, you will need to look to free and/or open source products beyond the Linux, MySQL, and LibreOffice that people normally associate with those categories.  Some vendors also offer Not-For-Resale licenses, although some of those offers may only be available if you possess a Microsoft or VMware Certification.

The list below does not include everything out there in the community that you can try out, but here are a few products that offer free or not-for-resale versions:

Bonus Tip: Be Creative

If you’ve ever read one of these types of lists on LinkedIn or the Huffington Post, you knew this was coming. 

If you look out in the community, you’ll see some very creative solutions to the problems that home labs can pose.  I’ve posted two of the best ideas below:

Frank Denneman built a rack for his servers using two Lack tables from Ikea.

Greg Rouche’s VSAN/Infiniband environment is built sans server cases on a wood bookshelf.

Book Review–The Phoenix Project

/ seanpmassey

Rating – Highly Recommend

When I sat down to start writing this review, it was almost 11:30 PM on Friday night.  For some reason, I’m wired even though I’m running on about five hours of sleep total.  My mind is racing, and I can’t get it to settle down.  So, I’ve decided to try writing my first book review instead.

I’ve just finished reading The Phoenix Project (Amazon)(Barnes and Noble) – a book about how IT is intertwined with the business and is now a critical role in it’s success or failure.  Modern businesses live and die by the information that is contained in and produced by those system.  If they don’t work, the business doesn’t work either.

The book follows Bill, a mid-level IT manager who gets gets a surprise promotion to acting VP of IT Operations one morning and has to take charge of an IT Operations group that is overworked to the point of being dysfunctional.  The group is sorely lacking in basic IT practices like adequate change control procedures that cause numerous severity 1 outages that bring the business to its knees, poor relationships with other departments, no budget, and most of the knowledge living inside the head of the senior systems architect.  At the start, Bill has to scrape by with what he has.

Bill is accompanied on this journey by Patty and Wes, the two department managers within IT Operations who act as Bill’s superego and id respectively; Brent, the systems architect who is the cause of, and solution to, many of the problems they face early on; John, the CISO who is loathed by everyone around him; and Steve, the CEO of “Parts Unlimited” who promoted him. 

Other key players include members of the senior management staff such as Chris, the acting CIO and head of Development, Dick, the CFO/COO, and Sarah, the …um…ambitious and troublemaking VP of Retail Operations and the driving force behind Project Phoenix, the expensive and expansive IT project that “Parts Unlimited” has staked its future on.

The story wouldn’t be complete without the enigmatic Eric, a prospective board member who acts as an eccentric mentor to Bill, John, and others at Parts Unlimited.  It is through Eric that Bill is introduced to many of the bits and pieces that will transform him and the Parts Unlimited IT Team.  In some senses, he’s the Yoda or Dumbledore to Bill’s Luke Skywalker or Harry Potter by not always giving the answers or answering with another question.  Eric also comes off as an author avatar at times to discuss the concepts that the book is trying to teach.  This isn’t necessarily a bad thing, though, because they allow Bill to reach many of the conclusions himself.

Overall, the characters are flat.  I’m not sure if they were intentionally designed to be that way to act as self-inserts or if it is because the authors are trying to present a management text as a work of fiction.  Either way, its not too big of a detriment to the concepts that the book is trying to teach.

There are a number of factors that come off a little too unrealistic for my tastes – the changes happen very quickly and without much resistance, or everyone comes around in the end to that way of doing things.  Or Wes falling in line under Bill rather quickly.  But like I said with the characterizations, this isn’t critical to what the book is trying to accomplish.

Where The Phoenix Project excels is breaking down the ideas behind lean operations such as kanban and the Toyota Production System and presenting them to technical people who might not have been exposed to them.  It also does a good job avoiding overly technical jargon and of comparing IT operations to that of a factory floor in order to make it accessible to non-technical managers who want, or need, to know about IT and how it relates to and integrates with the operation of a business.

It’s also spurred my interest in learning more about some of the lean manufacturing and continuous improvement process techniques.  I’m not the management type – I see myself as more of a systems architect – but I can see how techniques that are used on a production floor can be applied to improving IT operations – even if it is something as simple as automating routine operations like account creation or virtual machine deployments.  It also gives me an excuse to dig out my copy of Dr. Eliyahu Goldratt’s The Goal that I received from a former boss of mine as it is heavily referenced in the book.

The Phoenix Project would have been a great book for my boss at my last job.  She had been hired to oversee one department and had IT tacked onto it as one area of responsibility.  This would have given her a management-level perspective of what IT is about and would have better prepared her for managing that group without bogging her down in technical minutia.  This isn’t meant as a slam against her – in the year I worked under her, she learned a lot about IT and was able to instill a customer-oriented focus in the organization, but this type of resource may have better prepared her for the challenges that overseeing an IT Department presented and helped us better address the bottlenecks that we had as a department as a result of the focus change.

Provided that this book is approached as a management text disguised as a novel, I highly recommend this book to anyone who is in IT, especially if they aspire to a management or senior technical role.  I also think this book would be a good read for non-technical senior management.

vCenter Server Virtual Appliance 5.5 SSL Certificates Part 2 – Certificate Installation

/ seanpmassey / 2 Comments

Note: While writing this post, VMware has updated the instructions for installing certificates on the vCenter Server Appliance. The KB for this article can be found here.

In my last post, I walked through the process of generating the SSL certificates using OpenSSL and the OpenSSL configuration files for each service on the vCenter 5.5 appliance. So now that you have your SSL certificates, keys, and certificate authority chain, you can install them on your vCenter Server Appliance. This isn’t as hard as it actually looks, but the VMware documentation does miss a couple of the steps that are needed to do this successfully.

Getting Started

If you’re like me and using a Windows machine, or if your employer only provides you with a Windows machine, you’ll need to make sure you have a few tools installed in order to change the SSL certificates on the vCenter Appliance. Those tools are:

  • An application to transfer files to a Linux machine (I use WinSCP, which can be downloaded from here)
  • An SSH client application (I use PuTTY, which can be downloaded from here)

If you’re comfortable a Linux command shell, this process should be pretty straightforward. Until a year or two ago, I wasn’t, and I know that there are a lot of other Admins out there who don’t have a lot of Linux experience, so I tried to include all of the required commands and screenshots where possible.

Converting the Certificate Chain to .pem

The Microsoft certificate authority only provides certificate chains in a .p7b file format, and these cannot be used on the Linux appliance. So before you copy any files over to the appliance, you will want to convert the certificate chain from your certificate authority from a .p7b file to a .pem file. Since the OpenSSL components are installed on your computer, you might as well do this before copying anything over to the Linux appliance.

To convert the .p7b file to a .pem file, you need to open the windows command prompt as an administrator, navigate to where your certificate chain file is stored, and run the following command:

openssl pkcs7 -print_certs -in certchainname.p7b -out cachain.pem

This will create a new certificate file in the .pem format. You will need to edit the file slightly to remove a few lines of text as the pem file, so you will need to open it up in a text editor. You will want to remove the lines before each “—–BEGIN CERTIFICATE—–.” These lines may start with “subject=” and “issuer=” like the screenshot below.

After you save the changes to the .pem file, you should copy it into the folders that you created for each certificate. This will make it easier to work with the chains when you copy the files over to the appliance.

Copying the Certificate Files to the Appliance

Note: I do not use the Auto Deploy service in my lab at this time. Therefore, I will not be covering updating those certificates. I will probably cover this in a future post.

You will need to copy the certificate files, private keys, and the certificate chains over to the appliance before you can change the certificate that each service uses. You will need to open WinSCP and log into the vCenter Server Appliance as root (default password is vmware).

winSCP login 1

After you successfully login, you may see a MotD banner like the one below. You can click Continue to finish logging in.

winSCP login banner

Once you have logged in, you will need to navigate to the /SSL folder. This is where you will copy the certificate files, private keys, and the certificate authority chain to. You will need to create three folders inside the /SSL folder for the three groups of files that will need to be copied over. Those folders should be named:

  • vpxd
  • inventoryservice
  • logbrowser

winscp ssl folder

After these folders have been created, you will need to copy over your certificate files to each of these folders. WinSCP allows you to copy by dragging the files from the left side of the screen, which is your local computer, to the right side of the screen, which is the remote computer. When you have finished copying your files, you should have the certificate file (.crt), the certificate chain file (.pem), and the certificate private key file (.key) in each folder.

Updating the vCenter Service (vpxd) Certificate

After you’ve copied over your certificate files, it’s time to rock and roll. Updating the certificates on your vCenter Appliance isn’t that difficult, but it does require a little bit of command-linefu to make the magic happen. Before we can start, you need to log into your vCenter appliance using PuTTY to access the command line as the root user.

The first thing that you need to do is stop the vCenter Server service and the Single Sign-On Service. The command to stop these services is:

service vmware-stsd stop
service vmware-vpxd stop

After you have stopped these services, you need to change directories to /ssl/vpxd so we can create a certificate chain file that will be used by the vpxd service. This is a step that the VMware Documentation missed, but it isn’t hard to do. To create this file, you need to run the following command:

[sourceode]cat certificatefile.crt chainfile.pem > chain.pem[/sourcecode]

vpxd cert update 2

Once the chain.pem file has been created, you are ready to change the certificate for the vCenter Service. You need to run the following command to make this change:

/usr/sbin/vpxd_servicecfg certificate change chain.pem vpxdkeyfilename.key

vpxd cert update 3

If this completes successfully, you should receive a VC_CFG_RESULT=0. If you receive a result that is not VC_CFG_RESULT=0, you can check this VMware KB article to find out what the problem is. Also check you file names to make sure that you have the correct chain file and key file names.

If everything has completed successfully, you will need to restart the SSO service before continuing on to change the Inventory Service Certificate. The command to restart this service is:

service vmware-stsd start

Changing the Inventory Service Certificate

The next certificate that you need to update is the vCenter Inventory Service certificate. Unlike the other two services on the vCenter Server appliance, the certificate chain for this service needs to be converted to a pfx file. Updating this certificate is a little more complicated as well since you need to unregister the Inventory Service from the Single Sign-On service before you can update the certificate. This is a step that will also need to be repeated with the Log Browser Service.

In order to unregister the Inventory Service, you need to enter the following commands on the appliance:

cd /etc/vmware-sso/register-hooks.d
./02-inventoryservice --mode uninstall --ls-server https://server.domain.com:7444/lookupservice/sdk

inventory ssl cert 1

You will need to change the url in the second command by replacing server.domain.com with the fully qualified domain name of your appliance.

After you have successfully unregistered the Inventory Service, you will need to navigate to /ssl/inventoryservice to work with the certificate files that you copied over earlier. The first thing that needs to be done is to rename the rui_inventoryservice.crt and rui_inventoryservice.key files and remove the _inventoryservice from the file names. When this is done, the files should be named rui.crt and rui.key. There are multiple ways to do this with Linux, including using cat command to read the contents into a new file or copying the files to a new file name. One of the cleaner methods is to use the mv (move) command. The commands for this task are:

 mv rui_inventoryservice.crt rui.crt
mv rui_inventoryservice.key rui.key

Before you can create the pfx file that the Inventory Service needs, you need to create a certificate chain consisting of the rui.crt and the certificate authority chain.

 cat rui.crt cachain.pem > chain.pem

With the certificate chain file created, you can move onto creating the pfx certificate file for the Inventory Service. Like everything that we’ve done with certificates so far, we’ll be using the OpenSSL command to create the file. The command to create the pfx file is:

openssl pkcs12 -export -out rui.pfx -in chain.pem -inkey rui.key -name rui -passout pass:testpassword

After you have created the PCKS12 file, you will need to copy it along with the rui.crt and rui.key files to /usr/lib/vmware-vpx/inventoryservice/ssl. The command to do this is:

cp rui.pfx rui.crt rui.key /usr/lib/vmware-vpx/inventoryservice/ssl

After you have copied the files, you will need to change to the investory service ssl directory and modify the permissions on the files. The rui.pfx file is set to be read-only by the file owner, and the other two files are set to read-write by the owner and read by everyone else. The commands to do this are:

cd /usr/lib/vmware-vpx/inventoryservice/ssl/
chmod 400 rui.key rui.pfx
chmod 644 rui.crt

Finally, you need to reregister the Inventory Service with the Single Sign-On service. When you do this, it will use the new certificate file that you placed in the directory. Before you do this, you will want to run the unset HISTFILE command since you will be entering a plaintext password for a user account that has administrator rights to your single sign-on service. To run this command successfully, you need to change “server.domain.com” to your server’s fully qualified domain name, sso_administrator to an administrator account in your SSO administrator (probably administrator@vsphere.local) and sso_password with your SSO password (if you ran the default config, it is blank).

cd /etc/vmware-sso/register-hooks.d
./02-inventoryservice --mode install --ls-server https://server.domain.com:7444/lookupservice/sdk --user sso_administrator --password sso_administrator_password

A successful result will look like the screenshot below:

inventory ssl cert 3

You can have the vCenter inventory service reregister itself on a reboot by entering the following command:

rm /var/vmware/vpxd/inventoryservice_registered

You can also reregister the service manually by restarting the vpxd and inventory services.

service vmware-inventoryservice stop
service vmware-vpxd stop
service vmware-inventoryservice start
service vmware-vpxd start

inventory ssl cert 4

Replacing the Log Browser Certificate

Note: This will be the last certificate that we replace in this tutorial as the Auto Deploy certificate is not required if Auto Deploy is not used in your environment.

The last certificate that we need to replace is the Log Browser Service certificate. The process for replacing the Log Browser certificate is very similar to replacing the certificate for the Inventory Service, and you will need to take many of the same steps to complete this task.

To start this process, you will need to unregister the Log Browser service from Single Sign-On. You can do this with the following command after replacing server.domain.com with your appliance’s fully qualified domain name:

cd /etc/vmware-sso/register-hooks.d
./09-vmware-logbrowser --mode uninstall --ls-server https://server.domain.com:7444/lookupservice/sdk

After you have successfully unregistered the Log Browser Service, you will need to change to the /ssl/logbrowser directory so you can work with the certificates that you copied over from your machine. The process is similar to what you did with the Inventory Service certificates, and you will need to remove the _logbrowser from the file names before creating a .pfx certificate file. The steps we need to take, like the Inventory Service above, are to rename the certificate and key files, concatenate the certificate and the certificate chain, and then create the .pfx certificate file. The commands to do this are:

 mv rui_logbrowser.crt rui.crt
mv rui_logbrowser.key rui.key
cat rui.crt cachain.pem > chain.pem
openssl pkcs12 -export -out rui.pfx -in chain.pem -inkey rui.key -name rui -passout pass:testpassword

After you create the rui.pfx file, you will need to copy the rui.pfx, rui.crt, and rui.key files to /usr/lib/vmware-logbrowser/conf and then change the permissions on the certificate files. The commands for this is:

cp rui.pfx rui.crt rui.key /usr/lib/vmware-logbrowser/conf
cd /usr/lib/vmware-logbrowser/conf
chmod 400 rui.key rui.pfx
chmod 644 rui.crt

The last step is to reregister the Log Browser service. The command to do this is below. To run this command successfully, you need to change “server.domain.com” to your server’s fully qualified domain name, sso_administrator to an administrator account in your SSO administrator (probably administrator@vsphere.local) and sso_password with your SSO password (if you ran the default config, it is blank).

cd /etc/vmware-sso/register-hooks.d
./09-vmware-logbrowser --mode install --ls-server https://server.domain.com:7444/lookupservice/sdk --user sso_administrator --password sso_administrator_password

A successful result will look like the screenshot below:

log browser ssl cert 2

If you received a success result, you will need to restart the Log Browser service by using the following commands:

service vmware-logbrowser stop
service vmware-logbrowser start

The Log Browser service is the last certificate that we’re changing, so you will need to reboot the appliance after you have successfully updated the certificate. The reboot allows all of the new certificates to take effect, and you may encounter certificate errors when logging into the web interface or the C# client until you reboot the appliance.

After a reboot, you shouldn’t see any certificate warnings when logging into the Appliance management webpage or the vSphere Web Client…provided that you’ve installed the root and intermediate certificates on your machine.

successful update

successful update 2
Look, Ma! No Certificate Warnings!

This wraps up the process of updating certificates on the vCenter 5.5 appliance. I will be coming back to the vCenter Server Virtual Appliance in the coming weeks when I tackle building a VMware Horizon View environment on vSphere 5.5 (spoilers – Horizon View 5.2 works on vSphere 5.5), but I will be moving on to some other topics that I want to cover.

vCenter Server Virtual Appliance 5.5 SSL Certificates – Part 1

/ seanpmassey / 6 Comments

In my last few posts, I focused on configuring the vCenter Server Virtual Appliance. Now that the appliance is up and running, it’s time to install and configure SSL certificates from an internal certificate authority. Much like the Windows vCenter application, SSL certificates are very important for vCenter 5.5 appliance.

SSL certificates in vCenter became an issue in vCenter 5.1 when a self-signed or invalid certificate could prevent an upgrade from a previous version from going forward. Those difficulties have been eased somewhat in vCenter 5.5 with the release of a certificate management tool that helps install the required certificates.

Before you can install certificates on the vCenter appliance, there are a few key points you should know about how certificates are used in vCenter and how those certificates need to be created. Derek Seaman covers this very well in his vCenter 5.1 blog. Instead of trying to duplicate his research and work, the links to will be provided below. I HIGHLY recommend that you read his posts on SSL certificates before you attempt to update the SSL certificates on the appliance.

The links to Derek’s blog are:

In addition to Derek’s blogs, a few tools you will need to generate and install new SSL certificates. Those tools, which are geared towards a Microsoft environment, are:

  • WinSCP – used to copy new certificate files over to the appliance
  • Putty – used to access the appliance’s command line remotely to execute commands to install the certificates
  • OpenSSL for Windows – Version 0.9.8y is recommended for creating the certificate signing requests for the vCenter Server Appliance certificates
  • A PKI environment to mint certificates. This tutorial will use a Windows-based Certificate Authority

The instructions for updating the appliance’s certificates can be found in the VMware Knowledgebase. Those instructions, along with some of the information Derek provided in his blog, will be used for this process.

Getting Started

The first thing that you need to do is create a certificate template on your Windows Certificate Authority. The instructions in Part 6 of Derek Seaman’s blog covers this step very well. The template should be set up as a Windows Server 2003 Enterprise template if it should be available in the Certificate Services Self-Service web portal.

The next step is to download and install OpenSSL for Windows version 0.9.8y on a workstation or server. You should add OpenSSL to the Windows PATH variable. This will make it easier for you to use OpenSSL to generate the certificate signing requests later on.

Each service on the vCenter appliance will need to have it’s own unique certificate. The best way to keep track of the three (or four if Auto Deploy is used) certificate requests is to create folders to organize the certificate signing requests, certificate files, and OpenSSL configuration files. The folders that should be created are:

  • VMware vCenter Service Certificate
  • VMware Inventory Service Certificate
  • VMware Logbrowser Service Certificate
  • VMware vCenter Autodeploy Service Certificate (optional, not required if Auto Deploy is not used)

You will need to create an OpenSSL configuration file for each folder. This file will contain the details that OpenSSL needs to create the certificate signing request, and these files will keep OpenSSL from prompting for information when you go to generate the certificate request. You will need to configure the subjectAltName field because your vCenter server will have multiple names that it can be accessed under, including the DNS single label name, fully qualified domain name, and IP addresses. You will also need to fill in your organizational details in the [req_distinguished_name] section.

Most of the fields in the configuration file only need to be changed once. When testing this procedure out, I found that the easiest way to manage this was to create a template with all the fields filled out except for the organizationalunitname field. I would then copy this template into each of the folders that I created, rename it, and then modify the file to update the organizationalunitname.

To create the configuration file, open up a text editor and paste the code below into it:

[ req ]

default_md = sha512
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
input_password = testpassword
output_password = testpassword

[ v3_req ]

basicConstraints = CA:false
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS:<DNS short name change me>, IP:<IPv4 Change Me>, IP:<IPv6 Change Me>, DNS:<FQDN Change Me>

[ req_distinguished_name ]

countryName = <Country Change Me>
stateOrProvinceName = <State Change Me>
localityName = <City Change Me>
0.organizationName = <Org Change Me>
organizationalUnitName = <VMware vCenter Service Certificate, VMware Inventory Service Certificate, VMware Logbrowser Service Certificate, VMware vCenter Autodeploy Service Certificate>
commonName = <FQDN Change Me>

As mentioned above, the organizationalunitname field will need to be different for each configuration file. The value in this field should match the name of the service that the certificate will be used with.

Service Folder Name OrganizationalUnitName Config File Name
vCenter Server VMware vCenter Service Certificate VMware vCenter Service openssl_vpxd.cfg
Inventory Service VMware Inventory Service Certificate VMware Inventory Service openssl_inventoryservice.cfg
Log Browser Service VMware Logbrowser Service Certificate VMware Logbrowser Service openssl_logbrowser.cfg
AutoDeploy Service VMware vCenter Autodeploy Service Certificate VMware vCenter Autodeploy Service openssl_autodeploy.cfg

There is a sample configuration file below. OpenSSL will generate a certificate signing requests for the vCenter Service Certificate when it is run with this configuration file. The lines that have been bolded are the ones that were updated.

[ req ]

default_md = sha512
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
input_password = testpassword
output_password = testpassword

[ v3_req ]

basicConstraints = CA:false
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS:vctest2, IP:10.1.1.161, DNS:vctest2.homedomain.private

[ req_distinguished_name ]

countryName = US
stateOrProvinceName = Wisconsin
localityName = Kimberly
0.organizationName = homedomain
organizationalUnitName = VMware vCenter Service Certificate
commonName = vctest2.homedomain.private

Note: Make sure to use your ISO two letter country code otherwise you will get an error when trying to generate the certificate request. 

Creating the Certificate Signing Requests

The certificate signing requests are files that the certificate authority will use to mint new SSL certificates. OpenSSL will be used to generate these, along with a new private key, after the configuration files are made. Three (or four if AutoDeploy is used) of these files will need to be created.

In order to create them, open a command prompt as an adminstrator and navigate to the folder where your OpenSSL configuration files are stored. You will need to run the following command to generate the certificate requests:

openssl req -new -nodes -out rui_service.csr -keyout rui_service.key -config openssl_service.cfg

If you wanted to generate the certificate request for the vCenter Server certificate, you’ll want to open a command prompt as administrator, navigate to where your openssl_vpxd.cfg file is stored, and run OpenSSL this way:

openssl req -new -nodes -out rui_vpxd.csr -keyout rui_service.key -config openssl_vpxd.cfg

Submitting the Requests to the Certificate Authority

I’m not going to do a step-by-step walkthrough of submitting the requests to your certificate authority.

Once you have created all of the certificate requests, you need to submit them to your certificate authority to generate the certificates. If you are in a windows environment, there are a couple of ways to do this. You can submit them through the self-service web portal that can be accessed by going to http://nameofcertserver/certserv. You can also use the certreq.exe command line tool to submit the request.

If you created the VMware-SSL certificate template per Derek Seaman’s post, then you’ll want to use that template for generating the certificates. Otherwise you could use the Web Server template.

After you have created each certificate, you will need to download the certificate file as a Base64-encoded certificate file and save it as a .crt file. Windows Certificate authorities usually save the files as a .cer. You will also need a Base64-encoded copy of the certificate authority chain. This will include the root certificate and any intermediate certificates that might be used in your environment. This file comes as a .p7b, but I’ll cover converting that to a .pem file in the next post.

Speaking of next posts, I will be covering what to do now that the certificates have been created.

Finding and Remediating Horizon View Desktops with the Wrong Snapshot/Image

/ seanpmassey / 1 Comment

I want you to imagine this scenario for a moment.  You schedule an overnight recompose operation for some of your linked-clone desktop pools.  You go to bed knowing that your desktops will have that shiny new image in the morning.  But when you wake up, you have a warning from your monitoring system or the vCheck script that shows the datastore that your replica volumes run on are running out of space.  When you log in to check, you have a couple of extra replicas that shouldn’t be there.  There is no easy way to say which pools, or desktops, are the issue because the replica names are GUIDs that do not relate to the pools.

I’ve been in this spot many times.  Sometimes, a recompose fails with a recoverable error.  Other times a few desktops didn’t recompose because they were refreshed sometime between when I scheduled the recompose and when it actually kicked off. And on some very rare occassions, there was a desktop that clung to its image like a security blanket, and it wouldn’t update until it was deleted.

When this happens, there is, currently, only one way to find out which pools or desktops are the problem.  In order to properly diagnose and correct this, an administrator would need to log into View Administrator, open each pool individually, go into Inventory and select the View Composer details.  While it is a powerful management interface, View Administrator is not exactly the switftest tool for an administrator to use, and it could take a while for this information to load if you have a large number of pools and/or desktops per pool.
There had to be a better, and automated, way to handle this.  And there is.

View PowerCLI

The preferred method for automating Horizon View deployments is the View PowerCLI cmdlets.  These cmdlets are included on the View Connection Servers.  The two cmdlets that look like they would be useful for resolving this issue are Get-Pool and Get-DesktopVM.

However, this won’t quite work.  The output from the Get-DesktopVM cmdlet does not provide any information about the image or snapshot that the linked clones are using.  Even if we could find the desktops using View PowerCLI, there does not seem to be any way to remotely delete a linked-clone desktop using the View PowerCLI cmdlets, so this is not the solutions.  Deleting the desktop VMs in vCenter is not a good idea, and bad things happen when you remove the linked-clone desktop directly from vCenter.

Horizon View and LDAP

There is another way, though.  Before that option can be explored, we have to understand how Horizon View stores it’s configuration data.  The main database for Horizon View is an LDAP directory.  This directory contains all of the information about the Horizon View environment such as pools, desktop VMs, Active Directory users, and even ThinApp applications that are deployed through View Administrator.

Since this is an LDAP directory, we can directly query for the information that we need, and we’re not limited to PowerShell.  Any scripting or programming language that can talk to an LDAP directory can be used.  I use PowerShell with the Quest Active Directory Cmdlets because that is the language that I’m comfortable with.

If you’re not familar with LDAP, there are some terms that might not make sense.  Attribute is one term that I will be using.  An attribute is like a file, and this is where a data value is stored.  A group of attributes is called an object-class.  Object-classes are like folders, and there can be multiple levels of object-classes in the directory schema, or layout.
There are two object-classes that we are going to be concerned with.  Those two object-classes are:

  • pae-ServerPool –  This object-class defines the desktop pools.
  • pae-Server –  This object-class defines the desktops.

There is a memberDN attribute for each desktop pool.  This attribute contains the list of distinguished names for the desktops in that pool, and it will assist our search for desktops.
The Desktop Pool and Desktop VM records also contain a field that has the vCenter snapshot ID.  This field is the one that will be used to compare the desktop VM to the standard that has been applied to the pool.  This field is called pae-SVIVmSnapshotMOID in both the ServerPool (desktop pool) and Server(desktop VM) object-classes.
In order to find the desktops that aren’t using the correct image/snapshot, we need to compare the values in these two fields.  If they don’t match, they will be added to an object called DesktopExceptions that will be used for later processing.

Once we’ve identified all the desktops that need to be remediated, we need to find a method for removing the desktop so View will recreate it.  There are no native commands to do this, but there is a method provided by Andre Leibovici on his blog myvirtualcloud.net.  This method allows us to set the desktop state remotely through the LDAP datastore.  For our desktops that are using the wrong snapshots, the state will need to be set to “Deleting.”  Once we do this, the desktops will be deleted and recreated automatically.

These steps are wrapped up in the script below.  Since this talks directly to the LDAP directory for View and avoids using the View PowerCLI cmdlets, this can be run from any server with PowerShell and the Quest AD Cmdlets.

This code is also available on my GitHub site.

       

            <#
.SYNOPSIS
   Get-DesktopExceptions is a script that will locate VMware View Linked Clones that are not using the correct snapshot/image.  The script also contains an option to remediate any non-compliant desktops by deleting them and letting View recreate them.
.DESCRIPTION
   Get-DesktopExceptions will look in the View LDAP datastore to find the snapshot IDs used by the desktops and the pool. It compares these values to find any desktops that do not match the pool.  If the -Remediate switch is selected, the script will then remove them.  In order to run this script, the Quest Active Directory Cmdlets will need to be installed.
.PARAMETER ConnectionServer
   The View Connection server that you want to run this script against.
.PARAMETER Remediate
   Delete desktops that do not have the correct snapshots
.PARAMETER EmailRcpt
   Person or group who should receive the email report
.PARAMETER SMTPServer
   Email server
.EXAMPLE
   Get-DesktopExceptions -ConnectionServer connection.domain.com -Remediate -EmailRcpt user@domain.com -SMTPServer smtp.domain.com
#>
param($ConnectionServer,[switch]$Remediate,$EmailRcpt,$SMTPServer)

Function Send-Email
{
 Param([string]$SMTPBody,[string]$SMTPSubject = "View Snapshot Compliance Report",[string]$SMTPTo,$SMTPServer)
Send-MailMessage -To $SMTPTo -Body $SMTPBody -Subject $SMTPSubject -SmtpServer $SMTPServer -From "Notifications_noreply@gbdioc.org" -BodyAsHtml
}

Function Get-Pools
{
param($ConnectionServer)

$PoolList = @()

$arrIncludedProperties = "cn,name,pae-DisplayName,pae-MemberDN,pae-SVIVmParentVM,pae-SVIVmSnapshot,pae-SVIVmSnapshotMOID".Split(",")
$pools = Get-QADObject -Service $ConnectionServer -DontUseDefaultIncludedProperties -IncludedProperties $arrIncludedProperties -LdapFilter "(objectClass=pae-ServerPool)" -SizeLimit 0 | Sort-Object "pae-DisplayName" | Select-Object Name, "pae-DisplayName", "pae-SVIVmParentVM" , "pae-SVIVmSnapshot", "pae-SVIVmSnapshotMOID", "pae-MemberDN"

ForEach($pool in $pools)
{
$obj = New-Object PSObject -Property @{
           "cn" = $pool.cn
           "name" = $pool.name
           "DisplayName" = $pool."pae-DisplayName"
           "MemberDN" = $pool."pae-MemberDN"
           "SVIVmParentVM" = $pool."pae-SVIVmParentVM"
           "SVIVmSnapshot" = $pool."pae-SVIVmSnapshot"
           "SVIVmSnapshotMOID" = $pool."pae-SVIVmSnapshotMOID"
    }
$PoolList += $obj
}
Return $PoolList
}

Function Get-Desktop
{
param($MemberDN, $ConnectionServer)

$arrIncludedProperties = "cn,name,pae-DisplayName,pae-MemberDN,pae-SVIVmParentVM,pae-SVIVmSnapshot,pae-SVIVmSnapshotMOID".Split(",")
$Desktop = Get-QADObject -Service $ConnectionServer -DontUseDefaultIncludedProperties -IncludedProperties $arrIncludedProperties -LdapFilter "(&(objectClass=pae-Server)(distinguishedName=$MemberDN))" -SizeLimit 0 | Sort-Object "pae-DisplayName" | Select-Object Name, "pae-DisplayName", "pae-SVIVmParentVM" , "pae-SVIVmSnapshot", "pae-SVIVmSnapshotMOID"

Return $Desktop
}

$DesktopExceptions = @()
$pools = Get-Pools -ConnectionServer $ConnectionServer

ForEach($pool in $pools)
{
$MemberDNs = $pool.memberdn
 ForEach($MemberDN in $MemberDNs)
 {
 $Desktop = Get-Desktop -MemberDN $MemberDN -ConnectionServer $ConnectionServer

 If($Desktop."pae-SVIVmSnapshotMOID" -ne $pool.SVIVmSnapshotMOID)
  {
  $obj = New-Object PSObject -Property @{
           "PoolName" = $pool.DisplayName
     "DisplayName" = $Desktop."pae-DisplayName"
     "PoolSnapshot" = $pool.SVIVmSnapshot
     "PoolSVIVmSnapshotMOID" = $pool.SVIVmSnapshotMOID
           "DesktopSVIVmSnapshot" = $Desktop."pae-SVIVmSnapshot"
           "DesktopSVIVmSnapshotMOID" = $Desktop."pae-SVIVmSnapshotMOID"
     "DesktopDN" = $MemberDN
    }
$DesktopExceptions += $obj
  }
 }

}

If($DesktopExceptions -eq $null)
 {
 $SMTPBody = "All desktops are currently using the correct snapshots."
 }
Else
 {
 $SMTPBody = $DesktopExceptions | Select-Object DisplayName,PoolName,PoolSnapshot,DesktopSVIVmSnapshot | ConvertTo-HTML
 }

Send-Email -SMTPBody $SMTPBody -SMTPTo $EmailRcpt

If($Remediate -eq $true)
{
 ForEach($Exception in $DesktopExceptions)
 {
  Set-QADObject -Identity $Exception.DesktopDN -Service $ConnectionServer -IncludedProperties "pae-vmstate" -ObjectAttributes @{"pae-vmstate"="DELETING"}
 }

}

Utilizing Offsite Backups to Seed Backup Copy Jobs in @Veeam #V7

/ seanpmassey / 1 Comment

In the last couple posts about Veeam, I mentioned that $Work has been doing backups directly to our offsite storage.  Due to limits on bandwidth, any errors, changes, or server additions can have a drastic impact on our ability to complete backups in a timely manner.  And if you’ve ever tried to do a full backup of a 1.5TB file server over a 10Mbit connection because the VMID changed, you’ll know exactly what kind of pain I’ve felt in the past.

While a backup copy job will eliminate some of this pain, it still needs to be seeded at the remote site. 

I was a little disappointed to learn that an existing backup chain cannot be the target of a backup copy job.  The backup copy job needs to be pointed to a clean full backup that doesn’t include any forward or reverse incremental backups.  I’m not sure what the reason for this is, but it was confirmed in a Veeam in this thread on their support forums.

But there is a workaround to this, and it’s fairly easy.  The process is actually pretty simple.

  1. Create a backup copy job.  Use the backup job that currently saves to the remote site or the virtual machine as the source and use a backup repository at the remote site as the destination.
  2. Let the Backup Copy job run and create a new full backup of the file.
  3. Once the Backup Copy job has successfully completed (see notes below), create a new backup job for those servers and store that backup data at your local/primary site.  You cannot change the target of your existing backup job –  Veeam will require you to copy your existing backup chain to that location.  Its easier, and faster, to just create a new backup job.
  4. Edit your Backup Copy job to remove the job that backs up directly to the remote site and add the job that backs up to your primary site.
  5. The next time your VMs back up inside the copy window, it will sync the changes from the latest restore point to your remote site.

There are a couple of caveats with this.  You can’t set up a new backup chain at your primary site until you’ve created your backup copy job and created the new full backup file.  If there are more recent restore points available than the ones at your remote storage site, it will eschew the ones at the remote site in favor of the ones at your primary site.  This may mean copying a large amount of data over your WAN.

Second, you need to check to make sure that all of your data has copied over.  A copy job may end succcessfully if the interval expires and it copied some data.  If it hasn’t finished copying all of your data, it will restart and pick up where it left off.  This might give you a false sense of data security by making you think that your offsite backups have fully seeded when you still have to copy large amounts of data over your WAN.  In this case, it would be helpful to have a warning on any job notifications to inform the administrators that the seeding hasn’t completed and that there isn’t a restore point in the remote repository yet.