Book Review–The Phoenix Project

Rating – Highly Recommend

When I sat down to start writing this review, it was almost 11:30 PM on Friday night.  For some reason, I’m wired even though I’m running on about five hours of sleep total.  My mind is racing, and I can’t get it to settle down.  So, I’ve decided to try writing my first book review instead.

I’ve just finished reading The Phoenix Project (Amazon)(Barnes and Noble) – a book about how IT is intertwined with the business and is now a critical role in it’s success or failure.  Modern businesses live and die by the information that is contained in and produced by those system.  If they don’t work, the business doesn’t work either.

The book follows Bill, a mid-level IT manager who gets gets a surprise promotion to acting VP of IT Operations one morning and has to take charge of an IT Operations group that is overworked to the point of being dysfunctional.  The group is sorely lacking in basic IT practices like adequate change control procedures that cause numerous severity 1 outages that bring the business to its knees, poor relationships with other departments, no budget, and most of the knowledge living inside the head of the senior systems architect.  At the start, Bill has to scrape by with what he has.

Bill is accompanied on this journey by Patty and Wes, the two department managers within IT Operations who act as Bill’s superego and id respectively; Brent, the systems architect who is the cause of, and solution to, many of the problems they face early on; John, the CISO who is loathed by everyone around him; and Steve, the CEO of “Parts Unlimited” who promoted him. 

Other key players include members of the senior management staff such as Chris, the acting CIO and head of Development, Dick, the CFO/COO, and Sarah, the …um…ambitious and troublemaking VP of Retail Operations and the driving force behind Project Phoenix, the expensive and expansive IT project that “Parts Unlimited” has staked its future on.

The story wouldn’t be complete without the enigmatic Eric, a prospective board member who acts as an eccentric mentor to Bill, John, and others at Parts Unlimited.  It is through Eric that Bill is introduced to many of the bits and pieces that will transform him and the Parts Unlimited IT Team.  In some senses, he’s the Yoda or Dumbledore to Bill’s Luke Skywalker or Harry Potter by not always giving the answers or answering with another question.  Eric also comes off as an author avatar at times to discuss the concepts that the book is trying to teach.  This isn’t necessarily a bad thing, though, because they allow Bill to reach many of the conclusions himself.

Overall, the characters are flat.  I’m not sure if they were intentionally designed to be that way to act as self-inserts or if it is because the authors are trying to present a management text as a work of fiction.  Either way, its not too big of a detriment to the concepts that the book is trying to teach.

There are a number of factors that come off a little too unrealistic for my tastes – the changes happen very quickly and without much resistance, or everyone comes around in the end to that way of doing things.  Or Wes falling in line under Bill rather quickly.  But like I said with the characterizations, this isn’t critical to what the book is trying to accomplish.

Where The Phoenix Project excels is breaking down the ideas behind lean operations such as kanban and the Toyota Production System and presenting them to technical people who might not have been exposed to them.  It also does a good job avoiding overly technical jargon and of comparing IT operations to that of a factory floor in order to make it accessible to non-technical managers who want, or need, to know about IT and how it relates to and integrates with the operation of a business.

It’s also spurred my interest in learning more about some of the lean manufacturing and continuous improvement process techniques.  I’m not the management type – I see myself as more of a systems architect – but I can see how techniques that are used on a production floor can be applied to improving IT operations – even if it is something as simple as automating routine operations like account creation or virtual machine deployments.  It also gives me an excuse to dig out my copy of Dr. Eliyahu Goldratt’s The Goal that I received from a former boss of mine as it is heavily referenced in the book.

The Phoenix Project would have been a great book for my boss at my last job.  She had been hired to oversee one department and had IT tacked onto it as one area of responsibility.  This would have given her a management-level perspective of what IT is about and would have better prepared her for managing that group without bogging her down in technical minutia.  This isn’t meant as a slam against her – in the year I worked under her, she learned a lot about IT and was able to instill a customer-oriented focus in the organization, but this type of resource may have better prepared her for the challenges that overseeing an IT Department presented and helped us better address the bottlenecks that we had as a department as a result of the focus change.

Provided that this book is approached as a management text disguised as a novel, I highly recommend this book to anyone who is in IT, especially if they aspire to a management or senior technical role.  I also think this book would be a good read for non-technical senior management.

Where I Go Spelunking into the Horizon View LDAP Database–Part 1

Note: The items and techniques discussed in this post are not supported by VMware.  Before using it in a production VMware View environment, please be sure to test it in a non-production environment. 

Like many of the other product offerings, VMware includes a set of PowerCLI cmdlets that can be used with VMware View.  However, unlike the other PowerCLI cmdlets, there are some significant limits to what can be done with these cmdlets, including:

  • They can only be used on the View Connection Server*
  • They are feature-limited
  • They haven’t been upgraded with the rest of View, so features like View Storage Accelerator/Content-Based Read Cache and VMware Blast cannot be managed from PowerShell

*Note: Scripts utilizing the View PowerCLI commands can be executed via PowerShell Remoting or tools like PSEXEC.  However, the script still needs to be locally stored on the Connection Server.

But there is still one option for working with View.  Although the API for View isn’t as robust as other mainstream VMware products, the primary database is based is LDAP.  This opens up a whole new world of possibilities when trying to automate against VMware View.

Exploring the View LDAP Database

VMware doesn’t provide a lot of information about the View LDAP database in their support documents, and in almost all cases, directly editing entries in LDAP is not supported.  But there is information out there.

A few months ago, I posted a script that worked directly against the View LDAP database to find and remediate desktops that were using the wrong snapshot as a base image.  Other bloggers and community members have put together some scripts that work directly against the LDAP database.  Some of those resources are:

The link to Luc Dekens’ blog contained a very surprising revelation – VMware includes an entire copy of the schema on the Connection Servers.  This can provide a good starting point for working with View’s LDAP database as it does include a description of what each field does.  Some of the descriptions also include accepted and/or default values.

With a default install of VIew, the schema file is located at %ProgramFiles%\VMware\VMware View\Server\LDAP\ldif, and it can be opened with any text editor program.

The existence of this file has already saved me a lot of time.  My initial plan was to actually dive in and map attributes to feature sets by trial-and-error.  I may have to do that for some items, such as the pae-CitrixXMLServicePort attribute, but this provides a rough map of the LDAP layout and what the different attributes control.

This only scratches the surface of what can be done with the LDAP database.  And like a large cave system that has multiple branches and tunnels, the View LDAP database has several containers, entities, and attributes that need to be explored.

What I Accomplished On My Week Off.

In 1785, the Scottish Poet Robert Burns, in his poem “To A Mouse,” wrote:
“The best laid schemes of mice and men
Go often awry,”

That sums up how my week went.

Last week, I had the pleasure of taking a week off to de-stress and recuperate before I move into a new role. My plan was to make the best use of that time in my home lab making updates, diving deep into VMware View, and testing out some new items that caught my eye at VMworld. Things obviously didn’t go quite the way I had hoped.

But in the process, I’ve put my home lab on a more solid footing and even added new capabilities that I wouldn’t have otherwise had.

The problems

The problems I faced in my home lab centered entirely around storage. As part of my lab upgrade process, I planned to upgrade Nexenta to the latest version and add additional solid state drives to the HP xW4400 . I was able to add the drives and do the upgrade, but the ZFS pool that I added them to showed up as being in a degraded state with an error on the SSD drives. After doing a little digging, I realized that this issue was because that machine wasn’t set to use ACHI in the BIOS.

Changing the BIOS on the storage box would require a reinstall, and this is where my problems truely began. When I reinstall Nexenta using the latest version, it started to experience boot loops. It would boot to where the kernel would load, display an error, and then quickly reboot. Reverting the BIOS settings did not resolve the issue, and I was able to boot FreeNAS, a FreeBSD-based appliance OS with ZFS support, from a USB drive.

I figured that there was some odd hardware driver issue that was causing the error. Nexenta is an OS derived from OpenSolaris and it’s descendant Illumos, and while it has fairly broad hardware support, it isn’t as all-encompassing as Windows or Linux, so I decided to try a different machine.

The old lab server in my environment was a Dell PowerEdge T310. This server had been set aside to host a few services like my domain controller after I purchased a used PowerEdge T110 Mk II. Since I wasn’t using the full capabilites of this box, I figured I would move Nexenta and my ZFS pools over to it.

I was surprised when I started experiencing the same issues on this server as I did on my HP workstation. The server would start to boot, display an error, and then quickly reboot after installing the latest version. To make matters worse, the Nexenta Community forums were down for maintenance, which made troubleshooting this issue harder.

After doing a little Googling, I was able to find some information on how to halt the system on an error instead of rebooting. And I was surprised to see that the problem that I was having was caused by my USB keyboard. I switched to a different USB keyboard – same issue. Booted without a keyboard and an attached USB hard drive caused the same error.

Something in the Nexenta USB driver stack was causing the issue, and without the community support forums, I wouldn’t be able to dig deeper into it. I’m not a Unix and Linux expert. That area is way outside of my comfort zone, and I wanted to get it running sooner rather than later.

After digging through my stack of unlabeled CDs, I was able to find a Nexenta disk for an earlier version of 3.1. I was able to install this without any issue and then upgrade to the latest version. This time, everything booted without issue.

Improvements to my lab

Despite the issues with the storage that took most of my week to resolve, I was able to make several significant improvements to my home lab. Those improvements were:

  1. Adding additional gigbabit ports with a 48-port gigabit switch
  2. Improving the overall performance of my storage server by moving it to from a dual-core machine with 2GB of RAM to a quad-core machine with 8GB of RAM
  3. Added two SSDs to my ESXi box
  4. Added an additional SSD to my ZFS pool
  5. Added Fibre Channel between my ESXi box and my Nexenta box

I start a new job this week, and any posts that I would have scheduled are backlogged right now due to the issues I faced. But I have been digging into a new VMware Fling called Web Commander which allows PowerShell scripts to be run from a web interface, and I hope to start having some things ready by the end of the week.