July 2013

Think Twice Before Upgrading Core Firmware

Think Twice Before Upgrading Core Firmware

When it comes to vital data center infrastructure, the call for the latest and greatest firmware can end up being a siren song

By Paul Venezia


Hardware vendors say it all the time: Your firmware is out of date and should be upgraded.

I hear this almost every time I’m working on a hardware support contract. Disk failures on storage, part failures on modular switches — it doesn’t matter; the firmware issue almost always pops up. And I play along.

“Because there are tons of improvements and bug fixes in the newer version,” they all say when asked why I should upgrade. But unless I’m experiencing a problem that can be directly traced to firmware, I’ll usually decline. I’ve been bitten more than once by the siren song of the latest firmware. I’ll take stability over recency nearly every time.

The high price of firmware upgrades
Maintenance windows and downtime are significant factors when it comes to updating firmware on core devices. Unless your storage supports online firmware updates (as some do), you’re looking at a reboot of a storage array at best and longer downtime at worst. Most small organizations can’t afford hardware that supports online updates, so the entire virtualization infrastructure must be powered down or migrated just to upgrade the firmware of one storage array. Unless your current firmware is causing data integrity issues, it isn’t worth having to do either.

That’s nothing to say of the possibility that a “minor” firmware update can cause unforeseen problems with the device, rendering it inoperable or otherwise offline. Now you’re left with a situation of your own causing that has suddenly become a far bigger problem. That simple firmware update, which has a changelog that contains next to nothing pertinent for your infrastructure, has hit a bug and instead of rebooting normally, your device is now stuck at boot throwing errors.

Ideally there’s a backup image that can be loaded to get the device back online, but that’s not always a guarantee. Some vendors seem to think it’s OK to make such substantial changes to their firmware from release to release that backing out to a previous known-good version is not possible once the upgrade has reached a certain point. In some instances, this is unavoidable; still, hardware vendors should provide intermediate code to bridge the gap and allow for recovery in the event of a failed or aborted upgrade. Some vendors do this; many do not.

Because of this, you must fully inspect the release notes and changelogs of any firmware you intend to upgrade. If you come across any element of an upgrade that isn’t fully described or doesn’t quite make sense, research it and leverage your support contract to verify you are covered before starting the procedure.

The right way to upgrade
This is not to say that upgrading firmware is a bad idea — it’s not. But blithely upgrading firmware without a clear and useful reason to do so can easily cause more problems than it’s worth. Sure, it’s nice to see that you’re running the latest and greatest revision, but at least in your infrastructure, it’s essentially untested code and should be treated as such until it puts in enough miles to become trusted.

Another problem caused by not upgrading firmware: Eventually you get so far behind the current revision that you need to perform several upgrades in a row to get to the version addressing a new issue or providing a newly required feature. Yes, this is a pain and generally requires several reboots, but at least you get it out of the way all at once. On such occasions, it’s not a bad idea to have a support tech on the line, especially during touchy upgrades. No matter how much time you invest searching a vendor’s knowledgebase or related forums, there’s often an internal solution not published anywhere. I ran into one such issue recently when a certain virtualization vendor’s own scripts (taken directly from their knowledge base) were causing more problems than they were solving. A call to support revealed the right script — a script that appears nowhere on the Internet, as far as I can tell.

In the end, when it comes to firmware, the best approach is to split the middle. Be cognizant of the need to update firmware from time to time, but do not think that it’s a hard and fast requirement. In the face of functionality problems, security risks, or the need for specific features, do the research, plan your recovery strategy, and perform the upgrade.

Lacking those elements, there’s no need to be quite so proactive. It’s sometimes best to let sleeping dogs lie.

This story, “Think twice before upgrading core firmware,” was originally published at InfoWorld.com.