Tag Archives: Cisco UCS

Cisco UCS Manager Firmware Upgrade Procedure

I’ve been involved in many a Cisco UCS Manager Firmware Upgrade.   Cisco’s documentation if you don’t find the exact right page is confusing.  If you don’t elect to do an automatic installation, you need to do the components in the proper order.  Personally, I’ve had issues doing it with the automatic deployment.  I do it manually.  If you want to do it manually as well, here’s the correct order and some caveats.

Cisco UCS Manager Firmware Upgrade Procedure

  1. Verify the proper firmware, drivers, etc. for your hardware and the OS (whether it’s Windows, VMware, etc.) your servers run.
  2. Upload the Infrastructure Bundle into UCSM, so it’s ready to deploy.
  3. Determine the primary and subordinate Fabric Interconnect.  I like to SSH into UCSM, and run the following to give me that info plus general cluster health status before proceeding:
    show cluster extended-state
  4. Go to Admin > Communication Management > Call Home, and turn off call home.  You don’t want Cisco calling you thinking UCSM is on fire, when you’re doing an upgrade, right?
  5. Check alerts and verify the system is healthy before proceeding.  Fix anything that’s potentially a problem.
  6. Take a backup of your UCS configuration.
  7. Activate the new version of UCS Manager.  Verify it completes, and no unexpected errors result.  It is possible that sometimes errors are expected, and it’s OK to proceed.  Here’s an example.  Look them up!
  8. Update the firmware on the IO Modules by going to Equipment > Chassis > Chassis number > IO Modules > IO Module you want to upgrade > General > Update firmware.  Repeat on the second IO Module. You can track the progress on the Update Status portion of the general page.
  9. Activate the firmware on the IO Modules by going to Equipment > Chassis > Chassis number > IO Modules > IO Module you want to upgrade > General > Activate firmware.  Clear the checkbox for “Set Startup Version Only” to have the code change take effect immediately.  If you leave this option enabled, you’ll need to reboot the IOM Module yourself.  I recommend clearing it, and let UCSM reboot it for you. You may also receive the error: “Failed start activation.  Manual upgrade/activation is disallowed because the Default Infrastructure Policy ‘Startup Version’ is set.  Retry the operation after changing the version to ‘Not Set'”  Check out this post for the solution.
  10. Activate Firmware on subordinate Fabric Interconnect by going to Equipment > Installed Firmware.  Right click the subordinate FI, select Activate Firmware, and select the new firmware package.  Verify when the FI comes back up it is running the proper new version, and that your network and storage redundancy is working properly.
  11. Failover the UCSM cluster by connecting to UCSM via SSH, and run the following:
    connect local-mgmt
    cluster lead b
  12. Active the firmware on the formerly primary FI, which is now the subordinate by repeating the above, but do the other FI this time.  Verify it’s running the proper new version, and your network and storage redundancy is working properly.
  13. Validate network connectivity and storage multipathing.
  14. Turn back on CallHome.
  15. Take a backup of the final configuration.

That’s how to do a manual Cisco UCS Manager Firmware Upgrade.

Unregister Cisco UCS from Cisco UCS Central if you’re not using it

Just banged my head against a wall for hours trying to get a new Service Profile associated with a new blade.  It kept retrying to “Configure resolve identifiers”.  Finally stumbled on an obscure support forum article just when I was about to give up and call Cisco TAC…

https://supportforums.cisco.com/discussion/12174866/cannot-create-service-profiles-template-anymore

TLDR version…

If the UCS domain (FIs) have been registered with UCS Central, you can’t associate service profiles with UCS Manager.  If it looks like this in UCS Manager with Registration Status showing Lost Visibility…

unregisterucscentral

Your UCS Domain is either having problems connecting to UCS Central, or UCS Central is burnt… or dead…  Troubleshoot why it has lost connectivity, or if UCS Central is gone, click the red circled Unregister From UCS Central, and proceed.

If you see a nice green check box, that means UCS Central is present, and you should use it to associate the Service Profile, not UCS Manager.

In this case, the customer deployed UCS Central to play with, decided they didn’t need it, and removed it from their environment, but kept it registered.  So I unregistered it, and the Service Profile associated nice and easy.

Deploy Cisco UCS C-Series Servers via Serial

One thing I really like about Cisco UCS C-Series servers is the CIMC card.  Without paying extra, you get effectively a fully featured out of band management card, all the way down to the integrated KVM within the CIMC web interface.  Cool, right?  Technically, you arguably don’t need a KVM in your server rack if you have these servers.  Just get the CIMC card on the network, and you’re golden!

But what about initial deployments?  You gotta get that CIMC card on the network.

During a deployment today with a customer in a branch site of Cisco UCS C-Series standalone servers, I ran into an interesting scenario.  The customer did not have a keyboard and mouse at the site (it was still under construction).  This was partly because they opted not to purchase a KVM because the CIMC cards in the servers.

Did you know you can configure the BIOS settings and CIMC card via a Serial connection?

Yep, just plug your Cisco console cable into the back console port, or adapt it to serial and connect it into the front VGA/USB/Serial dongle, boot up the server, and enter the keystroke to go into the BIOS or CIMC configuration setup when you see it.

ucs-c-series-serial

The only notable thing is to set your terminal app to 115200 for baud rate, but the rest of the settings are typical for Cisco gear.

So, yeah, with my trusty Air Console serial device (more on that later), I am able to configure the BIOS or CIMC card with my iPhone or iPad.  Look ma, no keyboards or monitors!

Getting Cisco UCS drivers right with Windows

I’ve already mentioned one pain point with Cisco UCS – drivers – in my previous post concerning vSphere, but the same thing applies to other environments, including Windows servers. You better have the EXACT version Cisco wants for your specific environment. But how do you know which drivers to get, how do you get them, how do you know when you need to upgrade them, and how do you know what drivers you have installed? This post applies to Windows Server, which by extension, includes Hyper-V.

Why is getting the drivers so important?

I want to emphasize that getting the exact right version of Cisco UCS drivers is a big deal! I’ve personally now seen two vSphere environments that had issues because the drivers were not exactly correct. The best part is the issues never turned up during a testing of the environment. Just weird intermittent issues like bad performance, or VMs needing consolidation after backups, or a VM hangs out of nowhere a week or two down the road. Make sure you get the drivers exactly right!  I don’t work with Windows Servers on bare metal nearly as much as VMware, but I’m sure getting the drivers right is equally, if not more, crucial.

How do I install Windows Server 2012 on Cisco UCS?

You have two choices.  One is create a Windows Server installation ISO with the drivers slipstreamed into it, or you can insert the driver image during the routine to install the proper storage driver to see your storage to which you’ll install Windows, which is available from Cisco for download. Also, at least currently, remember that Cisco UCS does not support Windows in a boot from SAN configuration using FCoE.

Remember however you’re still not done.  You’ll need to still update the network card drivers.

How do I know which drivers should be installed?

This is relatively simple. First, collect some info about your Cisco UCS environment. You need to know these (don’t worry, if you don’t know what info you need, Cisco’s Interoperability page will walk you through it):
1. Standalone C-Series not managed by UCSM or UCSM managed B and/or C-Series? For those of you who don’t know, if you got blades, you got UCSM.
2. If UCSM is present, which version is it running? Ex. 2.2(3c)
3. Which server model(s) are present? Ex. B200-M3. Also note the processor type (ex. Xeon E5-2600-v2). They can get picky about that.
4. What OS and major version? Note there is a difference between support for Server 2012 and 2012 R2.  Cisco at least at the time of this blog post does not change support depending upon installed Service Packs.
5. What type and model of I/O cards do you have in your servers? Example – CNA, model VIC-1240

Then head on over to the Interoperability Matrix site.  Fill in your info, and you get a clear version of the driver and firmware.

ucswindriverlookup

It’s very straightforward to know which drivers are needed from that.

How do I figure out which drivers are installed?

 

You can do this one of two ways.  You can manually check them via Device Manager, or you can use PowerShell.  I’m assuming everyone knows how to check these with Device Manager.  To use PowerShell, use the following:

Get-WmiObject Win32_PnPSignedDriver | select-object devicename,driverversion

 

Note that you can use the -ComputerName parameter to check a remote system, or even a PowerShell array of remote systems for their drivers easily.

How do I update Cisco UCS drivers?

You need to go to Cisco’s support page for UCS downloads, and download the driver ISO that has your driver, which is typically just the driver ISO with the same version as UCS Manager.  For example, if you’re running 2.2.5(a), you need the 2.2.5 driver ISO.

Next, use the virtual media option within the Cisco KVM, mount the ISO, so it’s ready to go.  You could also extract the contents somewhere on the server, it doesn’t matter.

Next, login to the server, pull up Device Manager, and locate the adapter instances you need to update.  If you’re using a VIC, which is pretty much everyone, you need to find the relevant storage and network adapters.  This example, the customer wasn’t using the FC functionality, so we’re just doing the ENIC devices.

updatingwinucsdrivers

When the dialogue comes up, browse to select the zip file that was *contained* in the original zip file. If you select just the zip file you downloaded itself, it will fail. Repeat for the fnic and enic drivers.

Double click one of the VIC instances, click the Driver tab, note the currently installed driver version, and then Update Driver…

Next, click Browse my computer for driver software.

Next, click “Let me pick from a list of device drivers on my computer”.  Don’t even bother trying any other options, it will continue to want to install the old driver because… REASONS!!!

Next, click Have Disk… and browse on the CD to the right folder for the hardware and OS you’re running.  This is a Server 2012 non-R2 server, so as an example, it’s under Windows\Network\Cisco\VIC\W2K12\x64.

After the driver installs, verify the new driver version is now showing in Device Manager.  Unfortunately, you’re not done yet.  You gotta update the drivers on every other instance, but it’s a little easier for the rest.

For the other instances, double click each one, click Update Drivers…, “Browse my computer for driver software”, “Let me pick from a list of drive drivers on my computer”, and you will see both the old and new driver versions.  Pick the new one, and click Next.

updatingwinucsdrivers2

Rinse and repeat for all instances, and ensure the driver tab reflects the proper new version.  Remember that FNIC HBA drivers will need to be updated on those instances separately under Storage Controllers.

I took a quick look to see if Microsoft made some device driver PowerShell cmdlets, but unfortunately I don’t see any at this time.

When should I check my drivers?

You should do this during any of the following:

• During initial deployments
• When UCS Manager or a standalone C-Series BIOS is updated
• If you update to a new major OS, although I’d check when planning to install Service Packs to Windows as well.

Also, remember, newer drivers aren’t the right drivers necessarily. Check the matrix for what the customer is or will be running to see which drivers should go along with it!

Getting Cisco UCS drivers right with vSphere

I’ve noticed one pain point with Cisco UCS – drivers. You better have the EXACT version Cisco wants for your specific environment. But how do you know which drivers to get, how do you get them, how do you know when you need to upgrade them, and how do you know what drivers you have installed? These are all not necessarily straightforward, and getting the info you need can be a real pain.  This post will show how to accomplish this within vSphere.  For Windows servers, please see my follow-up post due out in a few days.

Why is getting the drivers so important?

I want to emphasize that getting the exact right version of Cisco UCS drivers is a big deal! I’ve personally now seen two environments that had issues because the drivers were not exactly correct. The best part is the issues never turned up during a testing of the environment. Just weird intermittent issues like bad performance, or VMs needing consolidation after backups, or a VM hangs out of nowhere a week or two down the road. Make sure you get the drivers exactly right!

How do I install ESXi on Cisco UCS?

First off, pretty much everyone knows that when you’re installing ESXi on Cisco, HP, Dell, IBM, or other vendor servers, use the vendor’s media. That’s common practice I hope by now. In most but not all cases, you get the drivers you need for an initial deployment from the get go, you get hardware health info within VMware, sometimes management and monitoring tasks for out of band management cards, and you ensure vendor support by doing this. We all know I think by now to do initial ESXi installs with vendor media, in this case Cisco. It’s important for Cisco UCS since so many installs require boot from SAN, that you gotta have those drivers within the media off the bat.

Now, if you think you’re done if you’ve downloaded the latest Cisco co-branded ESXi media for an initial deployment, you’re wrong (see below). Also, don’t assume that just because you use the co-branded media to install ESXi on a UCS server, you never need driver updates. You will likely when you update UCS Manager and/or update ESXi down the road.

How do I know which drivers should be installed?

This is relatively simple. First, collect some info about your Cisco UCS environment. You need to know these (don’t worry, if you don’t know what info you need, Cisco’s Interoperability page will walk you through it):
1. Standalone C-Series not managed by UCSM or UCSM managed B and/or C-Series? For those of you who don’t know, if you got blades, you got UCSM.
2. If UCSM is present, which version is it running? Ex. 2.2(3c)
3. Which server model(s) are present? Ex. B200-M3. Also note the processor type (ex. Xeon E5-2600-v2). They can get picky about that.
4. What OS and major version? Note the Update number. Ex. ESXi 5.5 Update 2
5. What type and model of I/O cards do you have in your servers? Example – CNA, model VIC-1240

Then head on over to the Interoperability Matrix site.  Fill in your info, and you get a clear version of the driver and firmware.

ucsdriverlookup

It’s very straightforward to know which drivers are needed from that.

How do I figure out which drivers are installed?

If you go looking at Cisco for how to find that out, you get treated to esxcli commands.  Do you really want to enable SSH on all your hosts, SSH into each host, run some commands, then have to disable SSH on all those boxes when you’re done, and not have an easy way to document what they are?  Nope!

BEHOLD! POWERCLI!

To get the fnic driver versions for all ESXi hosts:

$hosts = Get-VMHost
$versions = @{}
Foreach($vihost in $hosts){
$esxcli = Get-VMHost $vihost | Get-EsxCli
$versions.Add($vihost, ($esxcli.system.module.get("fnic") |
Select Version))
}
$versions.GetEnumerator() | Sort Name | Format-List 

You get this:

Name : esxi01.vspheredomain.local
Value : @{Version=Version 1.6.0.12, Build: 1331820, Interface: 9.2 Built on: Jun 12 2014}

Hey! That’s the wrong driver, even though I used the latest co-branded media! SON OF A…!

Let’s get some enic driver versions…

$hosts = Get-VMHost
$versions = @{}
Foreach($vihost in $hosts){
$esxcli = Get-VMHost $vihost | Get-EsxCli
$versions.Add($vihost, ($esxcli.system.module.get("enic") |
Select Version))
}
$versions.GetEnumerator() | Sort-Object Name | Format-List 

You get:

Name : esxi01.vspheredomain.local
Value : @{Version=Version 2.1.2.59, Build: 1331820, Interface: 9.2 Built on: Aug 5 2014}

Of course, Cisco apparently didn’t update those drivers in their co-branded media either.

Note for both scripts, you will get errors about get-esxcli not being supported without being connected directly to each host. It works for our purposes.

How do I update Cisco UCS drivers?

Now we know, despite using the latest Cisco co-branded media in my implementation, I need some driver updates. If you go to Cisco’s site for how to install these drivers, they’ll tell you to upload the package to each host and install them one at a time manually using esxcli commands. Do you really want to do that?

Let’s be smart/lazy/efficient and use VMware Update Manager. That way if a new host gets introduced, VUM will report that host non-compliant, and it’ll be easy to fix that one, too. And it’s easy to see which hosts do and don’t have those drivers down the road.

I find if I google the driver version, I find a download from VMware’s site with that exact version first or second link. Here’s our fnic driver and enic driver in this case.

Download those to your vCenter server or something with the vSphere thick client. Unzip them into their own folders. Open up a thick vSphere client connection to vCenter (Web Client won’t allow you to do this), click Home, then click Update Manager.

Next, click Patch Repository tab at the top, and then click Import Patches in the top right.

vumimportpatches

When the dialogue comes up, browse to select the zip file that was *contained* in the original zip file. If you select just the zip file you downloaded itself, it will fail. Repeat for the fnic and enic drivers.

When you’re finished, you can then build a baseline that includes the updated drivers. Click Baselines and Groups, then Create above the baselines pane.

vumcreatebaseline

Call it something like “Cisco UCS Current Drivers”.  Select “Host Extension” as a Host Baseline type.  In the following pain, find the drivers and click the down arrow to add them into the baseline.  Note the Patch ID field has driver version specifics, useful if you’ve already got some Cisco drivers imported before.

vumselectpatches

You can then attach that baseline directly to the appropriate object(s) within the host and clusters view, or I like to make a Baseline Group called “Critical and non-critical patches with Cisco updated drivers”, add all the appropriate baselines to that group, and attach that group to the appropriate objects in the Hosts and Clusters view.

Then remediate your hosts. When new drivers come out, import them in, then edit the Cisco baseline, swapping out the last updated drivers with the new ones, and remediate to push them out.

Done!

When should I check my drivers?

You should do this during any of the following:

• During initial deployments
• When UCS Manager or a standalone C-Series BIOS is updated
• Major ESXi version upgrades
• Update pack upgrades for ESXi (when ESXi 5.5 servers for example are to be upgraded to Update 2, or 3, etc)

Also, remember, newer drivers aren’t the right drivers necessarily. Check the matrix for what the customer is or will be running to see which drivers should go along with it!