Monthly Archives: August 2016

Cisco UCS Manager Firmware Upgrade Procedure

Cisco UCS

I’ve been involved in many a Cisco UCS Manager Firmware Upgrade.   Cisco’s documentation if you don’t find the exact right page is confusing.  If you don’t elect to do an automatic installation, you need to do the components in the proper order.  Personally, I’ve had issues doing it with the automatic deployment.  I do it manually.  If you want to do it manually as well, here’s the correct order and some caveats.

Cisco UCS Manager Firmware Upgrade Procedure

  1. Verify the proper firmware, drivers, etc. for your hardware and the OS (whether it’s Windows, VMware, etc.) your servers run.
  2. Upload the Infrastructure Bundle into UCSM, so it’s ready to deploy.
  3. Determine the primary and subordinate Fabric Interconnect.  I like to SSH into UCSM, and run the following to give me that info plus general cluster health status before proceeding: 
    show cluster extended-state
  4. Go to Admin > Communication Management > Call Home, and turn off call home.  You don’t want Cisco calling you thinking UCSM is on fire, when you’re doing an upgrade, right?
  5. Check alerts and verify the system is healthy before proceeding.  Fix anything that’s potentially a problem.
  6. Take a backup of your UCS configuration.
  7. Activate the new version of UCS Manager.  Verify it completes, and no unexpected errors result.  It is possible that sometimes errors are expected, and it’s OK to proceed.  Here’s an example.  Look them up!
  8. Update the firmware on the IO Modules by going to Equipment > Chassis > Chassis number > IO Modules > IO Module you want to upgrade > General > Update firmware.  Repeat on the second IO Module. You can track the progress on the Update Status portion of the general page.
  9. Activate the firmware on the IO Modules by going to Equipment > Chassis > Chassis number > IO Modules > IO Module you want to upgrade > General > Activate firmware.  Clear the checkbox for “Set Startup Version Only” to have the code change take effect immediately.  If you leave this option enabled, you’ll need to reboot the IOM Module yourself.  I recommend clearing it, and let UCSM reboot it for you. You may also receive the error: “Failed start activation.  Manual upgrade/activation is disallowed because the Default Infrastructure Policy ‘Startup Version’ is set.  Retry the operation after changing the version to ‘Not Set'”  Check out this post for the solution.
  10. Activate Firmware on subordinate Fabric Interconnect by going to Equipment > Installed Firmware.  Right click the subordinate FI, select Activate Firmware, and select the new firmware package.  Verify when the FI comes back up it is running the proper new version, and that your network and storage redundancy is working properly.
  11. Failover the UCSM cluster by connecting to UCSM via SSH, and run the following: 
    connect local-mgmt
cluster lead b
  12. Active the firmware on the formerly primary FI, which is now the subordinate by repeating the above, but do the other FI this time.  Verify it’s running the proper new version, and your network and storage redundancy is working properly.
  13. Validate network connectivity and storage multipathing.
  14. Turn back on CallHome.
  15. Take a backup of the final configuration.

That’s how to do a manual Cisco UCS Manager Firmware Upgrade.

vmware workstation

vCenter 6 VCSA External PSC in VMware Workstation

vCenter, vSphere, ,

I’ve been doing a lot of various oddball testing with vCenter for various scenarios, which have required me to deploy more complex configurations with vCenter 6 recently in my lab.  I found very quickly that there isn’t good consolidated documentation on how to do more advanced vCenter deployments other than directly to ESXi hosts.  It took me quite a bit of time to figure this all out.  I wanted to share this with anyone else who may be doing similar test.  Here’s how to deploy vCenter 6 VCSA with External PSC in VMware Workstation.

And I mean this to be a “proper” lab you can really test vCenter stuff in.  No plain IP addresses for host names!  We want proper FQDNs and what not here!

I am assuming you already have the following up and functioning, along with the following information:

  • DNS server
  • Proper networking for VMware workstation suitable for whatever you’re going to do
  • Document what you will want your host names, IP addresses, DNS IP(s), default gateway IP, and Single Sign-On site names in advance.  Fair warning: this all can get very confusing!  Don’t introduce confusion by deciding these things on the fly.  We’re going to use the following for this article:
    • 1st PSC – vcenter6-2-psc1.vs6lab.local, 192.168.1.61
    • 2nd PSC – vcenter6-2-psc2.vs6lab.local, 192.168.1.62
    • vCenter – vcenter6-2.vs6lab.local, 192.168.1.60
    • DNS server: 192.168.1.80
    • Default Gateway: 192.168.1.1

Deploy vCenter 6 VCSA External PSC in VMware Workstation – Preparation

Here are the things you should get out of the way first:

  1. Download VCSA 6 from VMware if you haven’t already done so.
  2. Extract the VCSA download package to a temporary directory.  For simplicity’s sake, we will assume you extracted the download to c:\VCSA.  Rename the c:\VCSA\vcsa\vmware-vcsa file with an OVA file extension.
  3. Create A AND PTR records for all PSC and vCenter nodes within your lab’s DNS server.

Deploy vCenter 6 VCSA 1st External PSC in VMware Workstation

In order to begin your lab deployment to have an external PSC in VMware Workstation, you must deploy the first PSC.

To deploy the first external PSC in VMware Workstation, do the following:

  1. Double click on c:\VCSA\vcsa\vmware-vcsa.ova
  2. Provide the name for the new virtual machine.  I’m calling mine vcenter6-2-psc1. Also provide the storage location for the virtual machine.
  3. After importing is completed, open the virtual machine’s VMX file before you power the VM up.  You need to add the following lines to the VMX file, adjusting values as needed: 
    guestinfo.cis.appliance.net.addr.family = "ipv4"
guestinfo.cis.appliance.net.mode = "static"
guestinfo.cis.appliance.net.addr = "192.168.1.61"
guestinfo.cis.appliance.net.prefix = "24"
guestinfo.cis.appliance.net.gateway = "192.168.1.1"
guestinfo.cis.appliance.net.dns.servers = "192.168.1.80"
guestinfo.cis.system.vm0.hostname = "vcenter6-2-psc1.vs6lab.local"
guestinfo.cis.vmdir.password = "P@ssw0rd"
guestinfo.cis.appliance.root.passwd = "P@ssw0rd"
guestinfo.cis.deployment.node.type = "infrastructure"
guestinfo.cis.vmdir.first-instance = "true"
  4. Ensure that you created both the A and PTR records for this appliance.  If you didn’t create them correctly, the remaining steps are a waste of time, as you’ll have to redeploy the appliance.
  5. Power the virtual machine on.  If you get error messages that the VMX file is corrupt, the above lines likely did not get added properly within the VMX file.  If you copied above from my web page, try retyping it all out in notepad.  Sometimes HTML invisible formating gets copied and pasted that you’re not aware of.  Allow the machine to complete its initialization.
  6. Verify it has completed properly.  To do this, you can open the VM’s console window, verify that it shows the correct name and IP address, and does not show any error messages that say firstboot failed.  If you see this error, you likely did not put in the proper information above, and/or the DNS A and PTR records were not properly created, or a similar issue with name resolution.  Also, you can go to https://vcenter6-2-psc1.vs6lab.local and verify the web page comes up, telling you to sign into a vCenter Management server to manage the PSC.

We’re assuming this completed successfully at this point.  If you encountered issues, correct this before proceeding.

Deploy vCenter 6 VCSA Non-Embedded Server in VMware Workstation

After you deploy the first external PSC in VMware Workstation, you need to deploy the vCenter server itself.

By default, deploying a vCenter 6 server appliance will automatically default to embedded within Workstation.  The line guestinfo.cis.deployment.node.type within the VMX file controls the node type.  As you saw above, setting it to “infrastructure” makes the VCSA instance a Platform Services Controller (PSC).  Let’s make a vCenter server!

To deploy a vCenter Server leveraging the external PSC in VMware Workstation, do the following:

  1. Double click on c:\VCSA\vcsa\vmware-vcsa.ova
  2. Provide the name for the new virtual machine.  I’m calling mine vcenter6-2. Also provide the storage location for the virtual machine.
  3. After importing is completed, open the virtual machine’s VMX file before you power the VM up.  You need to add the following lines to the VMX file, adjusting values as needed: 
    guestinfo.cis.appliance.net.addr.family = "ipv4"
guestinfo.cis.appliance.net.mode = "static"
guestinfo.cis.appliance.net.addr = "192.168.1.60"
guestinfo.cis.appliance.net.pnid = "vcenter6-2.vs6lab.local"
guestinfo.cis.appliance.net.prefix = "24"
guestinfo.cis.appliance.net.gateway = "192.168.1.1"
guestinfo.cis.appliance.net.dns.servers = "192.168.1.80"
guestinfo.cis.system.vm0.hostname = "vcenter6-2-psc1.vs6lab.local"
guestinfo.cis.vmdir.password = "P@ssw0rd"
guestinfo.cis.appliance.root.passwd = "P@ssw0rd"
guestinfo.cis.deployment.node.type = "management"
guestinfo.cis.vmdir.domain-name = "vsphere.local"
guestinfo.cis.vmdir.site-name = "default-first-site"
  4. Ensure that you created both the A and PTR records for this appliance.  If you didn’t create them correctly, the remaining steps are a waste of time, as you’ll have to redeploy the appliance.
  5. Power the virtual machine on.  If you get error messages that the VMX file is corrupt, the above lines likely did not get added properly within the VMX file.  If you copied above from my web page, try retyping it all out in notepad, as sometimes HTML invisible formating gets copied and pasted that you’re not aware of.  Allow the machine to complete its initialization.  Also, note that the vSphere Web Client takes a long time to initialize.  Be patient!
  6. Verify it has completed properly.  To do this, you can open the VM’s console window, verify that it shows the correct name and IP address, and does not show any error messages that say firstboot failed.  If you see this error, you likely did not put in the proper information above, and/or the DNS A and PTR records were not properly created, or a similar issue with name resolution.  Also, you can go to https://vcenter6-2.vs6lab.local and login to the vSphere Web Client.  Ensure you can access the administration and inventory sections of the Web Client.  Ensure both the vCenter appliances show up under Administration as healthy.vcenter using external PSC in VMware Workstation check1st external PSC in VMware Workstation check

Still with me?  Awesome!  We’re almost done!  What if you want to add an additional vCenter Platform Services Controller?

Deploy vCenter 6 VCSA Additional External PSC in VMware Workstation

You may be content with just a single external PSC, but additional PSCs can be deployed to test other scenarios as well. Here’s how to deploy an additional external PSC in VMware Workstation:

  1. Double click on c:\VCSA\vcsa\vmware-vcsa.ova
  2. Provide the name for the new virtual machine.  I’m calling mine vcenter6-2-psc2. Also provide the storage location for the virtual machine.
  3. After importing is completed, open the virtual machine’s VMX file before you power the VM up.  You need to add the following lines to the VMX file, adjusting values as needed: 
    guestinfo.cis.appliance.net.addr.family = "ipv4"
guestinfo.cis.appliance.net.mode = "static"
guestinfo.cis.appliance.net.addr = "192.168.1.62"
guestinfo.cis.appliance.net.pnid = "vcenter6-2-psc2.vs6lab.local"
guestinfo.cis.appliance.net.prefix = "24"
guestinfo.cis.appliance.net.gateway = "192.168.1.1"
guestinfo.cis.appliance.net.dns.servers = "192.168.1.80"
guestinfo.cis.vmdir.password = "P@ssw0rd"
guestinfo.cis.appliance.root.passwd = "P@ssw0rd"
guestinfo.cis.deployment.node.type = "infrastructure"
guestinfo.cis.vmdir.site-name = "default-first-site"
guestinfo.cis.vmdir.domain-name = "vsphere.local"
guestinfo.cis.vmdir.first-instance = "false"
guestinfo.cis.vmdir.replication-partner-hostname = "vcenter6-2-psc1.vs6lab.local"
  4. Ensure that you created both the A and PTR records for this appliance.  If you didn’t create them correctly, the remaining steps are a waste of time, as you’ll have to redeploy the appliance.
  5. Power the virtual machine on.  If you get error messages that the VMX file is corrupt, the above lines likely did not get added properly within the VMX file.  If you copied above from my web page, try retyping it all out in notepad, as sometimes HTML invisible formatting gets copied and pasted that you’re not aware of.  Allow the machine to complete its initialization.
  6. Verify it has completed properly.  To do this, you can open the VM’s console window, verify that it shows the correct name and IP address, and does not show any error messages that say firstboot failed.  If you see this error, you likely did not put in the proper information above, and/or the DNS A and PTR records were not properly created, or a similar issue with name resolution.  Also, you can go to https://vcenter6-2-psc2.vs6lab.local and verify the web page comes up, telling you to sign into a vCenter Management server to manage the PSC.  You should also go into the vSphere Web Client under Administration > System Configuration > Nodes and verify the new PSC shows up, and its services are healthy. additional external PSC in VMware Workstation check

To make a PSC in a different site, change the guestinfo.cis.vmdir.site-name value to a new site.

And there you have it!

powercli

Manage ESXi SSH Using PowerCLI

PowerCLI, vSphere,

Let’s face it. Starting and stopping SSH in ESXi is pain through GUI methods.  I often as a consultant need to connect via SSH to hosts to run data collect scripts, assess NIC and HBA firmware and driver versions, and for troubleshooting purposes, like to run esxtop.  The good news is you can manage ESXi SSH Using PowerCLI.  How cool is that?

Just remember to use get-vmhost to narrow down the specific hosts you want to execute the following commands.

Get the current status of ESXi SSH Using PowerCLI

get-vmhost  hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | select-object vmhost,policy,running

Policy is the start up mode.

  • Automatic = Start automatically if any ports are open, and stop when all ports are closed
  • On = Start and stop with host
  • Off = Start and stop manually

Start ESXi SSH Using PowerCLI

get-vmhost hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | start-vmhostservice -confirm:$false

Note the confirm switch.  If you don’t specify that, it will prompt you.

Stop ESXi SSH Using PowerCLI

get-vmhost hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | start-vmhostservice -confirm:$false

Note the confirm switch.  If you don’t specify that, it will prompt you.

Set startup policy for ESXi SSH Using PowerCLI to start and stop with host

get-vmhost hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | set-vmhostservice -policy "Off"

Be careful if you have any third party products that use SSH.  Nutanix for example comes to mind.  If you goofed and need it set to start and stop with host, just use “On” for the policy parameter.