Category Archives: vSphere

useful utilities duct tape

Useful Utilities Are Useful

If you’re an IT pro, no matter if you’re an admin, and engineer, a consultant, a PC technician, you have a toolbox of useful utilities, scripts, and software that you use to fix problems.  As time goes by, some of those tools get used more and more.  Others are used less and less for various reasons.  But what surprises me is how many tools in my toolbox on the surface have less and less use cases, but I still come back to them even when it seems I never would need to again.

Over the last few weeks, I’ve been working with a customer who has had significant turnover from consultants they’ve used.  They are moving off a troubled disparate datacenter environment that had over time developed numerous problems to a more consolidated environment that various SyCom resources including me have built for them that is functioning properly, has updated software and firmware, etc.  Along the way, we’ve run into numerous challenges that you wouldn’t normally anticipate.  Troubleshooting them to fix the problems often would take too much time to fix,.  Finding a duct tape solution was more expedient.

I wanted to give a few examples just to illustrate that having a wide knowledge of utilities out there and experience with them can help you solve problems.

In this case, the task was seemingly simple – move VMs running on a legacy NetApp array and vSphere 5.1 servers to a new(er) cluster running vSphere 5.5.  The clusters were managed by two different vCenter servers.  These clusters were within the same physical datacenter.  They had network connectivity between them.  They did not have access to the same storage arrays.  The customer allowed downtime to move them.  Therefore, the easiest way was a shared nothing cold migration (we’re running 5.1 on the source side, remember).  Simple, right?

Doing It the Textbook Way

I approached this like how any vSphere resource would.  Get the two clusters into the same vCenter instance, shut the VMs down, and migrate them cold.  How many times have you seen that fail?  Me?  Pretty much never.  Well, it wouldn’t work.  I’ll spare you the troubleshooting details, but trust me, doing it the native way wouldn’t work.

At this point, the time had come to get creative and bust out some useful utilities I hadn’t used in a long time.  We had to get the job done.  Tick tick!

Useful Utilities #1 – Veeam FastSCP

The customer wasn’t a Veeam customer (yet).  While the customer could take some downtime off hours, there was a limit to that.  We had to move about 2TB of data, so we needed to move this data as quickly as possible without a ton of labor to reconfigure the networks to get both environments access to the storage.

Sure, I could use WinSCP to just bulk copy the VMs over, but Veeam FastSCP, built into Veeam Backup and Replication trial, is free, and it moves data quicker as it disables encryption on the data transfer, which was acceptable to the customer.  I hadn’t had any reason to use FastSCP in probably five years because cold migration functionality and exporting VMs to OVFs and what not within vSphere made it unnecessary.  But here I was, using it yet again.

And sure enough, it worked like a champ.  We tested a quick procedure using it on a few development workloads.  We then proceeded moving all but the critical VMs, and it worked great… except for the last VM of course.  Come to find out, that was a critical SQL VM that the customer didn’t realize was using physical Raw Device Mappings.

Well, shoot, how do we do this one in a quick manner?

Useful Utilities #2 – VMware Converter

For numerous reasons, including perhaps sheer circumstance of projects I’ve worked on, I hadn’t until this had a need to use VMware Converter in years.  Virtualization is so prevalent now, that P2V is one of those things for me that’s like, “Hey man, remember that time we had to convert like 100 physical machines to virtual back in the day?  Good times!”

Also, I’ve generally recommended to customers to avoid converting physical to virtual anyway.  It should generally be seen as a shortcut, but never optimal.  If you could just build a fresh new VM and get the data moved, the resulting VM would be cleaner.  It would probably perform better.  There’s less chance of instability from old drivers and what would inevitably be a significant change in hardware for the OS and application.  Obviously, if you’re dealing with a ton of machines, rebuilding them all isn’t practical.  In that case, you might have to turn to a P2V tool.

But if you got a VM with physical RDMs, you can’t clone the VM.  You can’t bulk copy the Virtual Machine files over.  You could create new VMDKs and copy everything out of the RDM disks to those and reassign drive letters.  However, this SQL VM was nasty with complex mount points and drive letters assigned.  We had to get it done the weekend the RDMs were discovered.

Solution?  VMware Converter!  I tried installing it on an admin server and set up the job.  That of course failed because of Murphy’s Law.  The Converter agent wouldn’t install due to insufficient permissions.  I installed it directly on the SQL VM (with the same account I tried to push the agent, mind you), stopped the SQL services to ensure the data was static, and ran it.  Other than it shuffling a few drive letters around on the converted VM that a few mouse clicks fixed, it worked like a champ.

How about you?  Any useful utilities you’ve used recently you haven’t used in awhile?

Making life easier using vSphere Tags

One of the least used features in vSphere that I think almost all admins could really make use of but don’t is the ability to create custom vSphere tags within vSphere.

I wanted to take the time to point this feature out, and perhaps give people some ideas on how to make use of of them.  This can help with management and automation quite a bit.

What are vSphere Tags?

vSphere Tags are effectively custom metadata type info that can be applied to objects within vCenter.  You get to make your own to fit your own needs.  They assist basically with locating objects for more efficient administration and management.

They’re unique to other things such as folders for your VMs in that you can assign multiple tags to the same VM or other objects.

Let’s break this down by comparing vSphere tags to MP3 management software like iTunes.  An individual MP3 file must be in one file system folder or another.  It can’t be in both.  But suppose you want to find all your songs by an artist, by genre, or by album?  We intuitively understand this now with MP3s.

But we have the same problem with VMs.  You can organize your VMs into VM folders in vCenter, but a single VM can only be in one folder or another.  What if you wanted to organize your VMs by criticality?  By whether or not they have SQL?  Whether or not they need to be backed up?  Trying to do this with folders would be a nightmare to manage.  Plus, remember a VM folder is the mechanism for assigning permissions, too.  Maybe you don’t want this metadata having any impacts on anyone’s permissions to manage it.

That’s when you use vSphere tags!

Use Cases for vSphere Tags

Use cases for this functionality are numerous:

  • Criticality of VM – this would allow the expedited power up or down of VMs based on this nature.  Running out of resources within your cluster due to sudden host failures?  Power down the non-critical VMs.  It would also be helpful for vSphere Admins who aren’t the application admins to know when to handle a VM with care before doing anything to it.
  • Application groupings – Maybe it doesn’t make sense to put VMs that work together to provide an application or service, but you want to know those groups.  That could allow a SQL server that serves the backend of multiple application groups to be identified for both simultaneously.
  • Presence of a common application like SQL – This can be helpful for locating VMs that may require special settings on backup jobs to quiesce the file system before backing the VM up.  You might also use this to find potential VMs that other VMs are dependent on, so you can set their restart priority so they boot up first in an HA event scenario.
  • Lab/Test VMs – You could set the resource allocation for Lab/Test VMs to low to help ensure they are given less resources than production VMs.

OK, I convinced you (hopefully)!   Let’s make some tags.

Basic Concepts for vSphere Tags You Need To Know

You can create vSphere tags in both within the vSphere Web Client and with PowerCLI.  It’s simple, but you need to know a few concepts.

All vSphere tags belong to a Category.  There are two main types of categories.  This notion is called Cardinality.  It sounds more complicated than it is.  Basically, you can have a category where only a single tag from that category can be applied to any given object.  For example, let’s say you want to tag VMs by criticality.  Logically, a VM will only have one criticality rating, not multiple.  IE, it makes zero since for a VM to be both low and medium as far as how critical they are.

However, sometimes you might want a category that multiple tags could apply to the same object.  For example, let’s say you want to make a category called “Special Applications” to identify very specific apps within a VM to easily identify SQL servers, Domain Controllers, and Exchange servers.  While I wouldn’t recommend it, it’s possible for a single VM to be all three simultaneously.

vSphere tags can apply to all kinds of objects as well, not just VMs.  You can select which objects a tag can be applied to within the category.

Managing vSphere Tags Using the Web Client

To create tags within the vSphere client, navigate to the Tags section of the web client.

vm tags web client nav

You must create a category first if there isn’t one already made.  Click the Categories button, and then click the create categories icon.

For this example, we will make a category for criticality ratings for VMs.  We want one tag per object, not more, and we only want the tag to be applied to VMs or vApps.

vsphere category example

Now that we have our category, we can create tags within it.  Click on Tags, and the new tag icon.  Be sure to select the category during tag creation.

vsphere tags create tag example

Rinse and repeat for all the tags you want to create for the category.  One tip I recommend is to name the tags with incuding their category name, which refers to some kind of concept.  Since you usually search by the tag name, you want for example LowCriticality instead of Low.  (See below for search examples.) Low in and of itself could mean a lot of things.  Low resource usage, low criticality, etc.

To apply a tag to an object, simply right click the object, point to Tags & Custom Attributes > Assign Tag…

vsphere tags assign tag

A new dialog box appears where you can filter categories or see all categories and select the vSphere tags you wish to assign.  Also, notice you can remove tags here, too.

Managing vSphere Tags Using PowerCLI

PowerCLI has full tag management functionality within it, too.

Creating a category:

New-TagCategory -Name VMCriticality -Description "Criticality of the VM" -Cardinality Single -EntityType "VirtualMachine","VApp"

Creating a tag:

New-Tag -Name "LowCriticality" -Description "Non-Critical VMs" -Category VMCriticality

Assigning a tag to a VM:

get-vm Shoretel | New-TagAssignment -Tag "HighCriticality"

You can do lots of things with PowerCLI and tags.

Using vSphere Tags

Now that you have tags created and applied, you can now make use of them to make your life easier.

You can make use of tags in both the vSphere Web Client and via PowerCLI.  To find all VMs with a tag within the vSphere Web Client, simply type the tag value in the search box.  The tag name will automatically populate.

vsphere tags searching

Click on it.  Boom, you got your objects with that tag!

vsphere tags search results

There’s also a parameter on PowerCLI’s Get-VM cmdlet to identify the VMs with that tag.  You can then pipe that to another cmdlet.  Say for example you want to shutdown your non-critical VMs because you suddenly experience multiple host failures, so you need to make sure your more important VMs get the resources they need:

Get-VM –Tag “LowCriticality” | Shutdown-VMGuest

Imagine if you set up vSphere tags to identity all your VMs with SQL.  Imagine you’re setting up Veeam backup jobs, and you need to know which VMs you need to setup special quiescing.  You could easily just get that list of VMs.

That’s how to use vSphere tags!

How do you think you might be able to use them, or how do you use them within your environment?

vSphere 6.5 – New features I can’t wait for!

VMware announced vSphere 6.5 at VMworld Europe.   I don’t want to go through everything that’s new, but I do want to go over the vSphere 6.5 new features I think are the coolest that I can’t wait for.

vSphere 6.5 New Features – Me likey!

Here are the vSphere 6.5 new features I specifically wanted to highlight that I think are going to be the most useful to my customers.

vCenter 6.5 New Features

  • The vCenter Server Appliance (VCSA) FINALLY has an integrated VMware Update Manager.  No more Windows machine for VUM!  Even less excuses for using the Windows version!  Speaking of which…
  • Native VCSA high availability!  In vCenter 6.0, the only way to make vCenter truly highly available was to use Windows Clustering.  Not anymore!  Now the VCSA has its own ability.  VCSA NOW AND FOREVER!
  • File-based backup and recovery for VCSA, so it’s even easier to make any kind of recovery you may need.
  • HTML5 based vSphere Web Client!  Take that, Adobe Flash!  No more Flash vulnerabilities and issues to worry about!
  • Fully supported standalone HTML5 based thick client!

Clustering New Features

  • HA Orchestrated Restarts – Now you can enforce a chain of VMs to ensure VM interdependency for multi-tiered applications!
  • Proactive HA – Now you can integrate HA with hardware vendor monitoring tools to move VMs off hosts that have hardware problems before they actually result in an ESXi host crashing.  How cool is that?
  • DRS now takes network bandwidth into account, to ensure your workloads can be dynamically moved between hosts to ensure the best network performance.

Security New Features

I have numerous customers who for legal and other reasons are extremely security conscious.  These may be of particular interest:

  • vMotion traffic encryption – One of the reasons I recommend segregated isolated non-routable VLANs generally for vMotion traffic is due to the fact that vMotion traffic is unencrypted.  Think about the implications of that.  The running contents of RAM for a VM is copied in the clear over a network during a vMotion!  If that’s a VM processing let’s say credit card transactions or personally identifiable information like a Social Security number, that’s pretty scary!  Now consider the boundaries of vMotions have been lifted to the point you can conceivably vMotion a VM across datacenters.  Now, for the first time, you can encrypt this traffic.
  • VM disk encryption – If your shared storage solution can’t encrypt your data at rest, you used to be out of luck for doing whole VM encryption.  Not anymore!  Now it can be done at the VM level!
  • Better logging now to provide better auditing capability to see who did what within the environment.

There’s a whole lot more in this release.  I’m sure I’ll post more about these and other cool features and capabilities soon!

Before you ask, the tentative release date for vSphere 6.5 is Q4 2016.

Resolving VM MAC Conflict alarm with Veeam Replicas

It’s been awhile since I’ve deployed Veeam using replication with vSphere 6.0.  I recently implemented it for a customer who was replicating VMs to a secondary storage appliance in addition to backing the VMs up to a Data Domain.  Upon running the initial replication for the VM, a “VM MAC Conflict” alarm triggered on the replica VM.

vm mac conflict alarm triggered

Here’s a description of what’s going on and how to prevent the VM MAC Conflict alarm from triggering.

VM MAC Conflict Alarm

The VM MAC Conflict alarm is new to vCenter 6.0 Update 1a.  The intent of the alarm is to warn you if two vNICs on VMs within a vCenter instance have the same MAC address.  This can happen for a variety of reasons:

  1. vCenter malfunctioned and dynamically provided the same MAC address to two or more vNICs.
  2. Either intentionally or mistakenly, an admin or a third party product might have statically assigned a MAC address already in use within the environment.  In this case, Veeam created a copy of the VNX file with identical MAC addresses for the source and replica VM’s vNICs.

It’s a good alarm to have to notify you just in case.  But how do you keep this alarm while stopping it from triggering on replica VMs?

Stopping VM MAC Conflict Alarms from triggering for Veeam Replicas

The solution for preserving the VM MAC Conflict alarm while stopping it from triggering on Veeam replicas is quite simple.  You can modify the alarm itself by setting an exception to exclude VMs.  In the case of Veeam replicas, they have a “_replica” suffix within the VM name by default.  If you changed that suffix in the replica job, just adjust accordingly.

Go to the VM MAC Conflict alarm definition.  It’s in the vCenter inventory object under Manage > Alarm Definitions.  Click the alarm and on the right, click Edit.

Under the bottom box that reads, “The following conditions must be satisfied for the trigger to fire”, add a condition that says the VM name does not end with “_replica”.  Once applied, the alarm disappears for your replica VMs.

vm mac conflict alarm modified

That’s it!

Compromised vSphere 6.0 Certificates – Part 3

This is the final part of my series on how to deal with compromised vSphere 6.0 certificates.  If you are coming here first, I highly recommend reading:

Compromised vSphere 6.0 Certificates – Part 1

Compromised vSphere 6.0 Certificates – Part 2

We pick up where we left off in Part 2.  The scenario is there are suspected compromised vSphere 6.0 certificates in your environment that were provided to vCenter issued either via a root certificate within the Platform Services Controller (PSC), or the PSC generated certificates with an installed intermediate certificate from an external Public Key Infrastructure (PKI).  Again, VMware does not support certificate revocation when its PSC automatically generated the cert using either its own root or via the external PKI’s issued intermediate.  You must then regenerate all certs.

At this point, I assume certificates were regenerated for the PSC’s root or the intermediate certificate, along with all vCenter server certificates.  I outlined that process in Part 2.  Now, the question is what to do about everything else when dealing with compromised vSPhere 6.0 certificates?  What about ESXi servers?  What about external products that plug into vCenter like VMware Update Manager?  NSX Manager?  vRealize Operations Manager?

Let’s get to it!

Compromised vSphere 6.0 Certificates – ESXi servers

After resetting all the certificates within the PSC and vCenter, I have good news when it comes to your ESXi servers.  They won’t have any problems.  Resetting VCSA certificates has nothing to do with ESXi servers because they do not obtain certificates from VCSA, nor do ESXi servers have any trusts of the certificates that were reset.

Yay!

Compromised vSphere 6.0 Certificates – Most external vCenter dependent solutions from VMware

In most cases, with it comes to external vCenter products that establish relationships with vCenter, these products do often establish a trust of one of the certificates that were reset.  However, they do not obtain certificates from a PSC themselves.  Therefore, you need to fix these products by simply establishing trust with the new certificate that is now installed with vCenter.   And in most cases, this is as easy as it was when you registered it with vCenter in the first place.

I’m not going to show step by step of every product, as I don’t have the time.  I will however come back and update this post if/when I need to do this with various products.  I am going to use vRealize Operations Manager as an example of a typical product that is fixed the same basic way.

vRealize Operations Manager

Here is what vCOPS looks like following the PSC/vCenter certificate resets:

compromised vsphere 6.0 certificates vcops statusNote we are checking the same place you go to register the product with vCenter in the first place (Appliance portal > Solutions > VMware vSphere).  If we were talking about NSX, you would login to the NSX Manager’s direct portal and navigate to Manage Appliance Settings > NSX Management Service > Configure.  This is the same basic concept, even though where you navigate might be different on the product.

Go into settings and re-establish the connection.  In vCOPS, that means clicking the gear on that page, and on both the vCenter Adapter and vCenter Python Actions Adapter, click “Test Connection”.  Low and behold, a pop up comes up to ask if you wish to trust a new certificate it doesn’t yet trust.

compromised vSphere 6.0 certificates trust new certs

If you click OK to trust, vCOPS adds the new certificate to its trusted store.  However, you get an error that you effectively can’t trust two certificates for the same object.

compromised vSphere 6.0 certificates vCOPS trust new cert error

I show this just in case other products share similar behavior.  Delete the old trusted certificate from the appliance.

In this case, navigate to Certificates, and delete the trusted certificate.

compromised vSphere 6.0 certificates vCOPS delete old untrusted cert

Hovering over a column gives more specific info for that cert, which can help identify which certificate to delete.

Then, go back and issue the Test Connection command.

Stop and start the collections on the Solutions page as needed.

compromised vSphere 6.0 certificates vCOPS data receiving

Click refresh and wait to ensure  “Data Receiving” is shown for the collection status.  Otherwise, vCOPS is not functioning.

Other products will have their idiosyncrisies, but they have the same basic concept of establishing trust for the new vCenter certificate.  You perform this process pretty much where you registered the product with vCenter in the first place.

Compromised vSphere 6.0 Certificates – Abnormal external vCenter dependent solutions from VMware

Some products need specialized procedures to trust the new certificates installed in your vCenter/PSC servers.  Here are all of the ones so far I’ve run into, and how to fix those:

vCenter Update Manager (VUM)

I’m hardly shocked VUM would need a specialized procedure.  VUM runs on a Windows OS only.  It remains 32-bit, as opposed to almost every other VMware product.  Plus, it has had esoteric procedures when it came to certificates for a long time.

Navigating to an impacted VUM server within the vSphere Client nets you a pretty immediate error that clearly shows a problem with the SSL certificate.

compromised vSphere 6.0 certificates VUM SSL error

“sysimage.fault.SSLCertificateError”

Time to fix the trust of the new certificate!

First, remote into the VUM server.

Next, run the VMware vSphere Update Manager Utility under the installation directory for VUM (X:\Program Files (x86)\VMware\Infrastructure\Update Manager\VMwareUpdateManagerUtility.exe, where X is the drive in which you installed the VUM binaries).  Login, and select to re-register with vCenter.

compromised vSphere 6.0 certificates reregister VUM

Of course, restart vSphere Update Manager service to complete the process.

The error will go away, and VUM will function again.

Summary

Hopefully, this gives everyone enough info to complete the process or point them in the right direction.  If you have any insights to other products I didn’t cover, please post in the comments!  As I try more products, I will also update this article.

Thanks for reading!

vCenter to VCSA migration tool released

It’s out!  You can finally migrate a Windows vCenter install to a vCenter appliance VCSA based install using a supported VCSA migration tool!

vCenter Server Migration Tool: vSphere 6.0 Update 2m

Just a heads up, here are the requirements:

  • It only migrates from Windows installations to VCSA based installations
  • It only migrates from 5.5 (any revision) to VCSA 6.0 Update 2
  • You must used the embedded database in VCSA
  • It will not migrate VUM if VUM is installed on the same OS as your vCenter (no big deal IMO, not hard to reinstall VUM)

Cool things I noticed about this migration utility according to the link above:

  • Preservation of original vCenter settings including
    • IP address
    • FQDN
    • Certificates
    • Alarm settings (AWESOME!)
  • It doesn’t touch your current vCenter’s stuff, so easy rollback.

Not so cool things:

  • You can’t change deployment models during the migration.  A simple deployment in 5.5 becomes an embedded deployment in VCSA, which you don’t want to do, but you can always transition out of that.

I’m sure you’ll be seeing a blog article here kicking the tires on this thing.  But this will definitely help me convince people to go to the VCSA.

Compromised vSphere 6.0 Certificates – Part 2

In this second blog article, I discuss what to do with compromised vSphere 6.0 Certificates issues by a PSC to vSphere components.  As mentioned in the previous blog article, you cannot revoke certificates issued by a PSC either using an installed intermediate certificate from an external CA or using its own root.  You must regenerate all certificates instead.

FYI, this post assumes you’re using the VCSA.  Windows installable vCenter is nearly identical, aside from the path to Certificate Manager.

Compromised vSphere 6.0 Certificates – Embedded PSC With Own Root Certificate

If you have compromised vSphere 6.0 certificates automatically generated from an embedded PSC, you must regenerate all certificates.  Yes, you must regenerate even certificates you don’t suspect, too.

To do this:

  1. Login as root into your embedded vCenter server via console, SSH, etc.
  2. Enter into shell.  If you didn’t enable shell via the console, you can run “shell.set –enable True” and then run “shell”.
  3. Run the certificate manager utility.  For the VCSA, you simply run /usr/lib/vmware-vmca/bin/certificate-manager
  4. Select option 4 – Regenerate a new VMCA Root Certificate and replace all certificates.
  5. Certificate Manager asks for various pieces of information for each certificate regeneration such as the country, organization, OrgUnit, State, Locality, email, etc. These are cosmetic values mostly, and are only visible if someone really examines the certificate.  Functionally, they make no difference.  However, I wanted to call your attention to a couple of things that are very important. It is VERY CRITICAL you do the following for each certificate, or else the process will fail!
    1. There is a bug in the certificate automation tool, where if you answer identical values for all questions asked, the same certificate will be generated for that cert.  You’ll notice there are multiple certs that end up being regenerated.  You can tell which one is being regenerated with the following line: “Please configure root.cfg with proper values before proceeding to next step.”  That means the root certificate is being regenerated.  You’ll see various certs as well like “machine”, “machine-ssl”, “vpxd.cfg”, etc.   Each one of these certs must actually be unique.  Ensure that you give at least some different value for one of the questions asked for every cert regenerated for a server.  By far, the easiest way to do this is to answer the following question uniquely for every cert: “Enter proper value for ‘Name’ [Default value : CA]”  Simply name it an abbreviated name of the server and the certificate name.  In this case, you could call it “VC-ROOTCFG”.  Answering every other question identically won’t hurt.
    2. One question that is more than cosmetic that you must answer correctly is: “Enter proper value for ‘Hostname’ [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com]”.  Make sure this is the actual DNS name for the vCenter server.
  6. When prompted afterregenerating all certificates, stop and start all services using:
    1. service-control –stop –all
    2. service-control –start –all
  7. I would recommend rebooting your vCenter server now.
  8. Download your root certificate again and reimport into GPO or however you established trust on the clients for the root originally.
  9. Fix all trust issues with external products.  (See part 3 of this series!)

This is probably the one time you might actually want an embedded PSC for vCenter.  This is far simpler than if you have an external PSC.  (I still recommend external PSC’s in all cases for the record!!!)

Compromised vSphere 6.0 Certificates – External PSC(s) With Own Root Certificate

This is somewhat similar.  However, keep in mind each PSC is a CA.  Therefore, you probably should do this on every PSC that’s a part of the same environment if you suspect certificate(s) have been compromised.

To do this:

  1. Login as root into your external PSC server via console or SSH.
  2. Enter into shell.  If you didn’t enable shell via the console, you can run “shell.set –enable True” and then run “shell”.
  3. Run the certificate manager utility.  For the VCSA, you simply run /usr/lib/vmware-vmca/bin/certificate-manager
  4. Select option 4 – Regenerate a new VMCA Root Certificate and replace all certificates.
  5. Certificate Manager asks for various pieces of information for each certificate regeneration such as the country, organization, OrgUnit, State, Locality, email, etc. These are cosmetic values mostly, and are only visible if someone really examines the certificate.  Functionally, they make no difference.  However, I wanted to call your attention to a couple of things that are very important. It is VERY CRITICAL you do the following for each certificate, or else the process will fail!
    1. There is a bug in the certificate automation tool, where if you answer identical values for all questions asked, the same certificate will be generated for that cert.  You’ll notice there are multiple certs that end up being regenerated.  You can tell which one is being regenerated with the following line: “Please configure root.cfg with proper values before proceeding to next step.”  That means the root certificate is being regenerated.  You’ll see various certs as well like “machine”, “machine-ssl”, “vpxd.cfg”, etc.   Each one of these certs must actually be unique.  Ensure that you give at least some different value for one of the questions asked for every cert regenerated for a server.  By far, the easiest way to do this is to answer the following question uniquely for every cert: “Enter proper value for ‘Name’ [Default value : CA]”  Simply name it an abbreviated name of the server and the certificate name.  In this case, you could call it “PSC1-ROOTCFG”.  Answering every other question identically won’t hurt.
    2. One question that is more than cosmetic that you must answer correctly is: “Enter proper value for ‘Hostname’ [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com]”.  Make sure this is the actual DNS name for this server.  Even if it asks for a cert for the web client, do NOT put in the name of the vCenter server.  It will also ask for an optional IP address.  Obviously, if you input one, make sure it’s the correct one.
  6. When prompted after regenerating all certificates, stop and start all services using:
    1. service-control –stop –all
    2. service-control –start –all
  7. I recommend rebooting the machine when you’ve completed this.
  8. To verify the PSC cert reset worked, attempt to go to https://FQDNofPSC.domain.com/psc to ensure you get a login prompt.  If you don’t, the certificate reset failed.  Stop and redo this portion again.  You likely didn’t provide some kind of different answer to one of the questions for each certificate to make them unique.
  9. Run Certificate Manager on your vCenter server(s).  Here’s where it gets weird.  VMware says you should run Option 3 – Replace Machine SSL certificater with VMCA Certificate and answer the questions.  Next, run Option 6 – Replace Solution user certificates with VMCA certificates.  That didn’t work for me.  The only way I could get it to work is run Option 8 – Reset all certificates.  That’s the only way I could get it to work.  I found another oddity.  During this process, you are asked: “Performing operation on distributed setup, Please provide valid Infrastructure Server IP.”  If I entered an IP address, and did the rest correctly (remember to answer the questions but provide a different value for name for each certificate!), the process would kick off, get stuck at a long time here and eventually fail:Status : 85% Completed [starting services…]
    Error while starting services, please see log for more details
    Status : 0% Completed [Operation failed, performing automatic rollback]

    Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

    Then the certificates would roll back.  Enter the FQDN of one of your PSC servers instead!  That allows it to continue.

  10. Download your root certificate again and re-import into GPO or however you established trust on the clients for the root originally.
  11. Fix all trust issues with external products.  (See part 3 of this series!)

This is far more complicated than the first one, but it’s probably the one you’re more likely to need to do.

Compromised vSphere 6.0 Certificates – Intermediate CA

If you installed a now compromised intermediate CA certificate, revoke the intermediate certificate within the external PKI.  You should then request and install a new intermediate certificate within the PSC.  Then proceed with regenerating certificates for all other components. (See above…)

And that’s how you deal with compromised vSphere 6.0 Certificates.  In part 3, I’ll delve into how to fix trust issues with various products that might arise from regenerating these certificates.

Compromised vSphere 6.0 Certificates – Part 1

As I alluded to in a previous post, I’ve been needing to do some more in depth testing in relation to vSphere 6.0, which I run in VMware Workstation.  Now the cat is out of the bag!  I’m running through scenarios about what with compromised vSphere 6.0 certificates.

After scouring the internet, there are plenty of blog articles about how to go with various certificate management models.  There’s not a lot of information what to do if suspect a compromised vSphere 6.0 certificate.

I wanted to cover the basics here discuss more specifics in later articles.  I’m going to start with a short introduction to vSphere 6.0 Certificate Management, and then the implications of each when it comes to a compromised certificate.

vSphere 6.0 Certificate Management Basics

I’ve posted about this in the past.  Here’s a VERY quick recap.

  1. You can have the Platform Services Controller (PSC) act as a root Certificate Authority (CA), and hand out certificates automatically to other vSphere components automatically, which is the easiest to implement and manage.
  2. You can have the PSC act as an intermediate CA, and issue certificates using the intermediate certificate you install automatically.  This is arguably the second easiest to implement and manage.
  3. You can generate Certificate Service Requests to an external CA manually for various vSphere components, and install those certificates manually.  This is arguably the hardest to implement and manage.

One other note here is you can mix and match these options.  This is usually implemented by having non-client internal vSphere component certificates be issued by the PSC, and client facing certs such as the cert for the vSphere Web Client be issued by an external CA.

Again, the above information is not intended to be a primer for certificate management in vSphere 6.0.  It’s only to facilitate discussion about what to do if a certificate has been compromised.

Dealing with Compromised vSphere 6.0 Certificates Issued by an External CA

One advantage of using an external certificate authority to issue the certificates for vSphere is the support for certificate revocation.  If any certificate is compromised that was issued by an external CA, you can simply within that PKI environment revoke the certificate.  Replacing compromised vSphere 6.0 certificates is done the same way the certs was acquired in the first place.

The basic steps would be:

  1. Revoke the suspected compromised certificate within the PKI.
  2. Go through the process of obtaining a new certificate, and install it.
  3. Fix any trust issues that may occur with the new certificate.  For example, you must  manually fix VUM when you change a vCenter certificate.

It’s not so straightforward if the PSC generated the certificate you suspect is compromised.

Dealing with Compromised vSphere 6.0 Certificates Issued by a PSC

For all intents and purposes, it doesn’t really matter if compromised vSphere 6.0 certificates were issued by a PSC using its own root certificate or using an installed intermediate certificate obtained from an external CA.  If the PSC in the end generated the certificate used by a vSphere component, any vSphere component, certificate revocation is not supported.

If you suspect a certificate has been compromised, you have no choice but to regenerate all certificates, even certificates that you don’t expect to be compromised.  This should certainly be considered prior to deciding upon which model to use for vSphere 6.0 Certificate Management.

The basic steps would be:

  1. Run the Certificate Management Utility on the PSC in question to regenerate all certificates.  If any doubts, do this on all PSCs.
  2. Run the Certificate Management Utility on any vCenter server that obtained its certificates from that PSC.  If you’re running embedded PSC with vCenter, you already did this in step one.
  3. Fix any trust issues that may occur with the new certificate.  For example, you must  manually fix VUM when you change a vCenter certificate.

If the above seems like it’s just as easy, it isn’t.  For one, documentation on how to do this with external PSC’s is vague and confusing from VMware.  Secondly, it gets more complicated the more PSC and vCenter nodes you have.

I’ll go in more depth on how address compromised vSphere 6.0 Certificates issued by a PSC in Part 2.  In Part 3, I’ll address how to fix trust issues with certificates in various products.

vmware workstation

vCenter 6 VCSA External PSC in VMware Workstation

I’ve been doing a lot of various oddball testing with vCenter for various scenarios, which have required me to deploy more complex configurations with vCenter 6 recently in my lab.  I found very quickly that there isn’t good consolidated documentation on how to do more advanced vCenter deployments other than directly to ESXi hosts.  It took me quite a bit of time to figure this all out.  I wanted to share this with anyone else who may be doing similar test.  Here’s how to deploy vCenter 6 VCSA with External PSC in VMware Workstation.

And I mean this to be a “proper” lab you can really test vCenter stuff in.  No plain IP addresses for host names!  We want proper FQDNs and what not here!

I am assuming you already have the following up and functioning, along with the following information:

  • DNS server
  • Proper networking for VMware workstation suitable for whatever you’re going to do
  • Document what you will want your host names, IP addresses, DNS IP(s), default gateway IP, and Single Sign-On site names in advance.  Fair warning: this all can get very confusing!  Don’t introduce confusion by deciding these things on the fly.  We’re going to use the following for this article:
    • 1st PSC – vcenter6-2-psc1.vs6lab.local, 192.168.1.61
    • 2nd PSC – vcenter6-2-psc2.vs6lab.local, 192.168.1.62
    • vCenter – vcenter6-2.vs6lab.local, 192.168.1.60
    • DNS server: 192.168.1.80
    • Default Gateway: 192.168.1.1

Deploy vCenter 6 VCSA External PSC in VMware Workstation – Preparation

Here are the things you should get out of the way first:

  1. Download VCSA 6 from VMware if you haven’t already done so.
  2. Extract the VCSA download package to a temporary directory.  For simplicity’s sake, we will assume you extracted the download to c:\VCSA.  Rename the c:\VCSA\vcsa\vmware-vcsa file with an OVA file extension.
  3. Create A AND PTR records for all PSC and vCenter nodes within your lab’s DNS server.

Deploy vCenter 6 VCSA 1st External PSC in VMware Workstation

In order to begin your lab deployment to have an external PSC in VMware Workstation, you must deploy the first PSC.

To deploy the first external PSC in VMware Workstation, do the following:

  1. Double click on c:\VCSA\vcsa\vmware-vcsa.ova
  2. Provide the name for the new virtual machine.  I’m calling mine vcenter6-2-psc1. Also provide the storage location for the virtual machine.
  3. After importing is completed, open the virtual machine’s VMX file before you power the VM up.  You need to add the following lines to the VMX file, adjusting values as needed:
    guestinfo.cis.appliance.net.addr.family = "ipv4"
    guestinfo.cis.appliance.net.mode = "static"
    guestinfo.cis.appliance.net.addr = "192.168.1.61"
    guestinfo.cis.appliance.net.prefix = "24"
    guestinfo.cis.appliance.net.gateway = "192.168.1.1"
    guestinfo.cis.appliance.net.dns.servers = "192.168.1.80"
    guestinfo.cis.system.vm0.hostname = "vcenter6-2-psc1.vs6lab.local"
    guestinfo.cis.vmdir.password = "P@ssw0rd"
    guestinfo.cis.appliance.root.passwd = "P@ssw0rd"
    guestinfo.cis.deployment.node.type = "infrastructure"
    guestinfo.cis.vmdir.first-instance = "true"
  4. Ensure that you created both the A and PTR records for this appliance.  If you didn’t create them correctly, the remaining steps are a waste of time, as you’ll have to redeploy the appliance.
  5. Power the virtual machine on.  If you get error messages that the VMX file is corrupt, the above lines likely did not get added properly within the VMX file.  If you copied above from my web page, try retyping it all out in notepad.  Sometimes HTML invisible formating gets copied and pasted that you’re not aware of.  Allow the machine to complete its initialization.
  6. Verify it has completed properly.  To do this, you can open the VM’s console window, verify that it shows the correct name and IP address, and does not show any error messages that say firstboot failed.  If you see this error, you likely did not put in the proper information above, and/or the DNS A and PTR records were not properly created, or a similar issue with name resolution.  Also, you can go to https://vcenter6-2-psc1.vs6lab.local and verify the web page comes up, telling you to sign into a vCenter Management server to manage the PSC.

We’re assuming this completed successfully at this point.  If you encountered issues, correct this before proceeding.

Deploy vCenter 6 VCSA Non-Embedded Server in VMware Workstation

After you deploy the first external PSC in VMware Workstation, you need to deploy the vCenter server itself.

By default, deploying a vCenter 6 server appliance will automatically default to embedded within Workstation.  The line guestinfo.cis.deployment.node.type within the VMX file controls the node type.  As you saw above, setting it to “infrastructure” makes the VCSA instance a Platform Services Controller (PSC).  Let’s make a vCenter server!

To deploy a vCenter Server leveraging the external PSC in VMware Workstation, do the following:

  1. Double click on c:\VCSA\vcsa\vmware-vcsa.ova
  2. Provide the name for the new virtual machine.  I’m calling mine vcenter6-2. Also provide the storage location for the virtual machine.
  3. After importing is completed, open the virtual machine’s VMX file before you power the VM up.  You need to add the following lines to the VMX file, adjusting values as needed:
    guestinfo.cis.appliance.net.addr.family = "ipv4"
    guestinfo.cis.appliance.net.mode = "static"
    guestinfo.cis.appliance.net.addr = "192.168.1.60"
    guestinfo.cis.appliance.net.pnid = "vcenter6-2.vs6lab.local"
    guestinfo.cis.appliance.net.prefix = "24"
    guestinfo.cis.appliance.net.gateway = "192.168.1.1"
    guestinfo.cis.appliance.net.dns.servers = "192.168.1.80"
    guestinfo.cis.system.vm0.hostname = "vcenter6-2-psc1.vs6lab.local"
    guestinfo.cis.vmdir.password = "P@ssw0rd"
    guestinfo.cis.appliance.root.passwd = "P@ssw0rd"
    guestinfo.cis.deployment.node.type = "management"
    guestinfo.cis.vmdir.domain-name = "vsphere.local"
    guestinfo.cis.vmdir.site-name = "default-first-site"
  4. Ensure that you created both the A and PTR records for this appliance.  If you didn’t create them correctly, the remaining steps are a waste of time, as you’ll have to redeploy the appliance.
  5. Power the virtual machine on.  If you get error messages that the VMX file is corrupt, the above lines likely did not get added properly within the VMX file.  If you copied above from my web page, try retyping it all out in notepad, as sometimes HTML invisible formating gets copied and pasted that you’re not aware of.  Allow the machine to complete its initialization.  Also, note that the vSphere Web Client takes a long time to initialize.  Be patient!
  6. Verify it has completed properly.  To do this, you can open the VM’s console window, verify that it shows the correct name and IP address, and does not show any error messages that say firstboot failed.  If you see this error, you likely did not put in the proper information above, and/or the DNS A and PTR records were not properly created, or a similar issue with name resolution.  Also, you can go to https://vcenter6-2.vs6lab.local and login to the vSphere Web Client.  Ensure you can access the administration and inventory sections of the Web Client.  Ensure both the vCenter appliances show up under Administration as healthy.vcenter using external PSC in VMware Workstation check1st external PSC in VMware Workstation check

Still with me?  Awesome!  We’re almost done!  What if you want to add an additional vCenter Platform Services Controller?

Deploy vCenter 6 VCSA Additional External PSC in VMware Workstation

You may be content with just a single external PSC, but additional PSCs can be deployed to test other scenarios as well. Here’s how to deploy an additional external PSC in VMware Workstation:

  1. Double click on c:\VCSA\vcsa\vmware-vcsa.ova
  2. Provide the name for the new virtual machine.  I’m calling mine vcenter6-2-psc2. Also provide the storage location for the virtual machine.
  3. After importing is completed, open the virtual machine’s VMX file before you power the VM up.  You need to add the following lines to the VMX file, adjusting values as needed:
    guestinfo.cis.appliance.net.addr.family = "ipv4"
    guestinfo.cis.appliance.net.mode = "static"
    guestinfo.cis.appliance.net.addr = "192.168.1.62"
    guestinfo.cis.appliance.net.pnid = "vcenter6-2-psc2.vs6lab.local"
    guestinfo.cis.appliance.net.prefix = "24"
    guestinfo.cis.appliance.net.gateway = "192.168.1.1"
    guestinfo.cis.appliance.net.dns.servers = "192.168.1.80"
    guestinfo.cis.vmdir.password = "P@ssw0rd"
    guestinfo.cis.appliance.root.passwd = "P@ssw0rd"
    guestinfo.cis.deployment.node.type = "infrastructure"
    guestinfo.cis.vmdir.site-name = "default-first-site"
    guestinfo.cis.vmdir.domain-name = "vsphere.local"
    guestinfo.cis.vmdir.first-instance = "false"
    guestinfo.cis.vmdir.replication-partner-hostname = "vcenter6-2-psc1.vs6lab.local"
  4. Ensure that you created both the A and PTR records for this appliance.  If you didn’t create them correctly, the remaining steps are a waste of time, as you’ll have to redeploy the appliance.
  5. Power the virtual machine on.  If you get error messages that the VMX file is corrupt, the above lines likely did not get added properly within the VMX file.  If you copied above from my web page, try retyping it all out in notepad, as sometimes HTML invisible formatting gets copied and pasted that you’re not aware of.  Allow the machine to complete its initialization.
  6. Verify it has completed properly.  To do this, you can open the VM’s console window, verify that it shows the correct name and IP address, and does not show any error messages that say firstboot failed.  If you see this error, you likely did not put in the proper information above, and/or the DNS A and PTR records were not properly created, or a similar issue with name resolution.  Also, you can go to https://vcenter6-2-psc2.vs6lab.local and verify the web page comes up, telling you to sign into a vCenter Management server to manage the PSC.  You should also go into the vSphere Web Client under Administration > System Configuration > Nodes and verify the new PSC shows up, and its services are healthy. additional external PSC in VMware Workstation check

To make a PSC in a different site, change the guestinfo.cis.vmdir.site-name value to a new site.

And there you have it!

powercli

Manage ESXi SSH Using PowerCLI

Let’s face it. Starting and stopping SSH in ESXi is pain through GUI methods.  I often as a consultant need to connect via SSH to hosts to run data collect scripts, assess NIC and HBA firmware and driver versions, and for troubleshooting purposes, like to run esxtop.  The good news is you can manage ESXi SSH Using PowerCLI.  How cool is that?

Just remember to use get-vmhost to narrow down the specific hosts you want to execute the following commands.

Get the current status of ESXi SSH Using PowerCLI

get-vmhost  hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | select-object vmhost,policy,running

Policy is the start up mode.

  • Automatic = Start automatically if any ports are open, and stop when all ports are closed
  • On = Start and stop with host
  • Off = Start and stop manually

Start ESXi SSH Using PowerCLI

get-vmhost hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | start-vmhostservice -confirm:$false

Note the confirm switch.  If you don’t specify that, it will prompt you.

Stop ESXi SSH Using PowerCLI

get-vmhost hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | start-vmhostservice -confirm:$false

Note the confirm switch.  If you don’t specify that, it will prompt you.

Set startup policy for ESXi SSH Using PowerCLI to start and stop with host

get-vmhost hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | set-vmhostservice -policy "Off"

Be careful if you have any third party products that use SSH.  Nutanix for example comes to mind.  If you goofed and need it set to start and stop with host, just use “On” for the policy parameter.