Compromised vSphere 6.0 Certificates – Part 1

vCenter, vSphere, ,

As I alluded to in a previous post, I’ve been needing to do some more in depth testing in relation to vSphere 6.0, which I run in VMware Workstation.  Now the cat is out of the bag!  I’m running through scenarios about what with compromised vSphere 6.0 certificates.

After scouring the internet, there are plenty of blog articles about how to go with various certificate management models.  There’s not a lot of information what to do if suspect a compromised vSphere 6.0 certificate.

I wanted to cover the basics here discuss more specifics in later articles.  I’m going to start with a short introduction to vSphere 6.0 Certificate Management, and then the implications of each when it comes to a compromised certificate.

vSphere 6.0 Certificate Management Basics

I’ve posted about this in the past.  Here’s a VERY quick recap.

  1. You can have the Platform Services Controller (PSC) act as a root Certificate Authority (CA), and hand out certificates automatically to other vSphere components automatically, which is the easiest to implement and manage.
  2. You can have the PSC act as an intermediate CA, and issue certificates using the intermediate certificate you install automatically.  This is arguably the second easiest to implement and manage.
  3. You can generate Certificate Service Requests to an external CA manually for various vSphere components, and install those certificates manually.  This is arguably the hardest to implement and manage.

One other note here is you can mix and match these options.  This is usually implemented by having non-client internal vSphere component certificates be issued by the PSC, and client facing certs such as the cert for the vSphere Web Client be issued by an external CA.

Again, the above information is not intended to be a primer for certificate management in vSphere 6.0.  It’s only to facilitate discussion about what to do if a certificate has been compromised.

Dealing with Compromised vSphere 6.0 Certificates Issued by an External CA

One advantage of using an external certificate authority to issue the certificates for vSphere is the support for certificate revocation.  If any certificate is compromised that was issued by an external CA, you can simply within that PKI environment revoke the certificate.  Replacing compromised vSphere 6.0 certificates is done the same way the certs was acquired in the first place.

The basic steps would be:

  1. Revoke the suspected compromised certificate within the PKI.
  2. Go through the process of obtaining a new certificate, and install it.
  3. Fix any trust issues that may occur with the new certificate.  For example, you must  manually fix VUM when you change a vCenter certificate.

It’s not so straightforward if the PSC generated the certificate you suspect is compromised.

Dealing with Compromised vSphere 6.0 Certificates Issued by a PSC

For all intents and purposes, it doesn’t really matter if compromised vSphere 6.0 certificates were issued by a PSC using its own root certificate or using an installed intermediate certificate obtained from an external CA.  If the PSC in the end generated the certificate used by a vSphere component, any vSphere component, certificate revocation is not supported.

If you suspect a certificate has been compromised, you have no choice but to regenerate all certificates, even certificates that you don’t expect to be compromised.  This should certainly be considered prior to deciding upon which model to use for vSphere 6.0 Certificate Management.

The basic steps would be:

  1. Run the Certificate Management Utility on the PSC in question to regenerate all certificates.  If any doubts, do this on all PSCs.
  2. Run the Certificate Management Utility on any vCenter server that obtained its certificates from that PSC.  If you’re running embedded PSC with vCenter, you already did this in step one.
  3. Fix any trust issues that may occur with the new certificate.  For example, you must  manually fix VUM when you change a vCenter certificate.

If the above seems like it’s just as easy, it isn’t.  For one, documentation on how to do this with external PSC’s is vague and confusing from VMware.  Secondly, it gets more complicated the more PSC and vCenter nodes you have.

I’ll go in more depth on how address compromised vSphere 6.0 Certificates issued by a PSC in Part 2.  In Part 3, I’ll address how to fix trust issues with certificates in various products.