desk treadmill

Desk treadmill activity review – 2016

It’s a new year!  I know I haven’t mentioned my walking status for awhile, mainly because I forgot to.  My desk treadmill also had numerous problems, too.  After working with Lifespan extensively along with quite a bit of arm twisting on my TR1200-DT3, they finally agreed to upgrade my desk treadmill to the TR5000-DT3 model at a reduced price.  Basically, the motor kept going out, despite maintaining it properly, and my use being well within the specifications. The TR5000 has a more powerful motor.  So far, the new desk treadmill runs great.  However, I didn’t encounter issues with my TR1200-DT3 until about six months in, so the jury is still out.  I’m going to post a separate piece about my issues, in case it helps others.

I’m writing this post now without any idea about how many steps I actually walked in 2016.  I fully expect it to be lower though because of the above problems.  My treadmill has down for half the year.  I can’t do much about that, and how long it took Lifespan to get a solution.

As I did before, I’m compiling a list of all my weekly progress of steps tracked by my Lumo Lift, which helps me keep good posture that helps avoid neck pain, but also tracks the number of steps I take surprisingly accurately.  Roughly, 2,000 steps = 1 mile.

Obviously, not all walking was done on my desk treadmill, but a lot of it was.  When you see dips in weekly steps, that’s likely a business trip, vacation, I exercised some other way other than anything that would trigger detected steps, or I was outright lazy for whatever reason.  The prolonged downturn in the spring through fall was due to the treadmill breaking down.

Week (ending of)StepsMilesNotes
Total33279451640
1/1011032855
1/1711026755
1/2412103960
1/318339541
2/710225251
2/149996049
2/215303326Business trip
2/2811231256
3/66218131Business trip
3/136537432Business trip
3/2010373251
3/2811486857
4/311640858
4/1011806059
4/172958714Death in family
4/246831934Treadmill broke down
5/186614
5/83162115
5/154076320
5/223778518
5/294361421
6/53539217
6/123086915
6/192004310
6/262680613
7/33262916
7/10174398
7/17199949
7/245088425
7/31143977
8/72403812
8/142067210
8/212717013
8/287455837
9/44643623Vmworld, forgot Lumo charger
9/11129886
9/183108915
9/253580717
10/210803254Treadmill finally replaced!
10/911185755
10/164291921Treadmill broke again!
10/233339216Vacation
10/304468022Business trip
11/66319731Treadmill replaced!
11/137737238Business trip
11/209490647
11/279339946
12/410059050
12/119256546
12/1810776453
12/2510227051
1/110023250

You can clearly see that having an operational treadmill makes a huge difference!

Well, my step count is significantly lower than 2015.  I don’t feel too bad about that though because the treadmill broke down multiple times.

Still, 1640 miles in a year is quite a lot!  To give you an idea, that’s about the distance from Richmond, VA to Denver, CO!

I gained some weight unfortunately.  2016 was a very stressful year, and a lack of a convenient method of exercising while working didn’t help.  I do need to eat better.  That’s my goal for 2016.

Assuming my desk treadmill doesn’t break down again, I’m setting a goal of 2500 miles.  That’s roughly the distance from Richmond, VA to San Diego, CA.  I also am now going to try to couple walking with eating right to see how my weight does.  Just from my experiences in 2015 and 2016, walking this much alone doesn’t seem to allow me to lose weight.  Not that I ate terribly, but I didn’t watch what I ate closely either.

And as I finish this post, I just completed my 7.5 miles of walking for the day!  364 more days of walking to go!

VCP5 recertification featured image

VCP5 recertification backdoor

Recently, I satisfied my VCP5 recertification requirement of every two years by achieving VCP6-NV.  If you are not aware, achieving VCP in a different track is one of several ways to qualify to start the clock over again on your existing VCP certifications.  However, I was really cutting it close and failed my first attempt on my VCP6-DV.  With a mandatory one week waiting period to take the exam, I basically had two more shots to pass it, so I began looking at any other avenues to keep my existing VCP certifications.  I did stumble into one, so here’s a potential VCP5 recertification backdoor that could help get you out of a jam of losing your VCP certifications.

VCP5 Recertification – What’s this all about?

If you haven’t heard, VMware recently made it a requirement to recertify every two years.  If you take no action, you lose any and all VCP certifications.  As an example, I am a VCP3/4/5 in the Data Center Virtualization track.  All of those certifications would have expired had I took action this month.

You can satisfy the requirement to recertify in a variety of ways:

  • Pass again the VCP exam for the current exam you’re currently certified in within the same version.  For example, if you’re VCP5-DV, you could pass the VCP 5.5 exam again.
  • Pass the VCP exam within the same track in a newer version.  IE, if you’re a VCP5-DV, you could pass the delta or full exam for VCP6-DV.
  • Achieve a VCP certification in a different track.  This was the route I took.  By achieving VCP6-NV, I recertified my VCP3/4/5-DCV.
  • Achieve a VCAP or VCDX certification in the same or different track.  This was going to be my original path, as I was targetting VCAP6-DCV Deploy, but I got sidetracked unfortunately with other certifications and various things (Nutanix NPP, EMC Unity, VMware VCA6-Hybrid Cloud, and VCP6-NV).

Note that any of the above paths cost significant money.  A VCP exam costs $225, and a VCAP exam is over $400 without any discounts.  You also need to take the time to take a proctored exam at a testing center, too.

VCP5 Recertification – Potential Backdoor!

However, I did find a potential backdoor to take care of your VCP5-DCV recertification.   Note this officially works for people who are VCP5-DCV and achieved that certification through exam VCP510 only.

The VMware Certified Professional 5 – Data Center Virtualization Delta Exam apparently is still available.  VMware announced numerous times they would be ending its availability but never did.

vcp5 recertification delta exam

I thought this exam was no longer available, as it was announced to be discontinued and extended a few times, but finally was to be discontinued on 3/31/16. But there it is! I’ve clicked to register for it all the way up to the point you pay for it with Pearson Vue, and nothing is stopping me, so it appears to be a valid way if you’re a VCP5 through the VCP510 exam (so that presumably excludes the VCP550 exam) to extend your VCP status another two years.

If you’re not familiar with this exam, it’s taken at home, open book/note, with no waiting period if you fail, AND it’s cheaper than a VCP exam on top of all that.

Word of warning though – VMware has begun to expire certification exams for vSphere 5. For example, VMware is discontinuing VCAP5-DCV exams come November. I would assume VMware will retire all vSphere 5 exams soon. So, if you want some insurance and reset your two year clock, you might strongly consider doing this exam sooner rather than later while you still can if you took VCP510.

Also, FYI, there is a VCP6-DCV delta exam as well. However, Pearson Vue proctors this $225 exam just like a VCP6-DCV exam is.  I’m not entirely sure what the difference is between them. It would net you VCP6-DCV, though compared to the VCP5 delta exam.

Which way are you planning to recertify?

vSphere 6.5 – New features I can’t wait for!

VMware announced vSphere 6.5 at VMworld Europe.   I don’t want to go through everything that’s new, but I do want to go over the vSphere 6.5 new features I think are the coolest that I can’t wait for.

vSphere 6.5 New Features – Me likey!

Here are the vSphere 6.5 new features I specifically wanted to highlight that I think are going to be the most useful to my customers.

vCenter 6.5 New Features

  • The vCenter Server Appliance (VCSA) FINALLY has an integrated VMware Update Manager.  No more Windows machine for VUM!  Even less excuses for using the Windows version!  Speaking of which…
  • Native VCSA high availability!  In vCenter 6.0, the only way to make vCenter truly highly available was to use Windows Clustering.  Not anymore!  Now the VCSA has its own ability.  VCSA NOW AND FOREVER!
  • File-based backup and recovery for VCSA, so it’s even easier to make any kind of recovery you may need.
  • HTML5 based vSphere Web Client!  Take that, Adobe Flash!  No more Flash vulnerabilities and issues to worry about!
  • Fully supported standalone HTML5 based thick client!

Clustering New Features

  • HA Orchestrated Restarts – Now you can enforce a chain of VMs to ensure VM interdependency for multi-tiered applications!
  • Proactive HA – Now you can integrate HA with hardware vendor monitoring tools to move VMs off hosts that have hardware problems before they actually result in an ESXi host crashing.  How cool is that?
  • DRS now takes network bandwidth into account, to ensure your workloads can be dynamically moved between hosts to ensure the best network performance.

Security New Features

I have numerous customers who for legal and other reasons are extremely security conscious.  These may be of particular interest:

  • vMotion traffic encryption – One of the reasons I recommend segregated isolated non-routable VLANs generally for vMotion traffic is due to the fact that vMotion traffic is unencrypted.  Think about the implications of that.  The running contents of RAM for a VM is copied in the clear over a network during a vMotion!  If that’s a VM processing let’s say credit card transactions or personally identifiable information like a Social Security number, that’s pretty scary!  Now consider the boundaries of vMotions have been lifted to the point you can conceivably vMotion a VM across datacenters.  Now, for the first time, you can encrypt this traffic.
  • VM disk encryption – If your shared storage solution can’t encrypt your data at rest, you used to be out of luck for doing whole VM encryption.  Not anymore!  Now it can be done at the VM level!
  • Better logging now to provide better auditing capability to see who did what within the environment.

There’s a whole lot more in this release.  I’m sure I’ll post more about these and other cool features and capabilities soon!

Before you ask, the tentative release date for vSphere 6.5 is Q4 2016.

Resolving VM MAC Conflict alarm with Veeam Replicas

It’s been awhile since I’ve deployed Veeam using replication with vSphere 6.0.  I recently implemented it for a customer who was replicating VMs to a secondary storage appliance in addition to backing the VMs up to a Data Domain.  Upon running the initial replication for the VM, a “VM MAC Conflict” alarm triggered on the replica VM.

vm mac conflict alarm triggered

Here’s a description of what’s going on and how to prevent the VM MAC Conflict alarm from triggering.

VM MAC Conflict Alarm

The VM MAC Conflict alarm is new to vCenter 6.0 Update 1a.  The intent of the alarm is to warn you if two vNICs on VMs within a vCenter instance have the same MAC address.  This can happen for a variety of reasons:

  1. vCenter malfunctioned and dynamically provided the same MAC address to two or more vNICs.
  2. Either intentionally or mistakenly, an admin or a third party product might have statically assigned a MAC address already in use within the environment.  In this case, Veeam created a copy of the VNX file with identical MAC addresses for the source and replica VM’s vNICs.

It’s a good alarm to have to notify you just in case.  But how do you keep this alarm while stopping it from triggering on replica VMs?

Stopping VM MAC Conflict Alarms from triggering for Veeam Replicas

The solution for preserving the VM MAC Conflict alarm while stopping it from triggering on Veeam replicas is quite simple.  You can modify the alarm itself by setting an exception to exclude VMs.  In the case of Veeam replicas, they have a “_replica” suffix within the VM name by default.  If you changed that suffix in the replica job, just adjust accordingly.

Go to the VM MAC Conflict alarm definition.  It’s in the vCenter inventory object under Manage > Alarm Definitions.  Click the alarm and on the right, click Edit.

Under the bottom box that reads, “The following conditions must be satisfied for the trigger to fire”, add a condition that says the VM name does not end with “_replica”.  Once applied, the alarm disappears for your replica VMs.

vm mac conflict alarm modified

That’s it!

White box home computers – Am I alone?

I started really getting into computers, which eventually led me to IT (duh!), just as I was entering college back in 1995.  Not knowing any better, I bought a piece of crap AST computer, which was a Pentium 60, and 8MB of RAM.  It had no 3D accelerator, a 540MB hard drive, and was slow, despite it being one of the first Pentiums around.

I got into PC gaming, starting with Doom, and it grew to other games.  I also ended up finding a mom and pop computer shop locally in Richmond, VA, a trustworthy source for computer upgrades.  I ended up working for them eventually to pay for college.  I attempted to a few upgrades on the AST, before determining it was a piece of crap.  I learned how much better a white box could be, and that was that.  Short of laptops or mobile devices, that was it, I was a white box guy from then on.

I still, to this day, build my own computers.  There’s something about being able to research each part and buying the one that’s going to work best for you.  I know it takes time to do that, but the end result to me is better.  It’s great to know you can replace any part in it.  It’s great to build in the capability to upgrade.  It’s great to build in reliability.

But there are definite downsides.  It takes time to research all those parts.  It takes time to build it, install the operating system, etc.  It takes time to be your own tech support.  I notice other bloggers generally speak of various pre-built machines they bought.  I’m guessing that maybe why.

So my question to the community is do you still white box your own personal home computers?  Why or why not?

2v0-641 exam review

VCP6-NV 2V0-641 Exam Review and Resources

Sorry the updates haven’t been forth coming.  That’s mainly because my two year requirement for recertification was coming up for my VCP-DCV certification, and I was preparing to meet it by achieving VCP6-NV.  Saturday, I passed, so I wanted to share my experiences and provide a 2V0-641 exam review.

2V0-641 Exam Review – Thoughts on Exam

This exam is no slouch in difficulty.  I’ve ready plenty of reviews that seem to suggest the VCP-NV exam isn’t difficult, but maybe that was the predecessor to 2V0-641.  I made my first attempt a week before, and failed it despite putting a lot of time studying, and soaking in as much as possible using the Hands on Labs from VMware.

My first score was 276, and you must score 300 to pass.  I think part of the reason why was contained in my score report, which lists objectives recommended for review if you fail.  I noticed a string of questions that didn’t pertain to what the exam blueprint described for various network fabrics.  The blueprint had stated to know the challenges of various network designs, and how NSX solved them, but these questions were more about how to configure NSX when you have various network fabric design types.  Sure enough, one objective listed that I needed to review was “Describe considerations for running VMware NSX on physical network fabrics”.  As of the time I’m drafting this blog post, that is nowhere on the exam blueprint.  Needless to say, I was extremely frustrated, with another 2 weeks to go before I lost my VCP status.

After dusting myself off, I went back to studying and gathered information for this new objective, and scored 357 on my second attempt with one more week to go before my VCP-DCV expired.  A little too close for comfort, but I got it done.

2V0-641 Exam Review – How to Prepare

I attended an online version of the 6.2 Install and Configure Course back in August.  Be very aware that this course alone is absolutely not sufficient to pass this exam.  I can’t stress that enough.  You’ll know just by looking at the exam blueprint, as entire major exam objectives are completely not covered at all.  The course covers others, but only scratched the surface.  You absolutely have to know your stuff for this exam!

I highly recommend the VMware Press Official Cert Guide for VCP-6-NV by Elver Sena Sosa.  I had the pleasure of meeting him at VMworld 2016 this year in Las Vegas.  It’s one of the best written IT books I’ve ever read, and comes with simulation exams. However, it too is also not sufficient for preparing for this exam, either.  But it helped me understand how NSX works that I didn’t even realize I was short on despite attending the course.  I found the packet walkthroughs for both the logical switch and logical routers extremely helpful!  However, the book also doesn’t cover some sections for this exam in the blueprint.  The simulation questions also do not reflect the difficulty or the style of questions I found on both attempts on the exam well.  I scored consistently almost perfectly on these simulation exams on the first attempt prior to taking the actual exam, and yet failed anyway.  Still, this is in my opinion the most helpful resource for this exam.  In fact, I’m quite certain I’ll reference this book in future NSX deployments.

Finally, I really wanted to give a shot out to another blogger, Rich Dowling, who helped me immensely with his notes on various portions on the exam objectives listed in the blueprint.  While it’s not for the current exam, most of the exam objectives are the same. Make sure you check them out if you’re taking this exam!

Also, make sure you read carefully through the recommended documents within the exam blueprint.  This is especially the case for NSX Design Guide!  (see surprise objective!)

2V0-641 Exam Review – Resources

Here are some resources to help with your 2V0-641 exam review.

VMware Certified Professional 6 – Network Virtualization Exam page – Contains the exam blueprint

VMware NSX Network Virtualization Design Guide

vSphere Networking Guide

NSX for vSphere Administration Guide

NSX Command Line Interface Reference Guide

VCP6-NV Official Cert Guide (Exam #2V0-641)  (Kindle Version)

2V0-641 Exam Review – Check back for more

I’ll be posting some section reviews where I think my notes would be helpful for others beyond Rich Dowling’s excellent notes to try to help others as Rich helped me.  I hope you’ll find those useful!

Compromised vSphere 6.0 Certificates – Part 3

This is the final part of my series on how to deal with compromised vSphere 6.0 certificates.  If you are coming here first, I highly recommend reading:

Compromised vSphere 6.0 Certificates – Part 1

Compromised vSphere 6.0 Certificates – Part 2

We pick up where we left off in Part 2.  The scenario is there are suspected compromised vSphere 6.0 certificates in your environment that were provided to vCenter issued either via a root certificate within the Platform Services Controller (PSC), or the PSC generated certificates with an installed intermediate certificate from an external Public Key Infrastructure (PKI).  Again, VMware does not support certificate revocation when its PSC automatically generated the cert using either its own root or via the external PKI’s issued intermediate.  You must then regenerate all certs.

At this point, I assume certificates were regenerated for the PSC’s root or the intermediate certificate, along with all vCenter server certificates.  I outlined that process in Part 2.  Now, the question is what to do about everything else when dealing with compromised vSPhere 6.0 certificates?  What about ESXi servers?  What about external products that plug into vCenter like VMware Update Manager?  NSX Manager?  vRealize Operations Manager?

Let’s get to it!

Compromised vSphere 6.0 Certificates – ESXi servers

After resetting all the certificates within the PSC and vCenter, I have good news when it comes to your ESXi servers.  They won’t have any problems.  Resetting VCSA certificates has nothing to do with ESXi servers because they do not obtain certificates from VCSA, nor do ESXi servers have any trusts of the certificates that were reset.

Yay!

Compromised vSphere 6.0 Certificates – Most external vCenter dependent solutions from VMware

In most cases, with it comes to external vCenter products that establish relationships with vCenter, these products do often establish a trust of one of the certificates that were reset.  However, they do not obtain certificates from a PSC themselves.  Therefore, you need to fix these products by simply establishing trust with the new certificate that is now installed with vCenter.   And in most cases, this is as easy as it was when you registered it with vCenter in the first place.

I’m not going to show step by step of every product, as I don’t have the time.  I will however come back and update this post if/when I need to do this with various products.  I am going to use vRealize Operations Manager as an example of a typical product that is fixed the same basic way.

vRealize Operations Manager

Here is what vCOPS looks like following the PSC/vCenter certificate resets:

compromised vsphere 6.0 certificates vcops statusNote we are checking the same place you go to register the product with vCenter in the first place (Appliance portal > Solutions > VMware vSphere).  If we were talking about NSX, you would login to the NSX Manager’s direct portal and navigate to Manage Appliance Settings > NSX Management Service > Configure.  This is the same basic concept, even though where you navigate might be different on the product.

Go into settings and re-establish the connection.  In vCOPS, that means clicking the gear on that page, and on both the vCenter Adapter and vCenter Python Actions Adapter, click “Test Connection”.  Low and behold, a pop up comes up to ask if you wish to trust a new certificate it doesn’t yet trust.

compromised vSphere 6.0 certificates trust new certs

If you click OK to trust, vCOPS adds the new certificate to its trusted store.  However, you get an error that you effectively can’t trust two certificates for the same object.

compromised vSphere 6.0 certificates vCOPS trust new cert error

I show this just in case other products share similar behavior.  Delete the old trusted certificate from the appliance.

In this case, navigate to Certificates, and delete the trusted certificate.

compromised vSphere 6.0 certificates vCOPS delete old untrusted cert

Hovering over a column gives more specific info for that cert, which can help identify which certificate to delete.

Then, go back and issue the Test Connection command.

Stop and start the collections on the Solutions page as needed.

compromised vSphere 6.0 certificates vCOPS data receiving

Click refresh and wait to ensure  “Data Receiving” is shown for the collection status.  Otherwise, vCOPS is not functioning.

Other products will have their idiosyncrisies, but they have the same basic concept of establishing trust for the new vCenter certificate.  You perform this process pretty much where you registered the product with vCenter in the first place.

Compromised vSphere 6.0 Certificates – Abnormal external vCenter dependent solutions from VMware

Some products need specialized procedures to trust the new certificates installed in your vCenter/PSC servers.  Here are all of the ones so far I’ve run into, and how to fix those:

vCenter Update Manager (VUM)

I’m hardly shocked VUM would need a specialized procedure.  VUM runs on a Windows OS only.  It remains 32-bit, as opposed to almost every other VMware product.  Plus, it has had esoteric procedures when it came to certificates for a long time.

Navigating to an impacted VUM server within the vSphere Client nets you a pretty immediate error that clearly shows a problem with the SSL certificate.

compromised vSphere 6.0 certificates VUM SSL error

“sysimage.fault.SSLCertificateError”

Time to fix the trust of the new certificate!

First, remote into the VUM server.

Next, run the VMware vSphere Update Manager Utility under the installation directory for VUM (X:\Program Files (x86)\VMware\Infrastructure\Update Manager\VMwareUpdateManagerUtility.exe, where X is the drive in which you installed the VUM binaries).  Login, and select to re-register with vCenter.

compromised vSphere 6.0 certificates reregister VUM

Of course, restart vSphere Update Manager service to complete the process.

The error will go away, and VUM will function again.

Summary

Hopefully, this gives everyone enough info to complete the process or point them in the right direction.  If you have any insights to other products I didn’t cover, please post in the comments!  As I try more products, I will also update this article.

Thanks for reading!

vCenter to VCSA migration tool released

It’s out!  You can finally migrate a Windows vCenter install to a vCenter appliance VCSA based install using a supported VCSA migration tool!

vCenter Server Migration Tool: vSphere 6.0 Update 2m

Just a heads up, here are the requirements:

  • It only migrates from Windows installations to VCSA based installations
  • It only migrates from 5.5 (any revision) to VCSA 6.0 Update 2
  • You must used the embedded database in VCSA
  • It will not migrate VUM if VUM is installed on the same OS as your vCenter (no big deal IMO, not hard to reinstall VUM)

Cool things I noticed about this migration utility according to the link above:

  • Preservation of original vCenter settings including
    • IP address
    • FQDN
    • Certificates
    • Alarm settings (AWESOME!)
  • It doesn’t touch your current vCenter’s stuff, so easy rollback.

Not so cool things:

  • You can’t change deployment models during the migration.  A simple deployment in 5.5 becomes an embedded deployment in VCSA, which you don’t want to do, but you can always transition out of that.

I’m sure you’ll be seeing a blog article here kicking the tires on this thing.  But this will definitely help me convince people to go to the VCSA.

unity

How To Manage Unity Host LUN IDs

EMC’s new Unity arrays have been out for a bit now.  Every array has its weirdness to it.  I just found my first Unity oddity today.  Remember, the Unity replaces the VNX and the VNXe.  If you’ve never played with a VNXe, EMC simplified the Unisphere interface, and removed and/or hid many options in the process.  With that in mind, the Unity does this, too.  You will find this perhaps first managing Unity Host LUN IDs.

Why might you want to control the host LUN IDs?

  • Consistent LUN Masking for easier troubleshooting.
  • Storage based replication like RecoverPoint best practices want consistent Host LUN IDs for the replicated LUNs for both host access and RPAs.
  • Boot from SAN LUNs are either required to be a specific Host LUN ID such as LUN 0, or you may be running Cisco UCS and need to specificy a specific Host LUN ID for the boot LUN in the boot policy.

In this case, the customer decided to reinstall ESXi on a new boot LUN.  They created a new boot LUN, granted the correct ESXi host access, and then deleted the old one.

Provisioning LUNs

Provisioning LUNs on an EMC Unity array is easy, at least in most cases.  EMC streamlined the interface on these arrays.  You simply go to Storage > the subcategory Block or VMware if it’s for ESXi, and step through the wizard to set your options and grant host access.

unitycreatelun

When you get to the point where you grant access, you simply put checkmarks in the boxes for each host to which you wish to grant access.

unitygrantlunaccess

Do you see anything missing, even if you click that gear and add columns?  If you guessed there’s no way to control what host LUN ID will be used in the LUN masking, go pat yourself on the back.

Fortunately, the Unity does provide a way to set the Host LUN ID once a LUN is created… in most cases…

Managing Unity Host LUN IDs for LUN Masking

Here’s how Unity Host LUN IDs work.  Unity automatically picks a host LUN ID for the host, picking the next available host LUN ID for that host.  If you wish to change the host LUN ID for a LUN mask, simply navigate to Access > Hosts, click on the host you wish the change the Host LUN ID for, the pencil icon to edit.  Next, click LUNs.  Finally, click “Modify LUN IDs”.

change unity host lun id

Easy right?  As long as your host isn’t identified as a VMware host, you’re good!

Managing Unity Host LUN IDs for LUN Masking on VMware Hosts

If you follow those directions but your host is identified as a VMware host within Unity, you’re in for a nasty surprise.  Let’s play the game, “what’s missing?”

unity host lun id missing

If you guessed “no way to change Host LUN IDs”, you’re correct again.  YOU’RE ON FIRE!!!  That’s not all, though.  There’s no LUN access management here at all!  That includes Host LUN IDs.

Don’t bother going into the initiators either.  It’s not there.  It’s not under the LUN in question, either; although, that’s how to manage LUN access in general for VMware hosts.  Other than command line, there’s nothing you can do with identified VMware hosts.

Solution?  Don’t make it an identified VMware host.  Simply remove the vCenter server from Unity.  That places all ESXi servers discovered from this vCenter into the general Hosts group, giving you back the ability to change the Host LUN IDs.

unity vcenter server removed esxi hosts

Now, you can manage the Host LUN IDs again!

Compromised vSphere 6.0 Certificates – Part 2

In this second blog article, I discuss what to do with compromised vSphere 6.0 Certificates issues by a PSC to vSphere components.  As mentioned in the previous blog article, you cannot revoke certificates issued by a PSC either using an installed intermediate certificate from an external CA or using its own root.  You must regenerate all certificates instead.

FYI, this post assumes you’re using the VCSA.  Windows installable vCenter is nearly identical, aside from the path to Certificate Manager.

Compromised vSphere 6.0 Certificates – Embedded PSC With Own Root Certificate

If you have compromised vSphere 6.0 certificates automatically generated from an embedded PSC, you must regenerate all certificates.  Yes, you must regenerate even certificates you don’t suspect, too.

To do this:

  1. Login as root into your embedded vCenter server via console, SSH, etc.
  2. Enter into shell.  If you didn’t enable shell via the console, you can run “shell.set –enable True” and then run “shell”.
  3. Run the certificate manager utility.  For the VCSA, you simply run /usr/lib/vmware-vmca/bin/certificate-manager
  4. Select option 4 – Regenerate a new VMCA Root Certificate and replace all certificates.
  5. Certificate Manager asks for various pieces of information for each certificate regeneration such as the country, organization, OrgUnit, State, Locality, email, etc. These are cosmetic values mostly, and are only visible if someone really examines the certificate.  Functionally, they make no difference.  However, I wanted to call your attention to a couple of things that are very important. It is VERY CRITICAL you do the following for each certificate, or else the process will fail!
    1. There is a bug in the certificate automation tool, where if you answer identical values for all questions asked, the same certificate will be generated for that cert.  You’ll notice there are multiple certs that end up being regenerated.  You can tell which one is being regenerated with the following line: “Please configure root.cfg with proper values before proceeding to next step.”  That means the root certificate is being regenerated.  You’ll see various certs as well like “machine”, “machine-ssl”, “vpxd.cfg”, etc.   Each one of these certs must actually be unique.  Ensure that you give at least some different value for one of the questions asked for every cert regenerated for a server.  By far, the easiest way to do this is to answer the following question uniquely for every cert: “Enter proper value for ‘Name’ [Default value : CA]”  Simply name it an abbreviated name of the server and the certificate name.  In this case, you could call it “VC-ROOTCFG”.  Answering every other question identically won’t hurt.
    2. One question that is more than cosmetic that you must answer correctly is: “Enter proper value for ‘Hostname’ [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com]”.  Make sure this is the actual DNS name for the vCenter server.
  6. When prompted afterregenerating all certificates, stop and start all services using:
    1. service-control –stop –all
    2. service-control –start –all
  7. I would recommend rebooting your vCenter server now.
  8. Download your root certificate again and reimport into GPO or however you established trust on the clients for the root originally.
  9. Fix all trust issues with external products.  (See part 3 of this series!)

This is probably the one time you might actually want an embedded PSC for vCenter.  This is far simpler than if you have an external PSC.  (I still recommend external PSC’s in all cases for the record!!!)

Compromised vSphere 6.0 Certificates – External PSC(s) With Own Root Certificate

This is somewhat similar.  However, keep in mind each PSC is a CA.  Therefore, you probably should do this on every PSC that’s a part of the same environment if you suspect certificate(s) have been compromised.

To do this:

  1. Login as root into your external PSC server via console or SSH.
  2. Enter into shell.  If you didn’t enable shell via the console, you can run “shell.set –enable True” and then run “shell”.
  3. Run the certificate manager utility.  For the VCSA, you simply run /usr/lib/vmware-vmca/bin/certificate-manager
  4. Select option 4 – Regenerate a new VMCA Root Certificate and replace all certificates.
  5. Certificate Manager asks for various pieces of information for each certificate regeneration such as the country, organization, OrgUnit, State, Locality, email, etc. These are cosmetic values mostly, and are only visible if someone really examines the certificate.  Functionally, they make no difference.  However, I wanted to call your attention to a couple of things that are very important. It is VERY CRITICAL you do the following for each certificate, or else the process will fail!
    1. There is a bug in the certificate automation tool, where if you answer identical values for all questions asked, the same certificate will be generated for that cert.  You’ll notice there are multiple certs that end up being regenerated.  You can tell which one is being regenerated with the following line: “Please configure root.cfg with proper values before proceeding to next step.”  That means the root certificate is being regenerated.  You’ll see various certs as well like “machine”, “machine-ssl”, “vpxd.cfg”, etc.   Each one of these certs must actually be unique.  Ensure that you give at least some different value for one of the questions asked for every cert regenerated for a server.  By far, the easiest way to do this is to answer the following question uniquely for every cert: “Enter proper value for ‘Name’ [Default value : CA]”  Simply name it an abbreviated name of the server and the certificate name.  In this case, you could call it “PSC1-ROOTCFG”.  Answering every other question identically won’t hurt.
    2. One question that is more than cosmetic that you must answer correctly is: “Enter proper value for ‘Hostname’ [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com]”.  Make sure this is the actual DNS name for this server.  Even if it asks for a cert for the web client, do NOT put in the name of the vCenter server.  It will also ask for an optional IP address.  Obviously, if you input one, make sure it’s the correct one.
  6. When prompted after regenerating all certificates, stop and start all services using:
    1. service-control –stop –all
    2. service-control –start –all
  7. I recommend rebooting the machine when you’ve completed this.
  8. To verify the PSC cert reset worked, attempt to go to https://FQDNofPSC.domain.com/psc to ensure you get a login prompt.  If you don’t, the certificate reset failed.  Stop and redo this portion again.  You likely didn’t provide some kind of different answer to one of the questions for each certificate to make them unique.
  9. Run Certificate Manager on your vCenter server(s).  Here’s where it gets weird.  VMware says you should run Option 3 – Replace Machine SSL certificater with VMCA Certificate and answer the questions.  Next, run Option 6 – Replace Solution user certificates with VMCA certificates.  That didn’t work for me.  The only way I could get it to work is run Option 8 – Reset all certificates.  That’s the only way I could get it to work.  I found another oddity.  During this process, you are asked: “Performing operation on distributed setup, Please provide valid Infrastructure Server IP.”  If I entered an IP address, and did the rest correctly (remember to answer the questions but provide a different value for name for each certificate!), the process would kick off, get stuck at a long time here and eventually fail:Status : 85% Completed [starting services…]
    Error while starting services, please see log for more details
    Status : 0% Completed [Operation failed, performing automatic rollback]

    Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

    Then the certificates would roll back.  Enter the FQDN of one of your PSC servers instead!  That allows it to continue.

  10. Download your root certificate again and re-import into GPO or however you established trust on the clients for the root originally.
  11. Fix all trust issues with external products.  (See part 3 of this series!)

This is far more complicated than the first one, but it’s probably the one you’re more likely to need to do.

Compromised vSphere 6.0 Certificates – Intermediate CA

If you installed a now compromised intermediate CA certificate, revoke the intermediate certificate within the external PKI.  You should then request and install a new intermediate certificate within the PSC.  Then proceed with regenerating certificates for all other components. (See above…)

And that’s how you deal with compromised vSphere 6.0 Certificates.  In part 3, I’ll delve into how to fix trust issues with various products that might arise from regenerating these certificates.