Tag Archives: vsphere

useful utilities duct tape

Useful Utilities Are Useful

If you’re an IT pro, no matter if you’re an admin, and engineer, a consultant, a PC technician, you have a toolbox of useful utilities, scripts, and software that you use to fix problems.  As time goes by, some of those tools get used more and more.  Others are used less and less for various reasons.  But what surprises me is how many tools in my toolbox on the surface have less and less use cases, but I still come back to them even when it seems I never would need to again.

Over the last few weeks, I’ve been working with a customer who has had significant turnover from consultants they’ve used.  They are moving off a troubled disparate datacenter environment that had over time developed numerous problems to a more consolidated environment that various SyCom resources including me have built for them that is functioning properly, has updated software and firmware, etc.  Along the way, we’ve run into numerous challenges that you wouldn’t normally anticipate.  Troubleshooting them to fix the problems often would take too much time to fix,.  Finding a duct tape solution was more expedient.

I wanted to give a few examples just to illustrate that having a wide knowledge of utilities out there and experience with them can help you solve problems.

In this case, the task was seemingly simple – move VMs running on a legacy NetApp array and vSphere 5.1 servers to a new(er) cluster running vSphere 5.5.  The clusters were managed by two different vCenter servers.  These clusters were within the same physical datacenter.  They had network connectivity between them.  They did not have access to the same storage arrays.  The customer allowed downtime to move them.  Therefore, the easiest way was a shared nothing cold migration (we’re running 5.1 on the source side, remember).  Simple, right?

Doing It the Textbook Way

I approached this like how any vSphere resource would.  Get the two clusters into the same vCenter instance, shut the VMs down, and migrate them cold.  How many times have you seen that fail?  Me?  Pretty much never.  Well, it wouldn’t work.  I’ll spare you the troubleshooting details, but trust me, doing it the native way wouldn’t work.

At this point, the time had come to get creative and bust out some useful utilities I hadn’t used in a long time.  We had to get the job done.  Tick tick!

Useful Utilities #1 – Veeam FastSCP

The customer wasn’t a Veeam customer (yet).  While the customer could take some downtime off hours, there was a limit to that.  We had to move about 2TB of data, so we needed to move this data as quickly as possible without a ton of labor to reconfigure the networks to get both environments access to the storage.

Sure, I could use WinSCP to just bulk copy the VMs over, but Veeam FastSCP, built into Veeam Backup and Replication trial, is free, and it moves data quicker as it disables encryption on the data transfer, which was acceptable to the customer.  I hadn’t had any reason to use FastSCP in probably five years because cold migration functionality and exporting VMs to OVFs and what not within vSphere made it unnecessary.  But here I was, using it yet again.

And sure enough, it worked like a champ.  We tested a quick procedure using it on a few development workloads.  We then proceeded moving all but the critical VMs, and it worked great… except for the last VM of course.  Come to find out, that was a critical SQL VM that the customer didn’t realize was using physical Raw Device Mappings.

Well, shoot, how do we do this one in a quick manner?

Useful Utilities #2 – VMware Converter

For numerous reasons, including perhaps sheer circumstance of projects I’ve worked on, I hadn’t until this had a need to use VMware Converter in years.  Virtualization is so prevalent now, that P2V is one of those things for me that’s like, “Hey man, remember that time we had to convert like 100 physical machines to virtual back in the day?  Good times!”

Also, I’ve generally recommended to customers to avoid converting physical to virtual anyway.  It should generally be seen as a shortcut, but never optimal.  If you could just build a fresh new VM and get the data moved, the resulting VM would be cleaner.  It would probably perform better.  There’s less chance of instability from old drivers and what would inevitably be a significant change in hardware for the OS and application.  Obviously, if you’re dealing with a ton of machines, rebuilding them all isn’t practical.  In that case, you might have to turn to a P2V tool.

But if you got a VM with physical RDMs, you can’t clone the VM.  You can’t bulk copy the Virtual Machine files over.  You could create new VMDKs and copy everything out of the RDM disks to those and reassign drive letters.  However, this SQL VM was nasty with complex mount points and drive letters assigned.  We had to get it done the weekend the RDMs were discovered.

Solution?  VMware Converter!  I tried installing it on an admin server and set up the job.  That of course failed because of Murphy’s Law.  The Converter agent wouldn’t install due to insufficient permissions.  I installed it directly on the SQL VM (with the same account I tried to push the agent, mind you), stopped the SQL services to ensure the data was static, and ran it.  Other than it shuffling a few drive letters around on the converted VM that a few mouse clicks fixed, it worked like a champ.

How about you?  Any useful utilities you’ve used recently you haven’t used in awhile?

Making life easier using vSphere Tags

One of the least used features in vSphere that I think almost all admins could really make use of but don’t is the ability to create custom vSphere tags within vSphere.

I wanted to take the time to point this feature out, and perhaps give people some ideas on how to make use of of them.  This can help with management and automation quite a bit.

What are vSphere Tags?

vSphere Tags are effectively custom metadata type info that can be applied to objects within vCenter.  You get to make your own to fit your own needs.  They assist basically with locating objects for more efficient administration and management.

They’re unique to other things such as folders for your VMs in that you can assign multiple tags to the same VM or other objects.

Let’s break this down by comparing vSphere tags to MP3 management software like iTunes.  An individual MP3 file must be in one file system folder or another.  It can’t be in both.  But suppose you want to find all your songs by an artist, by genre, or by album?  We intuitively understand this now with MP3s.

But we have the same problem with VMs.  You can organize your VMs into VM folders in vCenter, but a single VM can only be in one folder or another.  What if you wanted to organize your VMs by criticality?  By whether or not they have SQL?  Whether or not they need to be backed up?  Trying to do this with folders would be a nightmare to manage.  Plus, remember a VM folder is the mechanism for assigning permissions, too.  Maybe you don’t want this metadata having any impacts on anyone’s permissions to manage it.

That’s when you use vSphere tags!

Use Cases for vSphere Tags

Use cases for this functionality are numerous:

  • Criticality of VM – this would allow the expedited power up or down of VMs based on this nature.  Running out of resources within your cluster due to sudden host failures?  Power down the non-critical VMs.  It would also be helpful for vSphere Admins who aren’t the application admins to know when to handle a VM with care before doing anything to it.
  • Application groupings – Maybe it doesn’t make sense to put VMs that work together to provide an application or service, but you want to know those groups.  That could allow a SQL server that serves the backend of multiple application groups to be identified for both simultaneously.
  • Presence of a common application like SQL – This can be helpful for locating VMs that may require special settings on backup jobs to quiesce the file system before backing the VM up.  You might also use this to find potential VMs that other VMs are dependent on, so you can set their restart priority so they boot up first in an HA event scenario.
  • Lab/Test VMs – You could set the resource allocation for Lab/Test VMs to low to help ensure they are given less resources than production VMs.

OK, I convinced you (hopefully)!   Let’s make some tags.

Basic Concepts for vSphere Tags You Need To Know

You can create vSphere tags in both within the vSphere Web Client and with PowerCLI.  It’s simple, but you need to know a few concepts.

All vSphere tags belong to a Category.  There are two main types of categories.  This notion is called Cardinality.  It sounds more complicated than it is.  Basically, you can have a category where only a single tag from that category can be applied to any given object.  For example, let’s say you want to tag VMs by criticality.  Logically, a VM will only have one criticality rating, not multiple.  IE, it makes zero since for a VM to be both low and medium as far as how critical they are.

However, sometimes you might want a category that multiple tags could apply to the same object.  For example, let’s say you want to make a category called “Special Applications” to identify very specific apps within a VM to easily identify SQL servers, Domain Controllers, and Exchange servers.  While I wouldn’t recommend it, it’s possible for a single VM to be all three simultaneously.

vSphere tags can apply to all kinds of objects as well, not just VMs.  You can select which objects a tag can be applied to within the category.

Managing vSphere Tags Using the Web Client

To create tags within the vSphere client, navigate to the Tags section of the web client.

vm tags web client nav

You must create a category first if there isn’t one already made.  Click the Categories button, and then click the create categories icon.

For this example, we will make a category for criticality ratings for VMs.  We want one tag per object, not more, and we only want the tag to be applied to VMs or vApps.

vsphere category example

Now that we have our category, we can create tags within it.  Click on Tags, and the new tag icon.  Be sure to select the category during tag creation.

vsphere tags create tag example

Rinse and repeat for all the tags you want to create for the category.  One tip I recommend is to name the tags with incuding their category name, which refers to some kind of concept.  Since you usually search by the tag name, you want for example LowCriticality instead of Low.  (See below for search examples.) Low in and of itself could mean a lot of things.  Low resource usage, low criticality, etc.

To apply a tag to an object, simply right click the object, point to Tags & Custom Attributes > Assign Tag…

vsphere tags assign tag

A new dialog box appears where you can filter categories or see all categories and select the vSphere tags you wish to assign.  Also, notice you can remove tags here, too.

Managing vSphere Tags Using PowerCLI

PowerCLI has full tag management functionality within it, too.

Creating a category:

New-TagCategory -Name VMCriticality -Description "Criticality of the VM" -Cardinality Single -EntityType "VirtualMachine","VApp"

Creating a tag:

New-Tag -Name "LowCriticality" -Description "Non-Critical VMs" -Category VMCriticality

Assigning a tag to a VM:

get-vm Shoretel | New-TagAssignment -Tag "HighCriticality"

You can do lots of things with PowerCLI and tags.

Using vSphere Tags

Now that you have tags created and applied, you can now make use of them to make your life easier.

You can make use of tags in both the vSphere Web Client and via PowerCLI.  To find all VMs with a tag within the vSphere Web Client, simply type the tag value in the search box.  The tag name will automatically populate.

vsphere tags searching

Click on it.  Boom, you got your objects with that tag!

vsphere tags search results

There’s also a parameter on PowerCLI’s Get-VM cmdlet to identify the VMs with that tag.  You can then pipe that to another cmdlet.  Say for example you want to shutdown your non-critical VMs because you suddenly experience multiple host failures, so you need to make sure your more important VMs get the resources they need:

Get-VM –Tag “LowCriticality” | Shutdown-VMGuest

Imagine if you set up vSphere tags to identity all your VMs with SQL.  Imagine you’re setting up Veeam backup jobs, and you need to know which VMs you need to setup special quiescing.  You could easily just get that list of VMs.

That’s how to use vSphere tags!

How do you think you might be able to use them, or how do you use them within your environment?

vSphere 6.5 – New features I can’t wait for!

VMware announced vSphere 6.5 at VMworld Europe.   I don’t want to go through everything that’s new, but I do want to go over the vSphere 6.5 new features I think are the coolest that I can’t wait for.

vSphere 6.5 New Features – Me likey!

Here are the vSphere 6.5 new features I specifically wanted to highlight that I think are going to be the most useful to my customers.

vCenter 6.5 New Features

  • The vCenter Server Appliance (VCSA) FINALLY has an integrated VMware Update Manager.  No more Windows machine for VUM!  Even less excuses for using the Windows version!  Speaking of which…
  • Native VCSA high availability!  In vCenter 6.0, the only way to make vCenter truly highly available was to use Windows Clustering.  Not anymore!  Now the VCSA has its own ability.  VCSA NOW AND FOREVER!
  • File-based backup and recovery for VCSA, so it’s even easier to make any kind of recovery you may need.
  • HTML5 based vSphere Web Client!  Take that, Adobe Flash!  No more Flash vulnerabilities and issues to worry about!
  • Fully supported standalone HTML5 based thick client!

Clustering New Features

  • HA Orchestrated Restarts – Now you can enforce a chain of VMs to ensure VM interdependency for multi-tiered applications!
  • Proactive HA – Now you can integrate HA with hardware vendor monitoring tools to move VMs off hosts that have hardware problems before they actually result in an ESXi host crashing.  How cool is that?
  • DRS now takes network bandwidth into account, to ensure your workloads can be dynamically moved between hosts to ensure the best network performance.

Security New Features

I have numerous customers who for legal and other reasons are extremely security conscious.  These may be of particular interest:

  • vMotion traffic encryption – One of the reasons I recommend segregated isolated non-routable VLANs generally for vMotion traffic is due to the fact that vMotion traffic is unencrypted.  Think about the implications of that.  The running contents of RAM for a VM is copied in the clear over a network during a vMotion!  If that’s a VM processing let’s say credit card transactions or personally identifiable information like a Social Security number, that’s pretty scary!  Now consider the boundaries of vMotions have been lifted to the point you can conceivably vMotion a VM across datacenters.  Now, for the first time, you can encrypt this traffic.
  • VM disk encryption – If your shared storage solution can’t encrypt your data at rest, you used to be out of luck for doing whole VM encryption.  Not anymore!  Now it can be done at the VM level!
  • Better logging now to provide better auditing capability to see who did what within the environment.

There’s a whole lot more in this release.  I’m sure I’ll post more about these and other cool features and capabilities soon!

Before you ask, the tentative release date for vSphere 6.5 is Q4 2016.


Manage ESXi SSH Using PowerCLI

Let’s face it. Starting and stopping SSH in ESXi is pain through GUI methods.  I often as a consultant need to connect via SSH to hosts to run data collect scripts, assess NIC and HBA firmware and driver versions, and for troubleshooting purposes, like to run esxtop.  The good news is you can manage ESXi SSH Using PowerCLI.  How cool is that?

Just remember to use get-vmhost to narrow down the specific hosts you want to execute the following commands.

Get the current status of ESXi SSH Using PowerCLI

get-vmhost  hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | select-object vmhost,policy,running

Policy is the start up mode.

  • Automatic = Start automatically if any ports are open, and stop when all ports are closed
  • On = Start and stop with host
  • Off = Start and stop manually

Start ESXi SSH Using PowerCLI

get-vmhost hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | start-vmhostservice -confirm:$false

Note the confirm switch.  If you don’t specify that, it will prompt you.

Stop ESXi SSH Using PowerCLI

get-vmhost hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | start-vmhostservice -confirm:$false

Note the confirm switch.  If you don’t specify that, it will prompt you.

Set startup policy for ESXi SSH Using PowerCLI to start and stop with host

get-vmhost hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | set-vmhostservice -policy "Off"

Be careful if you have any third party products that use SSH.  Nutanix for example comes to mind.  If you goofed and need it set to start and stop with host, just use “On” for the policy parameter.

Log Insight Manager – Install Content Pack

I’m working a bit with VMware Log Insight Manager for the first time, so I wanted to provide people with a taste of this product.  People don’t tend to know about Log Insight Manager, so hopefully this might ease your worries about it.  It is a pretty easy to use product.  If you’re not familiar with this product, Log Insight Manager is a syslog and event log aggregator and analyzer from VMware that helps to parse, analyze, and alert based on queries.

Installing Content Packs into Log Insight Manager

Content packs are the rule sets of what to look for in syslogs that Log Insight Manager is ingesting.  You can make your own.  You can also install content packs from the marketplace as well.  Finally, you can manually import them.

Installing from Marketplace

Installing from the marketplace is easy.  Just click the settings bar in the top right, click Content Packs, and you’ll be taken to the marketplace (in red).

log insight manager install content pack

Unfortunately, this customer has locked down internet access.  I don’t have time at the moment to show how easy this is.  Basically, you can install the content packs right out of this portal, and you’re off to the races.

Importing a Content Pack

This is also very easy.  First off, you would download the Content Pack file.  Once into the Content Packs portion of the web interface, click “Import Content Pack” in the bottom left (in green box).

Browse to the content pack file you downloaded, and select if this should be installed as a content pack, or your own content space.  In this case, I’m installing VMware’s NSX content pack, so I selected “Install as content pack”.  Click import.

log insight manager select content pack

Content packs often give you further instructions.  In this case, the NSX content pack gave instructions to point all NSX components for syslogging to Log Insight Manager.

You should then see the Content Pack listed under Installed Content Packs.

log insight manager installed content pack

And there you have it!

vSphere 5.0 & 5.1 End of General Support Coming

vsphere 5.0 5.1 end of general support

Just a little Public Service Announcement and reminder.  vSphere 5.0 and 5.1 end of general support is coming soon.  End of General Support for ESXi 5.0/5.1 along with vCenter 5.0/5.1 and ancillary products (SRM 5.0/5.1, Data Recovery 2.0, Update Manager 5.0/5.1) is set for 2016-08-28.

What does End of General Support mean?

General Support provides full support of the product, which includes:

  • Phone Support
  • Maintenance Updates
  • Upgrades
  • Bug and Security Fixes

So what happens to vSphere 5.0/5.1 now?

vSphere 5.0 and 5.1 now enter the technical guidance phase.  While phone support is not provided, help can be obtained through a self-help portal.  You can also still receive support and potential solutions for low severity problems.  You can get further details here.

So what should customers running 5.0/5.1 do?

I recommend customers should look to upgrade to vSphere 5.5 or 6.0.  If you have a valid support contract, it shouldn’t cost you anything to upgrade as far as licensing goes.  It’s important to verify your hardware is supported though, including:

  • Servers
  • Storage, whether it be a traditional storage array, or internal controller, or hyper-converged solution
  • I/O cards
    • HBAs
    • NICs
  • Third party products, such as
    • Backup products
    • Orchestration
    • Monitoring
    • VDI

While it’s not absolutely critical to upgrade exactly by 8/28/2016, plans should be made to upgrade.  You don’t want to handle that upgrade in a hurry!

Final tips

If you’re upgrading, be aware that you can mix ESXi versions with vCenter versions.  For this reason, if there’s a specific reason you can’t/don’t want to upgrade to ESXi 6.0, you can upgrade your hosts to 5.5, and run vCenter 6.0, assuming your other products are compatible with vCenter 6.  That might make it easier to upgrade to subsequent versions of vCenter or ESXI down the road.  You might want to consider the vCenter Appliance.

Happy upgrading!

How to backup ESXi servers

Do you backup your ESXi servers?  You don’t technically have to.  If you’ve documented your environment well, and/or you use things like host profiles, Distributed Virtual Switches, and Autodeploy, the need is lessened or even pointless.  However, if you have any manual configurations that need to persist beyond a reboot, you should do it because it’s stupid easy.  Here’s how to backup ESXi servers.

Backup ESXi servers the easy way

This is really easy if you do it with PowerCLI.  Ready?

Get-VMHost | Get-VMHostfirmware –BackupConfiguration –DestinationPath “C:\BackupLocation”

That’s it!  If you used Connect-VIServer to connect to vCenter, this one liner creates within the backup location a file for each server.  Done!

You might want to run this when:

  • You’re about to make any configuration changes.
  • Periodically in case others make configuration changes you’re not aware of.
  • Before you patch your servers.
  • After you patch your servers, and you’ve determined they’re functioning properly.
  • If the ESXi install is on storage that is reporting errors.
  • To migrate ESXi installations to different storage.

This is a great way to get your ESXi servers up and going in the event of a misconfiguration or failure in the ESXi server installation storage.

Restore from backup files

Great!  You have a backup of your ESXi servers.  How do you restore it?

Set-VMHostFirmware -VMHost <IP_or_FQDN> -Restore -Force -SourcePath C:\BackupLocation

If the ESXi server is not on the network anymore, it must be returned to a network connected state first.  You may need to reinstall ESXi first, and configure its management network first.  Then, you can use Connect-VIServer to connect directly to the ESXi server in order to run the above.

Also, be aware that configuration backups can be and almost always are ESXi server version specific.  So ensure you reinstall ESXi using the same version that the server was when the backup was taken.  Ensure you create new backups after you patch your server for this reason.


VMware ESXi 6.0 Express Patch 6 causing CBT issues

The always useful Veeam support digest is reporting that at the very least Veeam is seeing issues with Change Block Tracking (CBT) caused by vSphere 6.0 Express Patch 6.  This build was released on May 12th of this year.  It is the current build according to VMware’s build number KB article.

Veeam is reporting they’re seeing the issue if you’re not using application aware processing and using VMware Tools quiescence on your Veeam jobs.

Other blog articles are mentioning other backup products also impacted, including VMware Data Protection and IBM TSM.  It’s safe to assume this will broadly impact all VMware centered backup products.

If you’re using Express Patch 6, you currently have two options:

  1. Roll back to ESXi 6.0 Update 2.
  2. Don’t use quiesced snapshots.

Heads up!

EMC VSI RecoverPoint/SRM Integration

I’ve recently set a customer up with new VNX storage arrays, RecoverPoint , and it’s all to be integrated with VMware Site Recovery Manager.  Previously, the customer used SRM in conjunction with MirrorView/A.  Why RecoverPoint?

The really cool thing about RecoverPoint is you can easily rollback to specific points in time, as they like to call it DVR functionality for disaster recovery.  MirrorView/A only allows you to rollback to a specific snapshots at specific points in time.

EMC also provides their VSI for VMware environments.  This integrates with many of their storage products, including VNX, RecoverPoint, and it provides the DVR selection ability within SRM if you integrate it as well!

Setup is pretty straight forward:

  1. Deploy the OVA for the VSI in each site.
  2. Login to the VSI’s web portal by hitting https://<ip>:8443/vsi_vum with user name admin and password ChangeMe.  Change the password as prompted.
  3. Install the VSI’s plugin with vCenter by going to VSI Setup and provide the required info.  If you don’t get “The Operation is successful.”, do it again unless you’re provided an error to troubleshoot.  For me, that happened on one of the two vCenter servers I was deploying this on.  Also, be patient, as this can take quite sometime.  For me, the plugin took about 10-15 minutes to complete the installation.
  4. Login to the vCenter Web Client, and go to vCenter Inventory Lists. At the end, you should see an EMC VSI section. emcvsisection
  5. Click on Storage Integration Service.  Under Actions, click Register Solutions Integration Service, and enter the VSI’s info for that vCenter.  Click Test to ensure there’s connectivity to the VSI, and click OK.
  6. Under Storage Systems, add the storage array for that site.  Again, click Test to ensure there’s connectivity to the storage array, and click OK.  VSI supports VMAX, VNX, VNXe, ViPR, and XtremIO, so this isn’t just limited to the VNX on this project.
  7. Under Data Protection Systems, add the RecoverPoint cluster info for that site using the RPA cluster IP address, and be sure to select RecoverPoint as the Protection System Type.  rpprotectionsystemtypeClick Test to ensure communication will work.  If successful, OK will no longer be grayed out.  Click OK.
  8. Repeat step 7, but select SRM this time for the Data Proection System type.  Here’s where I ran into a gotcha.  The FQDN/IP address and port fields were grayed out.  I went ahead and clicked to Test, and got an error: “Could not communicate with the data protection system SRM at <IP of vCenter server>. Details: Cannot reach the target SRM server at <IP of vCenter server>:1” vsisrmregerrorGoogle didn’t yield any results for a solution, so I began troubleshooting.  Thankfully, I knew my ports, and decided to click the check box for the FQDN or IP/Port line, and entered in the FQDN of the SRM server and the port.  Be aware that SRM 6.X uses 9086.  I provided that, clicked Test, got my green “OK to go” text, and clicked OK.

Note that this needs to be done for each vCenter/RPA cluster/storage array/SRM server in the environment.  Note also only one VSI instance can be registered per vCenter server, so you’ll need to deploy one VSI per vCenter.

After setting up each site, go to a VM, click it, go to Manage, view the snapshots for its Consistency Group, click the one you want and apply, and launch your Failover or Test action from SRM.


And there you have it!

VCSA can’t enumerate AD accounts

Ran into an interesting issue.  After deploying greenfield vCenter 6 Server Appliances (VCSA) using an external PSC for a remote branch site, when I tried to do some permissioning with AD accounts.  Joining the PSC to the domain wasn’t a problem, nor was adding the AD domain as an identity source.  But when I tried to enumerate accounts for permissioning, that would fail with the error: “Cannot load the users for the selected domain”.

I found an excellent VMware KB article that gave lots of things to check when troubleshooting this.

I verified DNS was working.  No surprise there.  However, when I ran the command less /var/lib/likewise/krb5-affinity.conf, I noticed the DCs used were not the correct DCs that should be using, rather DCs from a different remote branch office site.  When I checked AD Sites and Services, it was clear that a subnet  object was associated to the wrong branch office that included the IP of the PSC.  Therefore, PSC was attempting to use the DCs in that site.  That’s good to know that vCenter Appliances are apparently AD Site aware.  Furthermore, the first DC used of the two in the remote branch site didn’t have a PTR record because the Reverse Lookup Zone for that subnet for the wrong remote branch didn’t exist.  Apparently, if the first domain controller to be used can be contacted but doesn’t have a PTR record, the PSC won’t enumerate users and groups for permissioning.

Creating the Reverse Lookup Zone and forcing the PTR record creation along with some AD replication fixed the issue, and I kindly suggested to the customer it was time for some tender loving care with AD Sites and Services, along with DNS.

So, FYI, it’s not a bad idea to review your Active Directory Sites and Services, and your DNS Forward and Reverse Lookup zones before you deploy the VCSA.