Monthly Archives: November 2017


Conquering ESXi upgrades with conflicting VIBs using PowerCLI

Today, I ran into an issue where I was upgrading ESXi 6.0 servers to 6.5 Update 1 using an HPE custom ISO.   Here’s another example of how PowerCLI can make you more productive.

Conflicting VIBs problem

While working with a customer on a vSphere 6.0 to 6.5 upgrade, I prepared everything as it should be.  I got the latest custom ISO from HPE for ESXi 6.0 Update 1, created a VUM baseline, and attached it to the clusters in question.  Upon scanning the ESXi hosst with VMware Update Manager, I received a warning that the HPE custom ISO was incompatible.

esxi upgrade conflicting vibs

Note that there aren’t actually four conflicting VIBS.  It repeated the problematic modules twice.  There’s actually only two.

Basically, these conflicting modules should be removed prior to upgrading the ESXi hosts.

Removing conflicting VIBs the manual way

There’s nothing special about how to remove them via ESXCLI.  You need the name of the conflicting module, and enable SSH on the ESXi hosts.  Then, run the following command:

esxcli software vib remove –vibname conflicting-vib-name

In the case above, they are named scsi-qla2xxx and scsi-lpfc820.

Watch for indications if the server needs to be rebooted when you run the command.  If so, reboot the servers before proceeding with the upgrade.

Removing conflicting VIBs the PowerCLI way

It’s even easier with PowerCLI to remove these conflicting VIBs.  You don’t have to enable SSH on all your ESXi hosts.  First, make a text file with the names of each conflicting VIB name, with one name per line.

Next, run the following commands after connecting to your vCenter server via PowerCLI:

$modules = gc c:\scripts\modulesnames.txt

$esxi = get-vmhost ""
$esxcli = get-esxcli -V2 -VMhost $esxi
$modules | foreach-object{${"vibname" = "$_"})}

You could obviously make a variable of all your ESXi hosts and do them all at once, but you might not want to leave your ESXi hosts sitting there waiting for a reboot for a while.  It’s your call how to handle that part, but this is how you can remove conflicting VIBs at a basic level.

Hope this helps!

VMware NTP timekeeping considerations in depth – Intro

Accurate timekeeping is important in almost every environment.  If time is not synced across your environment, authentication errors can occur, services and applications may not function properly, event logs and alerts can be off, which can inhibit troubleshooting.  You’re probably aware already that this is a big deal.  Beyond just referencing KB articles, I want to spend time to discuss NTP timekeeping in general, as well as practical methods and strategies that work, and in my experience what doesn’t work.

This will be a series of posts to try to address all major considerations with timekeeping via NTP, beginning with timekeeping within virtual machines.

NTP – Accuracy vs. Internal Synchronization

Obviously, you need your internal to have accurate time and synced with authentication sources.  Protocols like Kerberos for good reason don’t allow for much clock skewing in order to protect against authentication replay attacks. For example, Active Directory’s default tolerance for clock skewing is five minutes.

But sometimes both of those goals conflict each other.  In these cases, which is more important?  For probably almost all environments that the priority should be that clocks are synced over how accurate the clocks actually are.

Why?  Simple – application and service availability.  Chances are, if clocks are skewed too much within your environment, services and applications will become inaccessible to some or all users.  Generally, razor sharp clock accuracy in the real world if lacking can often be an annoyance, not a downtime event.  Obviously, that may not be the case for everyone, such as real time stock trading companies, but that’s generally for the most part true.  When making choices about how to configure things for time usually through NTP, if faced with a scenario where you must choose better internal synchronization instead of better accuracy to what the real time is, choose better synchronization over actual time accuracy.

When would these goals come into conflict?  As an example, VMs could be set to synchronize their clocks with their VM host via VM Tools, or they could be configured within their OS to use an external NTP server.  It’s theoretically possible that for some reason, your ESXi host’s clock might be more trustworthy than your Domain Controllers more often than not.  For most customers though, even if that were true, prioritize synchronization over clock accuracy.  Allowing VMTools to sync the clock of the VM to the host effectively means VMs running on different hosts could have different time.  Maybe the NTP service stopped on one ESXi host.  Maybe they’re not configured consistently.  It doesn’t matter why.  Prioritize synchronization instead by configuring each VM’s OS to synchronize to the same NTP servers somehow, some way.

How many NTP servers, and which ones?

When configuring anything for NTP, whether it be an ESXi server or  guests, the question always comes up – how many servers should an NTP client be set to use?

Many people know some obvious ones.  More than one, right?  Of course.  Providing more than one offers redundancy in case an NTP server fails.  However, I’ve encountered many environments where there were just two configured.  Of course three would be better just for resiliency, but configuring two NTP servers has risks beyond that.

Remember that NTP clients function by polling all their configured NTP servers, and then adopting the most consensus time values across all of them.  For example, if two NTP servers configured provide different values, the NTP client will adopt a value that’s a compromise between them.  In a scenario where NTP server 1 says the time is off by twenty minutes, but NTP server 2 is correct, the NTP client will likely to adopt a value of 10 minutes too fast, which is incorrect, and worse may cause clock skewing within the environment.  I recommend you use instead an odd number of NTP servers greater than one, and the more the merrier generally speaking.

But which ones?  Diversity that improves availability is good, but diversity that will be more likely to result in disparate values is bad.  Using NTP servers that are for example on separate compute, storage, and physical sites is good.  Mixing and matching for example internal and external NTP servers that are managed by different people on the same NTP client is generally bad, although it might be the best alternative among non-optimal choices.

In my next post in this series, I’ll go into specifics on how I generally apply these considerations to VMware environments.

2VB-601 Exam Study Guide for VSAN Specialist

As promised, here’s my 2VB-601 Exam Study Guide to achieve the VSAN Specialist 2017 badge.  Hope this helps!

Section 1- Storage Fundamentals

Objective 1.1 – Identify storage device characteristics

Objective 1.2 – Identify storage performance factors

Section 2 – VSAN Fundamentals

Objective 2.1 – Provide a high-level description of vSAN

Objective 2.2 – Describe vSAN requirements

Objective 2.3 – Understand how vSAN stores and protects data

Objective 2.4 – Describe vSAN space efficiency features

Section 3 – vSAN Configuration

Objective 3.1 – Identify physical network requirements

Objective 3.2 – Configure vSAN networking

Objective 3.3 – Configure a vSAN cluster

Objective 3.4 – Create and manage disk groups

Objective 3.5 – Validate a vSAN configuration

Section 4 – vSAN Policies and Virtual Machines

Objective 4.1 – Explain how storage policies work

Objective 4.2 – Create and manage storage policies

Objective 4.3 – Explain how storage policies are applied to virtual machines

Objective 4.4 – Check storage policy compliance

Objective 4.5 – Describe vsanSparse snapshots

Section 5 – Managing and Operating vSAN

Objective 5.1 – Configure vSAN encryption

Objective 5.2 – Understand various failure events and how vSAN responds

Objective 5.3 – Describe maintenance mode options

Objective 5.4 – Manage hardware replacement

Objective 5.5 – Describe iSCSI Target service

Section 6 – Stretched Clusters and Two-Node Clusters

Objective 6.1 – Describe a stretched cluster architecture

Objective 6.2 – Create a stretched cluster

Objective 6.3 – Understand data placement in a stretched cluster

Objective 6.4 – Describe the two-node cluster architecture

Section 7 – Monitoring and Troubleshooting vSAN

Objective 7.1 – Understand hardware failure scenarios

Objective 7.2 – Interpret vSAN Health in the vSphere Web Client

Objective 7.3 – Access performance information in the UI and using CLI

Objective 7.4 – Access capacity management information

Section 8 – Interoperability with vSphere Features

Objective 8.1 – Identify vSphere features that work with vSAN

Objective 8.2 – Understand vSAN compatibility with SRM

Objective 8.3 – Describe 3rd-party solution integration with vSAN

Objective 8.4 – Understand vSAN compatibility with Horizon

Section 9 – Designing a vSAN Deployment

Objective 9.1 – Understand vSAN design considerations

Objective 9.2 – Understand vSAN cache tier sizing

Objective 9.3 – Design a vSAN cluster

Objective 9.4 – Identify vSAN design and sizing tools

I hope my 2VB-601 Exam Study Guide helps anyone else out there prepare for the exam successfully!

2VB-601 – Objective 1.1 – Identify storage device characteristics

Here are my notes for 2VB-601.  These notes I took to help me prepare for the exam as I went through the Deploy and Manage VSAN course, and through the recommended documentation.  Often, if I already knew the info, I didn’t necessarily put it in my notes.

Hope these help!

  • Lifespans of SSD drives
  • SLC
  • 100,000 writes
  • MLC
  • 3000-10000 writes
  • TLC
  • 1000 writes
  • eMLC
  • 20,000-30,000 writes
  • NVMe
  • Specification developed specifically for SSDs, more parallelism, better performance
  • 3D Cross Point
  • PCIe NVMe cards
  • Improved even more on performance
  • HDDs
  • Slower but higher capacity than SSDs
  • 15K, 10K, 7.2K RPM drives, higher = better latency

Return to 2VB-601 Exam Guide.

VMware VSAN Specialist 2VB-601 Exam Review

On Friday, I sat the VMware VSAN Specialist 2VB-601 Exam.  I’ll be deploying more VSAN soon, so I used this recently released 2VB-601 exam as a guide to thoroughly learn the product.  Passing 2VB-601 along with a VCP6 version of a VMware certification grants you the VMware VSAN Specialist 2017 badge.  This isn’t a full certification, but it acknowledges candidates with VSAN knowledge and skills.

2VB-601 Exam Format

The 2VB-601 exam consists of 60 multiple choice questions, and you have 105 minutes to complete the exam.  It is very comparable to VMware VCP exams as far as format goes.  If you’ve taken VCP exams before, you certainly know the drill here.  The top score for the exam is 500, and passing is 300, just like VCP exams.

2VB-601 Exam Resources

There aren’t a ton of affordable learning resources out there for this exam as far as books and what not go unfortunately.  However, if you follow the exam guide and read the documents provided in VMware’s study guide, along with hands on experience, you can certainly pass 2VB-601.  I also highly recommend the VMware hands on labs pertaining to VSAN.  You could also build your own lab using EvalExperience included with VMUG Advantage.

VSAN 6.6 Deploy and Manage training isn’t necessary, but I did attend it to fulfill a partner requirement.  If you’re a partner and need to do the same, this will definitely help.

2VB-601 Exam Experience

The 2VB-601 exam is very straight forward.  VMware VCP exams are notorious for sometimes asking rote memorization type questions, such as the exact word for word options for configuration choices.  I did not find that to be the case generally speaking with this exam.  Most questions are fair and are generally more conceptual in nature.  You do need to know what needs to be done or what happens in various scenarios.  I also generally didn’t find myself wondering to which exam objective questions were related, like I often do on VCP exams for more questions than I’d like.  Virtually every question I felt was fair game on.

With that said, the exam I found wasn’t nearly as difficult as every other VCP exam I’ve taken, which are numerous at this point.  If you look over the exam guide and feel you know them, have hands on experience with VSAN, I would recommend reading over the substantial documents in the guide prior to sitting the exam.  Otherwise, you should be in good shape.

I passed it on my first attempt with a 456, which is the highest I’ve ever gotten on any VMware exam.  I finished the exam with 45 minutes to spare, so time won’t be an issue.  I found the questions mostly fell in the category of “you know it or you don’t”.

I’ll be posting resources to help study for the exam for those of you who wish to take the exam.

Hope this helps!