Tag Archives: PowerCLI

powercli script configure vcenter alarm email actions

PowerCLI Script to Configure vCenter Alarm Email Actions

Have you ever gone through your vCenter and configured alarms to email?  If you have, you know that if anything ever screamed for automation within vSphere, it’s this, as it is extremely tedious.  You really want to use a PowerCLI script to masnage vCenter alarm email actions!

For one, the action must be set individually on each alarm. On top of that, you configure each alarm with the email address(es) you want  to be sent alerts.  You can also set repeated email actions on each alarm.  While this provides granularity and customization opportunities, it creates a boring snoozefest, inviting confusion and human error in configuring them.  Plus, vCenter contains many alerts, and many people do not know which ones you should configure for emails, and how critical each one is.

Introduction to PowerCLI Script to Configure vCenter Alarm Email Actions

I found a good script about a year ago to do this with PowerCLI script using a CSV of the alarms.  I cannot seem to find who made it now.  If you stumble on this and know who originally made the similar script, please comment below.  I very much want people recognized for their hard work.

Not to simply take the script, I added additional functionality to it, to provide end to end configuration, including vCenter SMTP email server settings, as well as the ability to configure multiple vCenter servers in one fell swoop.  Also, I really love the design of this script for several reasons:

  • Simplicity rules this script.  Even if you want the script to do something else, it’s easy to follow and adapt.
  • One can easily adapt the script to new versions of vSphere.  Each version of vSphere adds, removes, or changes alarms.  I can easily dump those alarm names into a new CSV and set their values.
  • It provides a three-tiered priority system for email frequency.   Don’t like them?  It’s easy to change them within the script.
  • Ongoing maintenance of the alerts is a snap.  Change the values within the script, and simply rerun it.
  • The CSV file provides an opportunity to track changes to alerts and a deliverable document.  I intend also to maintain the CSV files here as I receive feedback on the alarms.

Ready to use a PowerCLI script to configure vCenter alarm email actions?

Directions for PowerCLI Script to Configure vCenter Alarm Email Actions

This PowerCLI script to configure vCenter alarm email actions is very straight forward.  You simply set the variables at the beginning of the script, which includes info such as vCenter servers, the CSV file to be used, vCenter user name and password, the SMTP server and port to use, etc.

Next, set the values within the CSV according to how you want that alert configured within the Priority column.  The values are as follows :

  • Low Priority – Removal of all email alert actions, and reconfigured to send one email without repeat emails for non-critical alerts.
  • Medium Priority – Removal of all email alert actions, and reconfigured to send one repeated email daily for more serious alerts.
  • High Priority – Removal of all email alert actions, and reconfigured to send one repeated email every four hours for more serious alerts.  (I originally set this to hourly, but customer feedback said this was far too much.)
  • Disabled – Removal of all email alert actions.  Use this on alerts that are far too chatty (looking at you VM CPU and memory usage!), and you wish to turn emails for them off.
  • Blank (aka no value) – Leave the alarm as is in case you manually configured the alert and wish to keep it that way.

$vcenterservers = "vcenter1.vs6lab.local","vcenter2.vs6lab.local"
$vcenterusername = 'administrator@vsphere.local'
$vcenteruserpwd = 'P@ssw0rd'
$alarmfile = import-csv c:\scripts\vsphere60-alarms.csv
$AlertEmailRecipients = @("email1@domain.com","email2@domain.com")
$SMTPServer = "FQDNorIP.domain.com"
$SMTPPort = "25"
$SMTPSendingAddress = "sender@domain.com"

#Import PowerCLI module
import-module -name VMware.PowerCLI

#----These Alarms will be disabled and not send any email messages at all ----
$DisabledAlarms = $alarmfile | where-object priority -EQ "Disabled"

#----These Alarms will send a single email message and not repeat ----
$LowPriorityAlarms = $alarmfile | where-object priority -EQ "low"

#----These Alarms will repeat every 24 hours----
$MediumPriorityAlarms = $alarmfile | where-object priority -EQ "medium"

#----These Alarms will repeat every 4 hours----
$HighPriorityAlarms = $alarmfile | where-object priority -EQ "high"

foreach ($vcenterserver in $vcenterservers){
Disconnect-VIServer -Confirm:$False
Connect-VIserver $vcenterserver -User $vcenterusername -Password $vcenteruserpwd
Get-AdvancedSetting -Entity $vcenterserver -Name mail.smtp.server | Set-AdvancedSetting -Value $SMTPServer -Confirm:$false
Get-AdvancedSetting -Entity $vcenterserver -Name mail.smtp.port | Set-AdvancedSetting -Value $SMTPPort -Confirm:$false
Get-AdvancedSetting -Entity $vcenterserver -Name mail.sender | Set-AdvancedSetting -Value $SMTPSendingAddress -Confirm:$false

#---Disable Alarm Action for Disabled Alarms---
Foreach ($DisabledAlarm in $DisabledAlarms) {
Get-AlarmDefinition -Name $DisabledAlarm.name | Get-AlarmAction -ActionType SendEmail| Remove-AlarmAction -Confirm:$false

#---Set Alarm Action for Low Priority Alarms---
Foreach ($LowPriorityAlarm in $LowPriorityAlarms) {
Get-AlarmDefinition -Name $LowPriorityAlarm.name | Get-AlarmAction -ActionType SendEmail| Remove-AlarmAction -Confirm:$false
Get-AlarmDefinition -Name $LowPriorityAlarm.name | New-AlarmAction -Email -To @($AlertEmailRecipients)
Get-AlarmDefinition -Name $LowPriorityAlarm.name | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Green" -EndStatus "Yellow"
#Get-AlarmDefinition -Name $LowPriorityAlarm.name | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Yellow" -EndStatus "Red" # This ActionTrigger is enabled by default.
Get-AlarmDefinition -Name $LowPriorityAlarm.name | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Red" -EndStatus "Yellow"
Get-AlarmDefinition -Name $LowPriorityAlarm.name | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Yellow" -EndStatus "Green"

#---Set Alarm Action for Medium Priority Alarms---
Foreach ($MediumPriorityAlarm in $MediumPriorityAlarms) {
Get-AlarmDefinition -Name $MediumPriorityAlarm.name | Get-AlarmAction -ActionType SendEmail| Remove-AlarmAction -Confirm:$false
Get-AlarmDefinition -Name $MediumPriorityAlarm.name | Set-AlarmDefinition -ActionRepeatMinutes (60 * 24) # 24 Hours
Get-AlarmDefinition -Name $MediumPriorityAlarm.name | New-AlarmAction -Email -To @($AlertEmailRecipients)
Get-AlarmDefinition -Name $MediumPriorityAlarm.name | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Green" -EndStatus "Yellow"
Get-AlarmDefinition -Name $MediumPriorityAlarm.name | Get-AlarmAction -ActionType SendEmail | Get-AlarmActionTrigger | Select -First 1 | Remove-AlarmActionTrigger -Confirm:$false
Get-AlarmDefinition -Name $MediumPriorityAlarm.name | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Yellow" -EndStatus "Red" -Repeat
Get-AlarmDefinition -Name $MediumPriorityAlarm.name | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Red" -EndStatus "Yellow"
Get-AlarmDefinition -Name $MediumPriorityAlarm.name | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Yellow" -EndStatus "Green"

#---Set Alarm Action for High Priority Alarms---
Foreach ($HighPriorityAlarm in $HighPriorityAlarms) {
Get-AlarmDefinition -Name $HighPriorityAlarm.name | Get-AlarmAction -ActionType SendEmail| Remove-AlarmAction -Confirm:$false
Get-AlarmDefinition -name $HighPriorityAlarm.name | Set-AlarmDefinition -ActionRepeatMinutes (60 * 4) # 4 hours
Get-AlarmDefinition -Name $HighPriorityAlarm.name | New-AlarmAction -Email -To @($AlertEmailRecipients)
Get-AlarmDefinition -Name $HighPriorityAlarm.name | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Green" -EndStatus "Yellow"
Get-AlarmDefinition -Name $HighPriorityAlarm.name | Get-AlarmAction -ActionType SendEmail | Get-AlarmActionTrigger | Select -First 1 | Remove-AlarmActionTrigger -Confirm:$false
Get-AlarmDefinition -Name $HighPriorityAlarm.name | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Yellow" -EndStatus "Red" -Repeat
Get-AlarmDefinition -Name $HighPriorityAlarm.name | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Red" -EndStatus "Yellow"
Get-AlarmDefinition -Name $HighPriorityAlarm.name | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Yellow" -EndStatus "Green"

File downloads:

PowerCLI script to configure vCenter alarm email actions

Recommended alarm configurations for vSphere 5.5

Recommended alarm configurations for vSphere 6.0

Recommended alarm configurations for vSphere 6.5

These CSV files include the alarms for each version, what I’ve found with customer feedback so far as the best values for each one, a notes field that has any relevant info that may help you decide how to configure each one, and an IfUsed column for each one I’ve set to disabled if you want to enable it what I would recommend.

When to Run PowerCLI Script to Configure vCenter Alarm Email Actions

I designed the PowerCLI script to configure vCenter alarm email actions not just for initial configurations, but also to set any changes that are made.  That’s why the script removes any email alarm actions for Low, Medium, High, and Disabled.  If you wish for the script to not change an alarm’s configuration, leave the Priority column blank for the alarm.

This way, you will have a current documented configuration of alarms that you can simply update and run the script again.p


Document DRS Rules with PowerCLI

As a consultant, I find myself doing a lot of reconnaissance within customers’ vSphere environments.  Here’s how to document DRS Rules with PowerCLI.

DRS Rules – Why have them?

You can use DRS rules for numerous purposes.  Use them to provide better reliability for applications and services.   You can also use them to ensure licensing compliance, and for other pragmatic purposes.

Here’s a very quick set of hand rules for DRS rules:

  • Use VM anti-affinity DRS rules to ensure redundant VMs are not running on the same host within a cluster.  In these configurations, having only one of the VMs up keeps the service or application online.  Examples include cluster members, web farm members, domain controllers, DNS servers, etc.  (Note: for Microsoft clusters, don’t forget to include File Share Witnesses in the rule as well as the nodes!)
  • Use VM affinity DRS rules for VMs that in total comprise a service and/or application.  In these configurations, downtime for any single VM in the group causes the entire application or service to be inoperable.  Examples include a web front end server, an application middleware server, and database backend server.  Or perhaps an email fax server that depends upon an email server.
  • Use VM host affinity/anti-affinity should DRS rules to try to ensure a VM runs or doesn’t run on specific hosts, but it can violate the rule if required, such as possible preferred nodes are down.  An example would be for vCenter to run on a specific host in case the vCenter VM goes down.  You then likely know which host to log directly into to start it back up.  Otherwise, if that preferred node is down, the VM can run on another node.
  • Use VM to host affinity must DRS rules to ensure a VM will only run on specific VMs, even if that results in downtime if those hosts aren’t up.  Must rules generally should only be used for licensing compliance purposes, where the software vendor licenses the product on all possible potential physical nodes it can run on, not how many hosts it could be actively running on at any given point in time.

Document DRS Rules with PowerCLI – Rules

DRS rules are a little bit of a challenge.  Members are a multi-valued property with VM IDs, which isn’t particularly useful.  We need to work a little magic to translate VM IDs to VM names, and then join the multi-valued property to allow it to be exportable into CSVs, etc.

This can be accomplished using the Get-DrsRule cmdlet.

Get-DrsRule -Cluster ClusterName | Select Name, Enabled, Type, @{Name="VM"; Expression={ $iTemp = @(); $_.VMIds | % { $iTemp += (Get-VM -Id $_).Name }; [string]::Join(";", $iTemp) }}

Now you can tack on an export-csv or what not to it, and it’s readable with useful information us humans would understand.

Note, there is also a specific cmdlet to get DRS to host rules only if that’s what you’re looking for : Get-DrsVMHostRule, but the above gets all DRS rules.

Document DRS Rules with PowerCLI – Groups

DRS rules can also have groups, so it’s important that they’re documented as well.  Members are a multi-valued property, but that’s the only challenge here.  We just need to use a join method to make it readable.

This can be accomplished using the Get-DrsClusterGroup cmdlet.

Get-DrsClusterGroup -Cluster ClusterName | select Name, Cluster, GroupType, @{Name="Member:"; Expression={[string]::Join(";", $_.Member)}}

Now you can tack on an export-csv or what not to it.


Nutanix Password Management

As more and more Nutanix is deployed out there, it’s useful to have some basic information about managing and maintaining it readily available.  Today, I want to talk about Nutanix password management to ensure to help people know what important user accounts and passwords there are, and how to change those passwords.  This is important because you generally do not want default passwords set within your environment.  Changing default account passwords is the same best practice you’ll find for any other technology.

Nutanix Password Management – Introduction

Within a Nutanix environment, there are several types of accounts to be aware of.  These accounts are:

  • Nutanix Cluster Accounts – This is the account used to login to Prism, as well as command line utilities such as ncli and PowerShell to manage cluster wide settings, such as container management, cluster health, alerts, etc. for Nutanix Acropolis.  Think storage management and heath mainly here.
  • Nutanix Controller VM – this is a local account within each Nutanix Controller virtual machine running on each hypervisor host.  This account is mainly used for troubleshooting and low level command line type actions.  Generally speaking, you will likely not use this without the direction of an advanced Nutanix resource or Nutanix support.
  • Nutanix node IPMI – These are accounts stored within the IPMI out of band management interfaces on each Nutanix node.  This account is used for imaging your Nutanix nodes with your hypervisor of choice, and for things like remote console access, power cycling the hypervisor host if you’re having problems, etc.
  • Hypervisor accounts – these are the local administrator accounts within your hypervisor of choice.  Think root for vmware, local Administrator for Hyper-V.

It’s important to secure these accounts properly.  That includes changing them regularly, using complex passwords, etc.

Nutanix Password Management – Default Accounts

The following are the default accounts and their passwords for each.  This can be helpful for deployments, but also to verify if your Nutanix environment is using the default passwords.

  • Nutanix Cluster Account:
    • User:  Admin (Note the capital “A”!)
    • Password: nutanix/4u
  • Nutanix Controller VM:
    • User:  nutanix
    • Password: nutanix/4u
  • Nutanix node IPMI:
    • Nutanix hardware
      • User:  ADMIN (Note the capital “A”!)
      • Password: ADMIN
    • Dell hardware
      • User: root
      • Password: calvin
  • Hypervisor accounts:
    • ESXi:
      • User: root
      • Password: nutanix/4u
    • Acropolis Hypervisor (Nutanix’s version of KVM):
      • User: root
      • Password: nutanix/4u

Nutanix Password Management – Changing Nutanix Cluster Account Password

This one is pretty straightforward.  Simply login to prism as Admin and click to change the password under Settings (Gear symbol) > Change Password.

nutanix password management prismrwunix-hq Hypervisor Summary 2 HYPERVISORS Storage Summary Home 4 Hæts Cluster-wide Controller IOPS In lops 1200 Cluster-wide Controller 10 B/W O lops O KBPS Health Disks GOOD Admin Change Password Update Profile Download Cmdlets Installer Download nCLl Download Prism Central REST API Explorer

Nutanix Password Management – Changing Nutanix Controller VM Password

This one is a little more complex.  Here are the steps if you’re using ESXi:

  1. First, SSH into ESXi host.
  2. Next, SSH into controller from the ESXi server with the following command: ssh nutanix@
  3. After that, run the following command: allssh passwd
  4. Lastly, enter in current and new passwords as it cycles through each controller VM in the cluster.

Here’s a sample output:
[root@NTNX-16SM6B123456-A:~] ssh nutanix@
Nutanix Controller VM
nutanix@’s password:
Last login: Thu Nov 3 11:10:47 PDT 2016 from on ssh
Last login: Thu Nov 3 11:11:02 2016 from
nutanix@NTNX-16SM6B123456-A-CVM:$ allssh passwd
Executing passwd on the cluster
================== =================
Changing password for user nutanix.
Changing password for nutanix.
(current) UNIX password:
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
Connection to closed.
================== =================
Changing password for user nutanix.
Changing password for nutanix.
(current) UNIX password:
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
Connection to closed.

(This continues through the entire cluster)

Most of all, be sure it shows a successful operation for each node.

Nutanix Password Management – Changing Nutanix Node IPMI Account Password

This can be accomplished two ways.  To change the ADMIN account within the GUI:

  1. First, login to the IPMI interface using a web browser.
  2. Next, click Configuration > Users.nutanix password management ipmi
  3. Now, click the ADMIN account user, click Modify User, and follow the rest of the prompts.nutanix password management 2

You can also change the password via SSH on your ESXi servers.  This is particularly useful when you do not know your IPMI account credentials.

  1.  First, SSH into your ESXi server.
  2. Next, find the user ID of the account by running /ipmitool user list
  3. Finally, use the following command to change the password: /ipmitool user set password <userid> <CoolNewPassword>

Also, you can do this with other hypervisors as well.

Nutanix Password Management – Changing Nutanix Node Hypervisor Account Password

To change the default account password for your hypervisor, you simply conduct this with your password change method of choice for that hypervisor.

For ESXi, you can use passwd, the vSphere Thick Client, or even PowerCLI.  Here’s a PowerCLI script I made to change the root password on ESXi servers.

$username = 'root'
$newpassword = 'N3wP@ssw0rd'
$oldpassword = 'nutanix/4u'
$vmhosts = 31..34 | ForEach-Object {"192.168.10." + $_}
foreach ($vmhost in $vmhosts){
connect-viserver $vmhost -User $username -Password $oldpassword
Set-VMHostAccount –UserAccount $username –Password $newpassword
disconnect-viserver -Confirm:$false

Nutanix Password Management – Best Practices

To keep things simple, it is recommended to use the same hypervisor password on each host.  Also, while not required, it is recommended to set the controller VM and Prism passwords the same, again for simplicity.  You don’t have to.

Hopefully, this allows you to more easily management passwords within your Nutanix environment!

Making life easier using vSphere Tags

One of the least used features in vSphere that I think almost all admins could really make use of but don’t is the ability to create custom vSphere tags within vSphere.

I wanted to take the time to point this feature out, and perhaps give people some ideas on how to make use of of them.  This can help with management and automation quite a bit.

What are vSphere Tags?

vSphere Tags are effectively custom metadata type info that can be applied to objects within vCenter.  You get to make your own to fit your own needs.  They assist basically with locating objects for more efficient administration and management.

They’re unique to other things such as folders for your VMs in that you can assign multiple tags to the same VM or other objects.

Let’s break this down by comparing vSphere tags to MP3 management software like iTunes.  An individual MP3 file must be in one file system folder or another.  It can’t be in both.  But suppose you want to find all your songs by an artist, by genre, or by album?  We intuitively understand this now with MP3s.

But we have the same problem with VMs.  You can organize your VMs into VM folders in vCenter, but a single VM can only be in one folder or another.  What if you wanted to organize your VMs by criticality?  By whether or not they have SQL?  Whether or not they need to be backed up?  Trying to do this with folders would be a nightmare to manage.  Plus, remember a VM folder is the mechanism for assigning permissions, too.  Maybe you don’t want this metadata having any impacts on anyone’s permissions to manage it.

That’s when you use vSphere tags!

Use Cases for vSphere Tags

Use cases for this functionality are numerous:

  • Criticality of VM – this would allow the expedited power up or down of VMs based on this nature.  Running out of resources within your cluster due to sudden host failures?  Power down the non-critical VMs.  It would also be helpful for vSphere Admins who aren’t the application admins to know when to handle a VM with care before doing anything to it.
  • Application groupings – Maybe it doesn’t make sense to put VMs that work together to provide an application or service, but you want to know those groups.  That could allow a SQL server that serves the backend of multiple application groups to be identified for both simultaneously.
  • Presence of a common application like SQL – This can be helpful for locating VMs that may require special settings on backup jobs to quiesce the file system before backing the VM up.  You might also use this to find potential VMs that other VMs are dependent on, so you can set their restart priority so they boot up first in an HA event scenario.
  • Lab/Test VMs – You could set the resource allocation for Lab/Test VMs to low to help ensure they are given less resources than production VMs.

OK, I convinced you (hopefully)!   Let’s make some tags.

Basic Concepts for vSphere Tags You Need To Know

You can create vSphere tags in both within the vSphere Web Client and with PowerCLI.  It’s simple, but you need to know a few concepts.

All vSphere tags belong to a Category.  There are two main types of categories.  This notion is called Cardinality.  It sounds more complicated than it is.  Basically, you can have a category where only a single tag from that category can be applied to any given object.  For example, let’s say you want to tag VMs by criticality.  Logically, a VM will only have one criticality rating, not multiple.  IE, it makes zero since for a VM to be both low and medium as far as how critical they are.

However, sometimes you might want a category that multiple tags could apply to the same object.  For example, let’s say you want to make a category called “Special Applications” to identify very specific apps within a VM to easily identify SQL servers, Domain Controllers, and Exchange servers.  While I wouldn’t recommend it, it’s possible for a single VM to be all three simultaneously.

vSphere tags can apply to all kinds of objects as well, not just VMs.  You can select which objects a tag can be applied to within the category.

Managing vSphere Tags Using the Web Client

To create tags within the vSphere client, navigate to the Tags section of the web client.

vm tags web client nav

You must create a category first if there isn’t one already made.  Click the Categories button, and then click the create categories icon.

For this example, we will make a category for criticality ratings for VMs.  We want one tag per object, not more, and we only want the tag to be applied to VMs or vApps.

vsphere category example

Now that we have our category, we can create tags within it.  Click on Tags, and the new tag icon.  Be sure to select the category during tag creation.

vsphere tags create tag example

Rinse and repeat for all the tags you want to create for the category.  One tip I recommend is to name the tags with incuding their category name, which refers to some kind of concept.  Since you usually search by the tag name, you want for example LowCriticality instead of Low.  (See below for search examples.) Low in and of itself could mean a lot of things.  Low resource usage, low criticality, etc.

To apply a tag to an object, simply right click the object, point to Tags & Custom Attributes > Assign Tag…

vsphere tags assign tag

A new dialog box appears where you can filter categories or see all categories and select the vSphere tags you wish to assign.  Also, notice you can remove tags here, too.

Managing vSphere Tags Using PowerCLI

PowerCLI has full tag management functionality within it, too.

Creating a category:

New-TagCategory -Name VMCriticality -Description "Criticality of the VM" -Cardinality Single -EntityType "VirtualMachine","VApp"

Creating a tag:

New-Tag -Name "LowCriticality" -Description "Non-Critical VMs" -Category VMCriticality

Assigning a tag to a VM:

get-vm Shoretel | New-TagAssignment -Tag "HighCriticality"

You can do lots of things with PowerCLI and tags.

Using vSphere Tags

Now that you have tags created and applied, you can now make use of them to make your life easier.

You can make use of tags in both the vSphere Web Client and via PowerCLI.  To find all VMs with a tag within the vSphere Web Client, simply type the tag value in the search box.  The tag name will automatically populate.

vsphere tags searching

Click on it.  Boom, you got your objects with that tag!

vsphere tags search results

There’s also a parameter on PowerCLI’s Get-VM cmdlet to identify the VMs with that tag.  You can then pipe that to another cmdlet.  Say for example you want to shutdown your non-critical VMs because you suddenly experience multiple host failures, so you need to make sure your more important VMs get the resources they need:

Get-VM –Tag “LowCriticality” | Shutdown-VMGuest

Imagine if you set up vSphere tags to identity all your VMs with SQL.  Imagine you’re setting up Veeam backup jobs, and you need to know which VMs you need to setup special quiescing.  You could easily just get that list of VMs.

That’s how to use vSphere tags!

How do you think you might be able to use them, or how do you use them within your environment?


Manage ESXi SSH Using PowerCLI

Let’s face it. Starting and stopping SSH in ESXi is pain through GUI methods.  I often as a consultant need to connect via SSH to hosts to run data collect scripts, assess NIC and HBA firmware and driver versions, and for troubleshooting purposes, like to run esxtop.  The good news is you can manage ESXi SSH Using PowerCLI.  How cool is that?

Just remember to use get-vmhost to narrow down the specific hosts you want to execute the following commands.

Get the current status of ESXi SSH Using PowerCLI

get-vmhost &nbsp;hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | select-object vmhost,policy,running

Policy is the start up mode.

  • Automatic = Start automatically if any ports are open, and stop when all ports are closed
  • On = Start and stop with host
  • Off = Start and stop manually

Start ESXi SSH Using PowerCLI

get-vmhost hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | start-vmhostservice -confirm:$false

Note the confirm switch.  If you don’t specify that, it will prompt you.

Stop ESXi SSH Using PowerCLI

get-vmhost hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | start-vmhostservice -confirm:$false

Note the confirm switch.  If you don’t specify that, it will prompt you.

Set startup policy for ESXi SSH Using PowerCLI to start and stop with host

get-vmhost hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | set-vmhostservice -policy "Off"

Be careful if you have any third party products that use SSH.  Nutanix for example comes to mind.  If you goofed and need it set to start and stop with host, just use “On” for the policy parameter.

How to backup ESXi servers

Do you backup your ESXi servers?  You don’t technically have to.  If you’ve documented your environment well, and/or you use things like host profiles, Distributed Virtual Switches, and Autodeploy, the need is lessened or even pointless.  However, if you have any manual configurations that need to persist beyond a reboot, you should do it because it’s stupid easy.  Here’s how to backup ESXi servers.

Backup ESXi servers the easy way

This is really easy if you do it with PowerCLI.  Ready?

Get-VMHost | Get-VMHostfirmware –BackupConfiguration –DestinationPath “C:\BackupLocation”

That’s it!  If you used Connect-VIServer to connect to vCenter, this one liner creates within the backup location a file for each server.  Done!

You might want to run this when:

  • You’re about to make any configuration changes.
  • Periodically in case others make configuration changes you’re not aware of.
  • Before you patch your servers.
  • After you patch your servers, and you’ve determined they’re functioning properly.
  • If the ESXi install is on storage that is reporting errors.
  • To migrate ESXi installations to different storage.

This is a great way to get your ESXi servers up and going in the event of a misconfiguration or failure in the ESXi server installation storage.

Restore from backup files

Great!  You have a backup of your ESXi servers.  How do you restore it?

Set-VMHostFirmware -VMHost <IP_or_FQDN> -Restore -Force -SourcePath C:\BackupLocation

If the ESXi server is not on the network anymore, it must be returned to a network connected state first.  You may need to reinstall ESXi first, and configure its management network first.  Then, you can use Connect-VIServer to connect directly to the ESXi server in order to run the above.

Also, be aware that configuration backups can be and almost always are ESXi server version specific.  So ensure you reinstall ESXi using the same version that the server was when the backup was taken.  Ensure you create new backups after you patch your server for this reason.


Configure Dump Collector with PowerCLI in vSphere 6

I had a script to configure Dump Collector settings that I used in previous versions of vSphere.  If you look around the web, you’ll find similar PowerCLI snippets to configure Dump Collector.

If you use that snippet in vSphere 6, it doesn’t work.  You’ll get the following error:

Message: Cannot set 2 server ip parameters.;
InnerText: Cannot set 2 server ip parameters.EsxCLI.CLIFault.summary
At line:4 char:1

This is because ESXCLI now has a parameter for whether to use IPv6, so when using get-esxcli, invoking the method to set requires an additional value.  Remember, esxcli is not intuitive in that “enabled” properties are either true or null, so don’t use $false.

The revised code should now be:

$vcenterip = ''
foreach($vmhost in Get-VMHost){
	$esxcli = Get-EsxCli -VMHost $vmhost.Name

Also not something commonly found on the internet – can you test the ESXi netdump configuration?  Yep!

foreach($vmhost in Get-VMHost){
$esxcli = Get-EsxCli -VMHost $vmhost.Name
Write-Host "Checking dump collector on host $vmhost.name"

And there you have it!

Change VMware MPIO policy via PowerCLI

This is one of those one liners I think that I’ll never use again, but once again, I found myself using it to fix MPIO policies on a vSphere 5.0 environment plugging into a Nexsan storage array.  I’ve previously used it on EMC and LeftHand when the default MPIO policy for the array type at the time of ESXi installation is not the recommended one after the fact, or in the case of LeftHand, it is wrong from the get go.

get-vmhost | get-scsilun | where vendor -eq "NEXSAN" | set-scsilun  -MultipathPolicy "RoundRobin"

In this case, it was over 300 LUN objects (LUNs x hosts accessing them), so that’s about 5 mouse clicks per object to fix via GUI.  Translation, you REALLY want to use some kind of scripting to do this, and PowerCLI can do it in one line.

You gotta love PowerShell!

Getting Cisco UCS drivers right with vSphere

I’ve noticed one pain point with Cisco UCS – drivers. You better have the EXACT version Cisco wants for your specific environment. But how do you know which drivers to get, how do you get them, how do you know when you need to upgrade them, and how do you know what drivers you have installed? These are all not necessarily straightforward, and getting the info you need can be a real pain.  This post will show how to accomplish this within vSphere.  For Windows servers, please see my follow-up post due out in a few days.

Why is getting the drivers so important?

I want to emphasize that getting the exact right version of Cisco UCS drivers is a big deal! I’ve personally now seen two environments that had issues because the drivers were not exactly correct. The best part is the issues never turned up during a testing of the environment. Just weird intermittent issues like bad performance, or VMs needing consolidation after backups, or a VM hangs out of nowhere a week or two down the road. Make sure you get the drivers exactly right!

How do I install ESXi on Cisco UCS?

First off, pretty much everyone knows that when you’re installing ESXi on Cisco, HP, Dell, IBM, or other vendor servers, use the vendor’s media. That’s common practice I hope by now. In most but not all cases, you get the drivers you need for an initial deployment from the get go, you get hardware health info within VMware, sometimes management and monitoring tasks for out of band management cards, and you ensure vendor support by doing this. We all know I think by now to do initial ESXi installs with vendor media, in this case Cisco. It’s important for Cisco UCS since so many installs require boot from SAN, that you gotta have those drivers within the media off the bat.

Now, if you think you’re done if you’ve downloaded the latest Cisco co-branded ESXi media for an initial deployment, you’re wrong (see below). Also, don’t assume that just because you use the co-branded media to install ESXi on a UCS server, you never need driver updates. You will likely when you update UCS Manager and/or update ESXi down the road.

How do I know which drivers should be installed?

This is relatively simple. First, collect some info about your Cisco UCS environment. You need to know these (don’t worry, if you don’t know what info you need, Cisco’s Interoperability page will walk you through it):
1. Standalone C-Series not managed by UCSM or UCSM managed B and/or C-Series? For those of you who don’t know, if you got blades, you got UCSM.
2. If UCSM is present, which version is it running? Ex. 2.2(3c)
3. Which server model(s) are present? Ex. B200-M3. Also note the processor type (ex. Xeon E5-2600-v2). They can get picky about that.
4. What OS and major version? Note the Update number. Ex. ESXi 5.5 Update 2
5. What type and model of I/O cards do you have in your servers? Example – CNA, model VIC-1240

Then head on over to the Interoperability Matrix site.  Fill in your info, and you get a clear version of the driver and firmware.


It’s very straightforward to know which drivers are needed from that.

How do I figure out which drivers are installed?

If you go looking at Cisco for how to find that out, you get treated to esxcli commands.  Do you really want to enable SSH on all your hosts, SSH into each host, run some commands, then have to disable SSH on all those boxes when you’re done, and not have an easy way to document what they are?  Nope!


To get the fnic driver versions for all ESXi hosts:

$hosts = Get-VMHost
$versions = @{}
Foreach($vihost in $hosts){
$esxcli = Get-VMHost $vihost | Get-EsxCli
$versions.Add($vihost, ($esxcli.system.module.get("fnic") |
Select Version))
$versions.GetEnumerator() | Sort Name | Format-List 

You get this:

Name : esxi01.vspheredomain.local
Value : @{Version=Version, Build: 1331820, Interface: 9.2 Built on: Jun 12 2014}

Hey! That’s the wrong driver, even though I used the latest co-branded media! SON OF A…!

Let’s get some enic driver versions…

$hosts = Get-VMHost
$versions = @{}
Foreach($vihost in $hosts){
$esxcli = Get-VMHost $vihost | Get-EsxCli
$versions.Add($vihost, ($esxcli.system.module.get("enic") |
Select Version))
$versions.GetEnumerator() | Sort-Object Name | Format-List 

You get:

Name : esxi01.vspheredomain.local
Value : @{Version=Version, Build: 1331820, Interface: 9.2 Built on: Aug 5 2014}

Of course, Cisco apparently didn’t update those drivers in their co-branded media either.

Note for both scripts, you will get errors about get-esxcli not being supported without being connected directly to each host. It works for our purposes.

How do I update Cisco UCS drivers?

Now we know, despite using the latest Cisco co-branded media in my implementation, I need some driver updates. If you go to Cisco’s site for how to install these drivers, they’ll tell you to upload the package to each host and install them one at a time manually using esxcli commands. Do you really want to do that?

Let’s be smart/lazy/efficient and use VMware Update Manager. That way if a new host gets introduced, VUM will report that host non-compliant, and it’ll be easy to fix that one, too. And it’s easy to see which hosts do and don’t have those drivers down the road.

I find if I google the driver version, I find a download from VMware’s site with that exact version first or second link. Here’s our fnic driver and enic driver in this case.

Download those to your vCenter server or something with the vSphere thick client. Unzip them into their own folders. Open up a thick vSphere client connection to vCenter (Web Client won’t allow you to do this), click Home, then click Update Manager.

Next, click Patch Repository tab at the top, and then click Import Patches in the top right.


When the dialogue comes up, browse to select the zip file that was *contained* in the original zip file. If you select just the zip file you downloaded itself, it will fail. Repeat for the fnic and enic drivers.

When you’re finished, you can then build a baseline that includes the updated drivers. Click Baselines and Groups, then Create above the baselines pane.


Call it something like “Cisco UCS Current Drivers”.  Select “Host Extension” as a Host Baseline type.  In the following pain, find the drivers and click the down arrow to add them into the baseline.  Note the Patch ID field has driver version specifics, useful if you’ve already got some Cisco drivers imported before.


You can then attach that baseline directly to the appropriate object(s) within the host and clusters view, or I like to make a Baseline Group called “Critical and non-critical patches with Cisco updated drivers”, add all the appropriate baselines to that group, and attach that group to the appropriate objects in the Hosts and Clusters view.

Then remediate your hosts. When new drivers come out, import them in, then edit the Cisco baseline, swapping out the last updated drivers with the new ones, and remediate to push them out.


When should I check my drivers?

You should do this during any of the following:

• During initial deployments
• When UCS Manager or a standalone C-Series BIOS is updated
• Major ESXi version upgrades
• Update pack upgrades for ESXi (when ESXi 5.5 servers for example are to be upgraded to Update 2, or 3, etc)

Also, remember, newer drivers aren’t the right drivers necessarily. Check the matrix for what the customer is or will be running to see which drivers should go along with it!

Howto: Fix vCenter 5.5 Syslog Collector bug

I just ran into an issue for a customer that is running vCenter 5.5.  Apparently, there is a known issue with the 5.5 version of the syslog collector that causes the debug log to grow indefinitely when it was upgraded  to 5.5 according to the article.  However, the customer in question was built fresh with 5.5, although I did update within 5.5 to newer builds during that time.  Bottom line is it should be checked if they’re running 5.5 version of the syslog collector in all cases to be sure.  The KB article outlines steps to stop this, which basically involves turning off debug logging altogether.

The debug log doesn’t contain actual syslog info for hosts, so this log file is only useful for troubleshooting issues with the syslog collector itself, so it’s almost certainly safe to delete.

Please note this only impacts the syslog collector.  If you did not install the syslog collector, this isn’t applicable.

You can copy and paste the following into an admin elevated PowerShell window to automate stopping the syslog service, changing the config file for the syslog collector to turn off debug logging completely, delete the probably massive debug log, and start the syslog collector again.  You can also save it as a .ps1 file and run it in an elevated prompt as well.

stop-service vmware-syslog-collector
(get-content "C:\ProgramData\VMware\VMware Syslog Collector\vmconfig-syslog.xml") | foreach-object {$_ -replace "<level>1</level>", "<level>0</level>"} | set-content "C:\ProgramData\VMware\VMware Syslog Collector\vmconfig-syslog.xml"
remove-item "C:\ProgramData\VMware\VMware Syslog Collector\logs\debug.log"
start-service vmware-syslog-collector

This assumes of course the syslog collector is running on a Windows machine, not the vCenter Appliance.  The article doesn’t seem to make clear if this issue only applies to the Windows version of Vcenter, and how to fix the vCenter Appliance if the issue did impact it as well.

Hope this helps!