Monthly Archives: January 2017

nutanix

Nutanix Password Management

As more and more Nutanix is deployed out there, it’s useful to have some basic information about managing and maintaining it readily available.  Today, I want to talk about Nutanix password management to ensure to help people know what important user accounts and passwords there are, and how to change those passwords.  This is important because you generally do not want default passwords set within your environment.  Changing default account passwords is the same best practice you’ll find for any other technology.

Nutanix Password Management – Introduction

Within a Nutanix environment, there are several types of accounts to be aware of.  These accounts are:

  • Nutanix Cluster Accounts – This is the account used to login to Prism, as well as command line utilities such as ncli and PowerShell to manage cluster wide settings, such as container management, cluster health, alerts, etc. for Nutanix Acropolis.  Think storage management and heath mainly here.
  • Nutanix Controller VM – this is a local account within each Nutanix Controller virtual machine running on each hypervisor host.  This account is mainly used for troubleshooting and low level command line type actions.  Generally speaking, you will likely not use this without the direction of an advanced Nutanix resource or Nutanix support.
  • Nutanix node IPMI – These are accounts stored within the IPMI out of band management interfaces on each Nutanix node.  This account is used for imaging your Nutanix nodes with your hypervisor of choice, and for things like remote console access, power cycling the hypervisor host if you’re having problems, etc.
  • Hypervisor accounts – these are the local administrator accounts within your hypervisor of choice.  Think root for vmware, local Administrator for Hyper-V.

It’s important to secure these accounts properly.  That includes changing them regularly, using complex passwords, etc.

Nutanix Password Management – Default Accounts

The following are the default accounts and their passwords for each.  This can be helpful for deployments, but also to verify if your Nutanix environment is using the default passwords.

  • Nutanix Cluster Account:
    • User:  Admin (Note the capital “A”!)
    • Password: nutanix/4u
  • Nutanix Controller VM:
    • User:  nutanix
    • Password: nutanix/4u
  • Nutanix node IPMI:
    • Nutanix hardware
      • User:  ADMIN (Note the capital “A”!)
      • Password: ADMIN
    • Dell hardware
      • User: root
      • Password: calvin
  • Hypervisor accounts:
    • ESXi:
      • User: root
      • Password: nutanix/4u
    • Acropolis Hypervisor (Nutanix’s version of KVM):
      • User: root
      • Password: nutanix/4u

Nutanix Password Management – Changing Nutanix Cluster Account Password

This one is pretty straightforward.  Simply login to prism as Admin and click to change the password under Settings (Gear symbol) > Change Password.

nutanix password management prismrwunix-hq Hypervisor Summary 2 HYPERVISORS Storage Summary Home 4 Hæts Cluster-wide Controller IOPS In lops 1200 Cluster-wide Controller 10 B/W O lops O KBPS Health Disks GOOD Admin Change Password Update Profile Download Cmdlets Installer Download nCLl Download Prism Central REST API Explorer

Nutanix Password Management – Changing Nutanix Controller VM Password

This one is a little more complex.  Here are the steps if you’re using ESXi:

  1. First, SSH into ESXi host.
  2. Next, SSH into controller from the ESXi server with the following command: ssh nutanix@192.168.5.254
  3. After that, run the following command: allssh passwd
  4. Lastly, enter in current and new passwords as it cycles through each controller VM in the cluster.

Here’s a sample output:
[root@NTNX-16SM6B123456-A:~] ssh nutanix@192.168.5.254
Nutanix Controller VM
nutanix@192.168.5.254’s password:
Last login: Thu Nov 3 11:10:47 PDT 2016 from 192.168.10.41 on ssh
Last login: Thu Nov 3 11:11:02 2016 from 192.168.5.1
nutanix@NTNX-16SM6B123456-A-CVM:192.168.10.41:~$ allssh passwd
Executing passwd on the cluster
================== 192.168.10.41 =================
Changing password for user nutanix.
Changing password for nutanix.
(current) UNIX password:
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
Connection to 192.168.10.41 closed.
================== 192.168.10.42 =================
Changing password for user nutanix.
Changing password for nutanix.
(current) UNIX password:
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
Connection to 192.168.10.42 closed.

(This continues through the entire cluster)

Most of all, be sure it shows a successful operation for each node.

Nutanix Password Management – Changing Nutanix Node IPMI Account Password

This can be accomplished two ways.  To change the ADMIN account within the GUI:

  1. First, login to the IPMI interface using a web browser.
  2. Next, click Configuration > Users.nutanix password management ipmi
  3. Now, click the ADMIN account user, click Modify User, and follow the rest of the prompts.nutanix password management 2

You can also change the password via SSH on your ESXi servers.  This is particularly useful when you do not know your IPMI account credentials.

  1.  First, SSH into your ESXi server.
  2. Next, find the user ID of the account by running /ipmitool user list
  3. Finally, use the following command to change the password: /ipmitool user set password <userid> <CoolNewPassword>

Also, you can do this with other hypervisors as well.

Nutanix Password Management – Changing Nutanix Node Hypervisor Account Password

To change the default account password for your hypervisor, you simply conduct this with your password change method of choice for that hypervisor.

For ESXi, you can use passwd, the vSphere Thick Client, or even PowerCLI.  Here’s a PowerCLI script I made to change the root password on ESXi servers.

$username = 'root'
$newpassword = 'N3wP@ssw0rd'
$oldpassword = 'nutanix/4u'
$vmhosts = 31..34 | ForEach-Object {"192.168.10." + $_}
$vmhosts
foreach ($vmhost in $vmhosts){
connect-viserver $vmhost -User $username -Password $oldpassword
Set-VMHostAccount –UserAccount $username –Password $newpassword
disconnect-viserver -Confirm:$false
}

Nutanix Password Management – Best Practices

To keep things simple, it is recommended to use the same hypervisor password on each host.  Also, while not required, it is recommended to set the controller VM and Prism passwords the same, again for simplicity.  You don’t have to.

Hopefully, this allows you to more easily management passwords within your Nutanix environment!

useful utilities duct tape

Useful Utilities Are Useful

If you’re an IT pro, no matter if you’re an admin, and engineer, a consultant, a PC technician, you have a toolbox of useful utilities, scripts, and software that you use to fix problems.  As time goes by, some of those tools get used more and more.  Others are used less and less for various reasons.  But what surprises me is how many tools in my toolbox on the surface have less and less use cases, but I still come back to them even when it seems I never would need to again.

Over the last few weeks, I’ve been working with a customer who has had significant turnover from consultants they’ve used.  They are moving off a troubled disparate datacenter environment that had over time developed numerous problems to a more consolidated environment that various SyCom resources including me have built for them that is functioning properly, has updated software and firmware, etc.  Along the way, we’ve run into numerous challenges that you wouldn’t normally anticipate.  Troubleshooting them to fix the problems often would take too much time to fix,.  Finding a duct tape solution was more expedient.

I wanted to give a few examples just to illustrate that having a wide knowledge of utilities out there and experience with them can help you solve problems.

In this case, the task was seemingly simple – move VMs running on a legacy NetApp array and vSphere 5.1 servers to a new(er) cluster running vSphere 5.5.  The clusters were managed by two different vCenter servers.  These clusters were within the same physical datacenter.  They had network connectivity between them.  They did not have access to the same storage arrays.  The customer allowed downtime to move them.  Therefore, the easiest way was a shared nothing cold migration (we’re running 5.1 on the source side, remember).  Simple, right?

Doing It the Textbook Way

I approached this like how any vSphere resource would.  Get the two clusters into the same vCenter instance, shut the VMs down, and migrate them cold.  How many times have you seen that fail?  Me?  Pretty much never.  Well, it wouldn’t work.  I’ll spare you the troubleshooting details, but trust me, doing it the native way wouldn’t work.

At this point, the time had come to get creative and bust out some useful utilities I hadn’t used in a long time.  We had to get the job done.  Tick tick!

Useful Utilities #1 – Veeam FastSCP

The customer wasn’t a Veeam customer (yet).  While the customer could take some downtime off hours, there was a limit to that.  We had to move about 2TB of data, so we needed to move this data as quickly as possible without a ton of labor to reconfigure the networks to get both environments access to the storage.

Sure, I could use WinSCP to just bulk copy the VMs over, but Veeam FastSCP, built into Veeam Backup and Replication trial, is free, and it moves data quicker as it disables encryption on the data transfer, which was acceptable to the customer.  I hadn’t had any reason to use FastSCP in probably five years because cold migration functionality and exporting VMs to OVFs and what not within vSphere made it unnecessary.  But here I was, using it yet again.

And sure enough, it worked like a champ.  We tested a quick procedure using it on a few development workloads.  We then proceeded moving all but the critical VMs, and it worked great… except for the last VM of course.  Come to find out, that was a critical SQL VM that the customer didn’t realize was using physical Raw Device Mappings.

Well, shoot, how do we do this one in a quick manner?

Useful Utilities #2 – VMware Converter

For numerous reasons, including perhaps sheer circumstance of projects I’ve worked on, I hadn’t until this had a need to use VMware Converter in years.  Virtualization is so prevalent now, that P2V is one of those things for me that’s like, “Hey man, remember that time we had to convert like 100 physical machines to virtual back in the day?  Good times!”

Also, I’ve generally recommended to customers to avoid converting physical to virtual anyway.  It should generally be seen as a shortcut, but never optimal.  If you could just build a fresh new VM and get the data moved, the resulting VM would be cleaner.  It would probably perform better.  There’s less chance of instability from old drivers and what would inevitably be a significant change in hardware for the OS and application.  Obviously, if you’re dealing with a ton of machines, rebuilding them all isn’t practical.  In that case, you might have to turn to a P2V tool.

But if you got a VM with physical RDMs, you can’t clone the VM.  You can’t bulk copy the Virtual Machine files over.  You could create new VMDKs and copy everything out of the RDM disks to those and reassign drive letters.  However, this SQL VM was nasty with complex mount points and drive letters assigned.  We had to get it done the weekend the RDMs were discovered.

Solution?  VMware Converter!  I tried installing it on an admin server and set up the job.  That of course failed because of Murphy’s Law.  The Converter agent wouldn’t install due to insufficient permissions.  I installed it directly on the SQL VM (with the same account I tried to push the agent, mind you), stopped the SQL services to ensure the data was static, and ran it.  Other than it shuffling a few drive letters around on the converted VM that a few mouse clicks fixed, it worked like a champ.

How about you?  Any useful utilities you’ve used recently you haven’t used in awhile?

Making life easier using vSphere Tags

One of the least used features in vSphere that I think almost all admins could really make use of but don’t is the ability to create custom vSphere tags within vSphere.

I wanted to take the time to point this feature out, and perhaps give people some ideas on how to make use of of them.  This can help with management and automation quite a bit.

What are vSphere Tags?

vSphere Tags are effectively custom metadata type info that can be applied to objects within vCenter.  You get to make your own to fit your own needs.  They assist basically with locating objects for more efficient administration and management.

They’re unique to other things such as folders for your VMs in that you can assign multiple tags to the same VM or other objects.

Let’s break this down by comparing vSphere tags to MP3 management software like iTunes.  An individual MP3 file must be in one file system folder or another.  It can’t be in both.  But suppose you want to find all your songs by an artist, by genre, or by album?  We intuitively understand this now with MP3s.

But we have the same problem with VMs.  You can organize your VMs into VM folders in vCenter, but a single VM can only be in one folder or another.  What if you wanted to organize your VMs by criticality?  By whether or not they have SQL?  Whether or not they need to be backed up?  Trying to do this with folders would be a nightmare to manage.  Plus, remember a VM folder is the mechanism for assigning permissions, too.  Maybe you don’t want this metadata having any impacts on anyone’s permissions to manage it.

That’s when you use vSphere tags!

Use Cases for vSphere Tags

Use cases for this functionality are numerous:

  • Criticality of VM – this would allow the expedited power up or down of VMs based on this nature.  Running out of resources within your cluster due to sudden host failures?  Power down the non-critical VMs.  It would also be helpful for vSphere Admins who aren’t the application admins to know when to handle a VM with care before doing anything to it.
  • Application groupings – Maybe it doesn’t make sense to put VMs that work together to provide an application or service, but you want to know those groups.  That could allow a SQL server that serves the backend of multiple application groups to be identified for both simultaneously.
  • Presence of a common application like SQL – This can be helpful for locating VMs that may require special settings on backup jobs to quiesce the file system before backing the VM up.  You might also use this to find potential VMs that other VMs are dependent on, so you can set their restart priority so they boot up first in an HA event scenario.
  • Lab/Test VMs – You could set the resource allocation for Lab/Test VMs to low to help ensure they are given less resources than production VMs.

OK, I convinced you (hopefully)!   Let’s make some tags.

Basic Concepts for vSphere Tags You Need To Know

You can create vSphere tags in both within the vSphere Web Client and with PowerCLI.  It’s simple, but you need to know a few concepts.

All vSphere tags belong to a Category.  There are two main types of categories.  This notion is called Cardinality.  It sounds more complicated than it is.  Basically, you can have a category where only a single tag from that category can be applied to any given object.  For example, let’s say you want to tag VMs by criticality.  Logically, a VM will only have one criticality rating, not multiple.  IE, it makes zero since for a VM to be both low and medium as far as how critical they are.

However, sometimes you might want a category that multiple tags could apply to the same object.  For example, let’s say you want to make a category called “Special Applications” to identify very specific apps within a VM to easily identify SQL servers, Domain Controllers, and Exchange servers.  While I wouldn’t recommend it, it’s possible for a single VM to be all three simultaneously.

vSphere tags can apply to all kinds of objects as well, not just VMs.  You can select which objects a tag can be applied to within the category.

Managing vSphere Tags Using the Web Client

To create tags within the vSphere client, navigate to the Tags section of the web client.

vm tags web client nav

You must create a category first if there isn’t one already made.  Click the Categories button, and then click the create categories icon.

For this example, we will make a category for criticality ratings for VMs.  We want one tag per object, not more, and we only want the tag to be applied to VMs or vApps.

vsphere category example

Now that we have our category, we can create tags within it.  Click on Tags, and the new tag icon.  Be sure to select the category during tag creation.

vsphere tags create tag example

Rinse and repeat for all the tags you want to create for the category.  One tip I recommend is to name the tags with incuding their category name, which refers to some kind of concept.  Since you usually search by the tag name, you want for example LowCriticality instead of Low.  (See below for search examples.) Low in and of itself could mean a lot of things.  Low resource usage, low criticality, etc.

To apply a tag to an object, simply right click the object, point to Tags & Custom Attributes > Assign Tag…

vsphere tags assign tag

A new dialog box appears where you can filter categories or see all categories and select the vSphere tags you wish to assign.  Also, notice you can remove tags here, too.

Managing vSphere Tags Using PowerCLI

PowerCLI has full tag management functionality within it, too.

Creating a category:

New-TagCategory -Name VMCriticality -Description "Criticality of the VM" -Cardinality Single -EntityType "VirtualMachine","VApp"

Creating a tag:

New-Tag -Name "LowCriticality" -Description "Non-Critical VMs" -Category VMCriticality

Assigning a tag to a VM:

get-vm Shoretel | New-TagAssignment -Tag "HighCriticality"

You can do lots of things with PowerCLI and tags.

Using vSphere Tags

Now that you have tags created and applied, you can now make use of them to make your life easier.

You can make use of tags in both the vSphere Web Client and via PowerCLI.  To find all VMs with a tag within the vSphere Web Client, simply type the tag value in the search box.  The tag name will automatically populate.

vsphere tags searching

Click on it.  Boom, you got your objects with that tag!

vsphere tags search results

There’s also a parameter on PowerCLI’s Get-VM cmdlet to identify the VMs with that tag.  You can then pipe that to another cmdlet.  Say for example you want to shutdown your non-critical VMs because you suddenly experience multiple host failures, so you need to make sure your more important VMs get the resources they need:

Get-VM –Tag “LowCriticality” | Shutdown-VMGuest

Imagine if you set up vSphere tags to identity all your VMs with SQL.  Imagine you’re setting up Veeam backup jobs, and you need to know which VMs you need to setup special quiescing.  You could easily just get that list of VMs.

That’s how to use vSphere tags!

How do you think you might be able to use them, or how do you use them within your environment?

desk treadmill

Desk treadmill activity review – 2016

It’s a new year!  I know I haven’t mentioned my walking status for awhile, mainly because I forgot to.  My desk treadmill also had numerous problems, too.  After working with Lifespan extensively along with quite a bit of arm twisting on my TR1200-DT3, they finally agreed to upgrade my desk treadmill to the TR5000-DT3 model at a reduced price.  Basically, the motor kept going out, despite maintaining it properly, and my use being well within the specifications. The TR5000 has a more powerful motor.  So far, the new desk treadmill runs great.  However, I didn’t encounter issues with my TR1200-DT3 until about six months in, so the jury is still out.  I’m going to post a separate piece about my issues, in case it helps others.

I’m writing this post now without any idea about how many steps I actually walked in 2016.  I fully expect it to be lower though because of the above problems.  My treadmill has down for half the year.  I can’t do much about that, and how long it took Lifespan to get a solution.

As I did before, I’m compiling a list of all my weekly progress of steps tracked by my Lumo Lift, which helps me keep good posture that helps avoid neck pain, but also tracks the number of steps I take surprisingly accurately.  Roughly, 2,000 steps = 1 mile.

Obviously, not all walking was done on my desk treadmill, but a lot of it was.  When you see dips in weekly steps, that’s likely a business trip, vacation, I exercised some other way other than anything that would trigger detected steps, or I was outright lazy for whatever reason.  The prolonged downturn in the spring through fall was due to the treadmill breaking down.

Week (ending of)StepsMilesNotes
Total33279451640
1/1011032855
1/1711026755
1/2412103960
1/318339541
2/710225251
2/149996049
2/215303326Business trip
2/2811231256
3/66218131Business trip
3/136537432Business trip
3/2010373251
3/2811486857
4/311640858
4/1011806059
4/172958714Death in family
4/246831934Treadmill broke down
5/186614
5/83162115
5/154076320
5/223778518
5/294361421
6/53539217
6/123086915
6/192004310
6/262680613
7/33262916
7/10174398
7/17199949
7/245088425
7/31143977
8/72403812
8/142067210
8/212717013
8/287455837
9/44643623Vmworld, forgot Lumo charger
9/11129886
9/183108915
9/253580717
10/210803254Treadmill finally replaced!
10/911185755
10/164291921Treadmill broke again!
10/233339216Vacation
10/304468022Business trip
11/66319731Treadmill replaced!
11/137737238Business trip
11/209490647
11/279339946
12/410059050
12/119256546
12/1810776453
12/2510227051
1/110023250

You can clearly see that having an operational treadmill makes a huge difference!

Well, my step count is significantly lower than 2015.  I don’t feel too bad about that though because the treadmill broke down multiple times.

Still, 1640 miles in a year is quite a lot!  To give you an idea, that’s about the distance from Richmond, VA to Denver, CO!

I gained some weight unfortunately.  2016 was a very stressful year, and a lack of a convenient method of exercising while working didn’t help.  I do need to eat better.  That’s my goal for 2016.

Assuming my desk treadmill doesn’t break down again, I’m setting a goal of 2500 miles.  That’s roughly the distance from Richmond, VA to San Diego, CA.  I also am now going to try to couple walking with eating right to see how my weight does.  Just from my experiences in 2015 and 2016, walking this much alone doesn’t seem to allow me to lose weight.  Not that I ate terribly, but I didn’t watch what I ate closely either.

And as I finish this post, I just completed my 7.5 miles of walking for the day!  364 more days of walking to go!