Monthly Archives: September 2015

Routing VMotion Traffic in vSphere 6

One of the new features that is officially supported and exposed in the UI for vSphere 6 is routed VMotion traffic.  This blog post will identify what the use cases are, why there was difficulty leading up to vSphere 6, and how vSphere 6 overcomes it.

Use Cases

So why would you want to route Vmotion traffic, anyway?  Truthfully, in the overwhelming majority of cases, you wouldn’t and shouldn’t route Vmotion traffic for various reasons.

Why?  Remember a few facts about Vmotion traffic:

Additional latency, delays, and reduced throughput exponentially reduces vSphere performance.  When a Vmotion operation gets underway, the running contents of memory for a VM is copied from one host to another and changes in the VM’s working set of memory are monitored for changes.  Interative copies continue to copy changes until there is a small enough delta difference that those small differences can be copied very quickly.  Therefore, the longer a Vmotion takes, the more changes in the working set accumulate, and the more changes that accumulate, the longer the operation will take, which invites even more changes to occur during the operation.  Adding an unnecessary hop in the network can only reduce Vmotion performance.  Therefore, if you are within the same datacenter, it is almost certainly the case that routing Vmotion traffic is ill advised at best.  About the only situation I could possibly think this might be a good idea is if you have a very large datacenter with hundreds of hosts, which causes performance deteriortation because of too many broadcasts within a single LAN segment, but you infrequently need to Vmotion VMs between hosts within different clusters.  But you would need A LOT of ESXi hosts that may need to Vmotion between each other before that would make sense.

So when would routed Vmotion traffic make sense?  Vmotioning VMs between datacenters!  Sure, you could stretch the VMotion Layer 2 network between the datacenters with OTV instead, but at that point, you are choosing the lesser of two evils – Vmotioning with a router between hosts in different datacenters, or the inherent perils of stretching an L2 network across sites.  The WAN link will take the bigger toll over an extra hop in the network by far, so there’s no question here the better choice would be to route the Vmotion traffic instead of stretching the Vmotion network between sites.

This is important because cross vCenter VMotioning is now possible, too, and Vmware has enabled additional network portability via other technologies such as NSX, so the need to do this is far greater than in the past, when the only scenario routing Vmotion traffic would make sense is in stretched storage metro clusters and the like.

Why was this a problem in the past?

If you’ve never done stretched metro storage clusters, this may never have occurred to you because there was pretty much never a need to route any kernel port group traffic other than host management traffic.  The fundamental problem was ESXi had a single TCP/IP stack, with one default gateway.  If you followed best practices, you would make multiple kernel port groups to segregate iSCSI, NFS, Vmotion, Fault Tolerance, and Host Management traffic, each in their own segregated VLANs.  You would configure the host’s default gateway as an IP in the host’s management traffic subnet, because you probably shouldn’t route any of that other traffic.  Well, now we need to.  Your only option to do this would be to create static route statements via command line to make this happen on every single host.  As workload mobility increases with vSphere 6 cross-vCenter Vmotion capabilities, NSX, and vCloud Air, this just isn’t a very practical solution.

How does VMware accomplish this in vSphere 6?

Very simple, at least conceptually anyway.  ESXi 6 has the capability of having multiple independent TCP/IP stacks.  By default, there already exists separate TCP/IP stacks for Vmotion and other traffic.  Each can be assigned separate default gateways.

Simple to manage, and configure!  Just configure the stacks appropriately, and ensure your kernel port groups are configured to use the appropriate stack.  Vmotion port groups should use the Vmotion stack, while pretty much everything else should use the default stack.

How cool is that?

vSphere 6 – Certificate Management Intro

I like VMware and their core products like vCenter, ESXi, etc.  Personally, one thing I really admire is the general quality of these products, how reliable they are, how well they work, and how VMware works to address pain points of them to make them extremely usable.  They just work.

However, certificate management has been a big pain point of the core vSphere product line.  There’s just no way around it.  And certificates are important.  You want to ensure the systems you’re connecting to when you manage them are those systems.  For many customers I’ve worked with, because of the pain of certificate management within vSphere, the fact that some customers are too small and don’t have an on premise Certificate Authority, and to ensure the product continues to work, they often don’t replace the default self-signed certificates generated by vSphere.

That’s obviously less than ideal.  The good news is certificate management has been completely revamped in vSphere 6.  It’s far easier to replace certificates if you like, and you have some flexibility as to how you go about this.

Three Models of Certificate Management

Now, you have several choices for managing vSphere certificates. This post will outline them.  Later, I’ll show you how you can implement each model.  Much of this information comes from a VMworld session I attended called “Certificate Management for Mere Mortals.”  If you have access to the session video, I would highly encourage viewing it!

Before we get into the models, be aware that certificates can basically fall under one of two categories – certificates that facilitate client connections from users and admins, and certificates that allow different product components to interact.  Also, vCenter also has built in Certificate Authority functionality within it.  That’s a bit obvious since you already had self-signed certificates, but this functionality has been expanded.  For example, you can allow vCenter to act as a subordinate authority of your enterprise PKI, too!

Effectively, this means you have some questions up front you want to answer:

  1. Are you cool with vCenter acting as a certificate authority at all?  The biggest reason to use vCenter is it is easier to manage certificates this way, but your security guidelines may not allow it.
  2. Are you cool with vCenter being a root certificate authority should you be cool with it generating certificates?  If not, you could make it a subordinate CA.
  3. For each certificate, which certificate authority should generate them?  Maybe your security requirement that the internal PKI must be used is only for certificates viewable on client connections as an example.

From these questions, typically a few models emerge for certificate management.  You effectively have four models that emerge, which is a combination of your vCenter acting as a certificate authority or not, and which certificates it will generate.

Model 1: Let vCenter do it all!

This model is pretty straight forward.  vCenter will act as a certificate authority for your vSphere environment, and it will generate all the certificates for all the things!  This can be attractive for several reasons.

  1. It’s by far the easiest to implement.  It will generate all your certificates for you pretty much, and install them.
  2. It’ll definitely work.  No worries about generating the wrong certificate.
  3. If you don’t have an internal CA, you’re covered!  vCenter is now your PKI for vSphere.  Sweet!  You can even export vCenter’s root CA certificate, and import it into your clients using Active Directory Group Policy, or other technologies to get client machines to automatically trust these certificates!  Note that it is unsupported for vCenter to generate certificates for anything other than vSphere components.

Model 2: Let vCenter do it all as a subordinate CA to your enternal PKI

Very similar model to the above.  The only exception is instead of vCenter being the root CA, you make vCenter become a subordinate CA for your enterprise PKI.  This allows your vCenter server to more easily generate certificates that are trusted automatically by client machines.  Yet it also ensures that certificates are still easily generated and installed properly.

However, it is a bit more involved than the first model, since you must create a certificate request (CSR) in vCenter to submit to your enterprise PKI, and then install the issued certificate within vCenter manually.

Model 3: Make your enterprise PKI issue all the certificates

Arguably the most secure if your enteprise PKI is secured, this model is pretty self-explanatory.  You don’t make use of any of the certificate functionality within vCenter.  Instead, you must manually generate all certificate requests for all vCenter components, ESXi servers, etc., submit them to your enterprise PKI, and install all the resulting certificates for each yourself.

While this could be the most secure way to go about certificate management, it is by far the most laborious solution to implement, and it is the solution that is most likely to be problematic.  You have to ensure your PKI is configured to issue the correct certificate type and properties, you have to install the right certificates on the right components, etc.  It’s all pretty much on you to get everything right!

Model 4: Mix and match!  (SAY WHAT?!?!?)

When I first heard this being discussed in the session, my immediate reaction by my security inner conscious was, “This sounds like a REALLY bad idea!!!”

But as I listened, it actually makes quite a bit of sense when done properly.  You can mix and match which certificates are and are not generated by the PKI components within vCenter.  However, the model that makes sense if you go hybrid (a hybrid solution doesn’t make sense for everyone!) would be to allow vCenter to manage the certificate generation for all certificates that facilitate vSphere component communication, but use either Model 1, 2, or 3 for all other certificates that facilitate client connections.  Should this meet your security requirement, it meets the best of both worlds – certificates issued by your internal PKI that your clients automatically trust and thereby (potentially) more secure, but ease of management and better reliability for all the certificates that clients don’t see for internal vSphere components.

Which should you go with?

I hate using the universal consultant answer, but I have to.  It depends.  If you don’t have an internal PKI, go with Model 1.

If you have an internal PKI just because you had to for something else, and you want easy trusting of vSphere connections by your clients, go with model 1 and import vCenter’s root CA into your client machines, OR go with Model 2.  Which one in this case?  If you don’t consider yourself really good at PKI management, or if you don’t need many machines to be able to connect to vSphere components, probably Model 1.  The more clients that need to connect, the more it might lean you towards Model 2.

Do you have security requirements that prevent you from using vCenter’s PKI capabilities altogether?  You have no choice, go with Model 3.

I would generally try though for people who think they need to go with Model 3 to look at Model 4’s hybrid approach.  Unless you absolutely have to go with Model 3, go Model 4.

Hope this helps!

Long time no blog

Sitting down here to start writing out some blog posts.  Obviously, if you’ve kept up with my blog at all, you’ll know I haven’t been keeping with my schedule of trying to do three blog posts a week.

This is because:

  1. I think three blog posts a week is overly ambitious on my part.  It kinda led me a bit to burn out a bit on blogging, but honestly, it also made me try to churn out posts instead of stuff I want to do more of when I blog – more in depth posts about topics. That didn’t help keep me motivated to do it. So for now, I’m going to try to post twice a week from here on out, but if I feel so inclined, you might see three.
  2. It’s hard to post at VMworld on an iPad.  Or maybe it’s just hard for someone starting out trying to blog regularly to do it while at a conference.  I dunno, but what I do know is I didn’t blog like I wanted to there.
  3. Our dog, Megan, who we had for 14 years, we had to put down.  We don’t have any children, so she was like a child to us in a way.  It was heartbreaking to try to figure out what was wrong, what to do, watching her everyday, etc.  My wife also took it really hard, as they were especially joined to the hip, so to speak.  It wasn’t a quick thing, and I just think, probably understandably, I had zero motivation to blog going through that.  The very next week, my wife had shoulder surgery, so a lot of the day to day stuff she does fell on me for a while, so something had to take a back seat, so blogging was one of them.  Now, I’ve got my proverbial crap together in my head, so I’m ready to go!

So, expect to see some more blog posts you hopefully find valuable coming.

Let’s do this!

VMworld Day 1 – Recap

So much for live blogging VMworld.  I need to find something to post to WordPress from my ipad, as the web editor doesn’t work when the web bandwidth isn’t good…  Actually, the web editor isn’t good on iOS, period.  Oh, well.

Monday was more labs, Solutions Exchange, and sessions.  The general session, VMware stated it’s goal is to make a single logical cloud that could span public and private clouds, where you could run all apps, both enterprise apps we have had for years, and the new “cloud native apps” of today and increasingly in tomorrow.

So most of the 23,000 attendees were greeted with a well produced but a bit weird video that looked like something cooked up by somebody smoking a substance still illegal in most states watching X-Men, as this guy…

cloudprofxWas teaching the young mutant…err…cloud native apps and enterprise apps to hone their powers in security, performance, flexibility, and more.

We learned that we would now be able to vMotion applications between vCloud Air and your private VMware cloud potentially… Cool!

We learned that SRM would now be offered as a cloud offering in conjunction with vCloud Air as well.  Also, very cool!

They also announced vSphere Integrated Containers, and discussed Photon, which is a VMware optimized linux container technology that will interoperate with other container technologies, such as Docker.  It’s good to see VMware embrace a technology that is a bit of a counter to their bread and butter – VMs.  Resisting change is often futile.

Also, an EVO SDDC Manager was announced, which will help automate the management of all components of the Software Defined Data Center, including network virtualization and virtualized storage within VSAN, in a single pane of glass.

Upgrades to VSAN have also been announced, and one of the biggest improvements will be the ability to stretch a VSAN across datacenters, effectively making a stretched storage cluster with synchronous replication.  Considering how much solutions like VPLEX cost to do the same thing, this could potentially be a much lower cost option for organizations looking for this type of DR protection.

I’ll have more on specific sessions later, but I wanted to get this out in the meantime.


VMworld Day 0 – Update

Sorry about the late post from yesterday, but I was too exhausted from disembarking from the cruise, getting to VMworld, blah blah blah.

Sunday was a good day to get some quick sessions in, and do a lot of labs.  There’s not enough here to do a lot of posts, so here’s a quick summary of Sunday for me.

  • VMware certifications – Expect VCIX exams for Data Center Virtualization to be available January and February. Design will be first, followed shortly after with Administration.
  • Dell FX line of servers are an interesting piece of hardware.  I’ll do a future blog post about them, but they present an interesting solution for a few scenarios.
  • I played around quite a bit with VSAN in the labs, particularly around policy based management scenarios.  I’m sure that will be another blog post coming soon.

Much more from Day 1 coming…