BlogPost Series – Creating a Roadmap for vSphere Upgrades:
Phase 5.1 – Technical vSphere Challenges

By Samir of vSAM.pro | | No Comments | BPS: Creating Roadmap for vSphere Upgrades, VMware, vSphere Upgrades

Intro
Without knowledge, one cannot make reliable (technical) decisions. Quite simple right. That’s why it is so important that you do your research well and spend enough time gathering information about the current environment. It makes the previous phases invaluable, and it shall act as your foundation for the upcoming phases. At this stage, you should have grasped a good understanding of how the environment works, and how it is made up. It is time to go through some technical decisions.

In this part of the BlogPost Series, we’re going to talk about: the different upgrade and migrations paths, how to make use of some of the new features, and the pitfalls they can have. We also cover some upgrading & migration challenges, sketch a few scenario’s and give you some good stuff to think about. I do want to note, that since version 5.1 up til version 6.7, there have been lots of changes. Unfortunately, I don’t have the time to go trough each feature or change. I will however go trough the most important challenges and changes. Especially the difficult ones that can’t be changed afterwards. Without Redeploying halve the environment of course. With that being said. Let’s go.

BlogPost Serie Overview:
✓          Phase 1: Kick-off
✓          Phase 2: Assessment
✓          Phase 3: Defining the IST (Current State)
✓           Phase 4: Dependencies & Compatibility
⇒          Phase 5.1: Technical vSphere Challenges
⚬           Phase 5.2: Common vSphere Challenges
⚬           Phase 5.3: Solving vSphere Compatibility Challenges with VVD
⚬           Phase 6: Defining the Soll & Roadmap (Future State)

Upgrading, Migrating, both?
When upgrading towards a higher vSphere version, there are several roads that we can take. We can do:

  1. In-place upgrade(s) of vCenter(s) & ESXi hosts
    • The vSphere Environment, will have one or several upgrades. This way we maintain settings, configuration, logs and historical (meta)data.
  2. Lift & Shift,
    • You build a new environment and migrate everything once you’re done. It is the cleanest way of doing things, and it gives you the opportunity to redefine your vSphere environment. Sometimes new features can only be leveraged if the vSphere environment is rebuild.
  3. Small Upgrade, and then lift & Shift
    • Handy when the environment is really far behind and the new vCenter cannot manage the older hosts.
      Example: a vCenter of 6.5 can only manage hosts up till version 5.5, hosts with version 5.1 cannot be managed directly. In this case a small upgrade can help to make the hosts get managed by the new environment.

Deciding which road is best suitable for you, can depend on many factors, but I found that the following factors are the most common and important reasons for organizations to choose one road over another. These factors are:

  1. Dependencies & Compatibility
  2. New vSphere Features & Changes
  3. Available Resources (Example: Compute & Storage)
  4. Downtime & (App) Availability
  5. Rollback Options
  6. VMware vSphere Stack Update Sequence

Before we go more in-depth about these 6 technical challenges, lets us first start with an overview of the paths that we can technically take in the upgrade scenario.

Supported Paths for Upgrading
I made a simple illustration that shows the possible roads that you can take when upgrading a vCenter and ESXi host. At the time of writing that is of course. This also depends on vCenter his ability to manage older host until a certain version.

The general rule of thumb is, that you can only upgrade to a maximum of 1.0 major release. So version 5.0 –> 6.0 or 5.5 –> 6.5. Do note that there is a difference between v6.0 u2, and v6.0 u3. You can’t upgrade directly from version 5.0/5.1 to 6.0 u3. This also means that a vCenter of 6.0 u3 cannot manage hosts below 5.5.

This brings us to the manageability, which I illustrated here.

Do note that these illustration are made during the time of this writing. This mean that this info can change, since VMware updates quite regularly. Therefor I always suggest that you should check this information on the official VMware page. If you also want to know more about this stuff, check out the following screenshot and official page of VMware. Where you can find the interoperability of the different versions and VMware products.

https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#upgrade&solution=2

Challenges – New vSphere Features & Changes
We now know what kind of jumps we can make, but there are still lots of challenges. Some more difficult, then others, which also can depend on the version you are. Lets list up all the important changes that can affect your Upgrade path, and how to deal with them.

vSphere Challenge – Changing towards the vCSA (vCenter Server Appliance)
The vCSA has been out there for actually quite a while. However, since version 6.x a lot has changed. Especially with version 6.5, which lead to a full feature-parity between the vCSA and  the Windows vCenter Server. Since version 6.x, VMware has been moving forward with the vCSA and with that, also started to say goodbye to the vCenter Server running on Windows machines. For some, this is maybe old news, but it is still important since this will affect your upgrade path if you’re coming from older versions. Lets sum up some facts:

    • vCSA version 6.5, has Full Feature Parity and even supports more than the original Windows vCenter Server.
    • Windows vCenter Server is supported until version 6.7. After that it will be vCSA only, according to VMware.
    • Migrating a Windows vCenter Server from 5.5 to version vCSA 6.0, is more of a headache then it is towards 6.5. Besides vCSA 6.5 has full feature parity, the 6.5 upgrade installer also has a build-in migration tool. Which is especially handy for Windows vCenter Servers who also have an external database. If you still want to upgrade first to the vCSA 6.0 for whatever reason, then you can use the following fling. https://labs.vmware.com/flings/vcs-to-vcva-converter. The fling is an award winning idea from 2013, that helped customers with the migration traject towards vCSA. Since it was so helpful, it eventually became fully developed and has turned into an official build-in tool in vCSA 6.5. So like I said, if you can, migrate to vCSA 6.5. It is the best and easiest way.
    • During a vCenter upgrade towards 6.5 or 6.7, you can select if you want to use the new vCenter Appliance. This will make use of the Migration Tool, that will create a new vCSA that imports all the settings from your previous vCenter, as well for the database.
    • The Migration tool helps you with keeping the settings, but because of that you cannot change the (VMware) SSO domain, which will therefor stay vsphere.local. This (probably) also mean that the FQDN of the vCenter Server, IP adres, cannot be changed during the migration, but I haven’t double checked that.
    • If you want to make use of a new (VMware) SSO name, you’ll need to manually create a new vCenter server, and migrate your existing workload towards the new environment.
    • Even though you keep most of the settings and data with the build-in migration tool in 6.5, be sure that any dependency or integration of 3th party apps, can deal with this change. For instance, Veeam backup solution or a Citrix VDI integration, will not know about the new vCSA, and maybe need some reconfiguration. Just keep that in mind.

vSphere Challenge – Update Manager
vCSA 6.0 doesn’t support Update Manager, in that version, you still have to have a separate Windows Machine to run Update Manager. Update Manager is supported as an embedded service on vCSA from version 6.5. So I suggest that you should upgrade to that version, or higher if possible.

vSphere Challenge – PSC (Platform Service Controller)
The Platform Service Controller is new since version 6.0. It takes the SSO service, that was introduced in version 5.1/5.5, up towards a new level. The PSC is quite important, and it can also be deployed in two flavors. Embedded with the vCSA, or external. Now this can be a big topic, and I want to keep it short so that we can focus on the upgrade. The most important give away is, that VMware is sort of changing its approach, and seems to move away from the External PSC idea. As from version 6.7, the embedded PSC supports the same kind of features as an embedded PSC, and this makes its deployment a lot easier. If you want to know more about all the external PSC Topology types, check out this awesome blogpost.
http://www.electricmonk.org.uk/category/vmware/vsphere-6/psc-replication/

It is really well written and explains everything you need to know. Keep in mind that this topic of an external PSC is more important for version 6.0 and 6.5. At version 6.7 it changes a lot again, and some reasons are not that valid anymore. I also used an awesome PSC decision flow chart in the past, for version 6.0. Also definitely worth checking out: https://blogs.vmware.com/vsphere/files/2016/04/vSphere_Topology_Decision_Tree_Poster-v5_0804016.pdf

Besides that, there is a kb article that describes the deprecated topologies for PSC in version 6.5. https://kb.vmware.com/s/article/2147672

Now Lets Sum up some of the most important facts for your Upgrade:

  • The SSO service is part of the new PSC
  • In version 6.x a new (VMware) SSO domain can be created during installation. This isn’t possible during an upgrade, and when upgrading from version 5.x the SSO domain will stay vsphere.local.
  • A (VMware) SSO can only be created or joined during installation of a new vCenter. After the install, it cannot be changed! Be sure that you join the SSO domain of a second vCenter if you want to. Creating the same (VMware) SSO domain, doesn’t makes them joined, and just means there are 2 separate SSO domains with the same name. Be sure that you don’t name the (VMware) SSO domain, the same as your corporate (Windows) Domain. This can lead up to some unexpected problems.
  • In version 6.0 and 6.5, a PSC can only join another (VMware) SSO domain, if it is using external PSC’s. If one of the two vCenter is using an embedded PSC, it cannot be joined.
  • In version 6.7, a PSC can only join another (VMware) SSO domain, if it is embedded.
    I know….
  • Upgrading a Windows vCenter Server 5.x with an embedded SSO, towards a vCSA 6.5 version with an external PSC is not directly possible. Before you upgrade, you need to repoint the SSO and make it external. It takes several steps to make it work, so make sure if you want to go through that challenge.
  • If a PSC is Synced

A simple summary of the above is, that the easiest way for getting multiple vCenters joined in the same (VMware) SSO Domain, is by creating 2 new vCenters. Where the first vCenter sets up the (VMware) SSO domain, and the second vCenter joins it. Easiest and best would be version 6.7, since from that version on, the embedded PSC is the only option that allows others to be joined in the SSO domain. If you only have one vCenter or don’t feel the need for multiple vCenters to be joined to the same (VMware) SSO domain, then you can skip this all. I would then recommend an embedded PSC, if your situation isn’t a too specific use-case, but mostly because we seem to move back towards the embedded PSC anyway. However, there are still valid use-cases for using an external PSC topology, but that is up to you to decide. Most use-cases will probably suffice with an embedded PSC in the future, but that is just my honest opinion. Especially since the release of version 6.7.

vSphere Challenge – ELM (Enhanced Linked Mode)
Enhanced Linked Mode, one of the features that many customers look forward to. It is really a cool and handy feature, but also one feature that has lots of misconceptions and challenges. So let’s sum them up:

  • To start, Enhanced Linked Mode is only possible if the vCenters are (VMware) SSO Domain Joined. So all the PSC challenges that we talked earlier about, are directly linked to the usability of ELM. If you can’t make the vCenters (VMware) SSO joined, then you can’t use ELM between different vCenters. So it is really important that you think beforehand if you want to use ELM, since it cannot be done that easily afterwards.
  • ELM is a “single pane of glass” experience. Which means that you can manage and see both vCenters and their objects through one view.
  • ELM cannot be used in a way to make vCenter Servers redundant in their operations. This means that, if one vCenter is down, the other vCenter can’t see or manage the objects of that other vCenter anymore. It is purely a view sharing thing. One cool thing though, is that you can sync the databases between the PSCs. This mean, that if one PSC of a vCenter is down, the vCenter can still be reached through the other PSC (Mostly applicable for v6.0 and v6.5). This means that the downtime of one ore multiple PSC’s, doesn’t have to lead towards a downtime for a vCenter.
  • The other side of ELM is, that vCenters can impact each other viewing and management performance in the client. If one vCenter is starting to have some problems, it can impact the speed of viewing or managing objects. Even if the objects are not in the vCenter that have the problems. This is not really documented, but it can happen, and me and my colleagues have also seen it already a couple of times. So even though ELM is more a View Sharing thing, and not a way to make vCenter operations or administration redundant, it can still impact the performance of your webclient when you’re experience some problems with one of them. So it is something that you should keep in mind.

vSphere Challenge – VMFS6 and UNMAP
Another cool new thing, is the VMFS6 filesystem that has been introduced in version 6.5. With VMFS6, 4K drives are now supported, but also the UNMAP feature is back again. For those that don’t know, UNMAP is a feature that helps to reclaim deleted space on the OS as well on the datastore. This is mostly important for thin provisioned disks as well as thin provisioned datastores. In the past, if something was deleted on a datastore or within an OS, the higher layer didn’t knew that the space was available again, and thus marked the space as still being claimed. This leads up to higher storage consumption, since there was no way of reducing the consumption and therefor thin provisioned disks or datastores, has always kept growing. With UNMAP that same space is simply said, being zeroed out and VMware detects the freed-up space as available again. UNMAP has been introduced into the vSphere ecosystem already for several times (5.x) and has been deprecated several times as well. Mostly because of Storage performance issues, but I think that it is suffice to say that in version 6.5 and especially in 6.7, VMware has found the sweet spot. Anyway, Lets Sum up a few challenges that come with these new changes:

  • VMFS5 cannot be upgrade towards VMFS6. It means that you have to delete the old VMFS5 datastore and need to create a new VMFS6 datastore. That is the only way, and it means that you need to have some additional resources or space left for this to work.
  • VMFS6 was released at version 6.5. For now it means that only version 6.5 or higher can make use of VMFS6. Versions prior to that can only use VMFS5.
  • UNMAP, like earlier said, works on 2 places and runs in the background. It works on the Operating System level, as well on Datastores.
  • Not every OS supports the feature UNMAP, and thus it means that you need to check if your operating system supports it.
  • In vSphere 6.0 UNMAP wasn’t enabled automatically, and could only be run manually trough command line. This has been changed in 6.5 as an automatic feature that can be turned on.
  • Do note that in vSphere 6.0, the amount of Operating Systems that support or could use UNMAP commands is also lower. Especially for Linux operating systems, since the support for the iSCSI protocol was on a lower version, due to the limit of the (vm) Hardware version. In 6.5 this has changed and thus more operating systems could successfully transfer the UNMAP commando’s trough the iSCSI protocol, were before, a lot of them failed.
  • Making use of VMFS6 capabilities, like UNMAP, can mean that you have to change some storage settings. Especially if you make use of special storage capabilities through a vendor. So definitely check if your vendor recommends a change in configuration if you go for VMFS6. For example, some storage systems make use of deduplication. The last thing that you want to do, change some settings to make use of UNMAP, only to find out that it will interfere with the original design & features of the vendor. At the other hand, sometimes you need to make changes in your configuration in order to make new VMware features and vendor capabilities work together or even enhance each other. A simple change for example, can be the way of provisioning (Thin vs Thick). It is therefore always wise to test these changes and to consult your vendor. So, make sure that when you’re changing to VMFS6, that you’ll re-evaluate your storage design.

Alright that’s it for it now. We’ve discussed quite a lot, but there are still a few things that I want to talk about before I continue with the next phase. So in the next post we will continue with Part II of this Phase.

Hope you enjoyed it and that you got some value out of it.
Until the next time😁,

See ya.

↑↑ Follow me on my Socialz ↑↑ - Or - ↓↓ Care & Share ↓↓

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.