Intro
During the upgrade of the vCenters of one of my customers, I stumbled several times upon the issue down below.
It suggest to check the logs, and to check if the ‘service-control –start applmgmt is running.
On both machines the service applmgmt is running.
I also checked if SSH was on, which was the case.
The Problem & Solution was different then it seems.
For this error I’ve encountered to different types of root causes.
In this procedure I’ve documented the 2 solutions that worked for different vCenter upgrades.
Possible Solution 1: Removing the old vCenter from inventory
After searching the logs, one of the problem was found. From one of our previous upgrades in the vCF environments, the old vCenters that we had were still available and registered in the vCenter. They were turned off and placed in a folder, but they were still in the environment. What happened was that the SDDC-Manager sometimes tried to upgrade the old vCenter vms that we kept as a reference. I could find back in the events of the older vCenter that the SDDC-Manager also created a snapshot and tried to mount the ISO. So in that case, the solution is quite obvious. Remove the vCenters at least from the inventory. You probably can also rename them and then do a storage vMotion, since it seems that the SDDC-manager just simply picks the vCenters based on their name in the VMX file. That’s why that some vCenters were still upgraded in my customers environment eventhough they also had the old vCenters as a reference. The SDDC-Manager just simply picks the first vm that it comes across with the name that he knows that should be upgraded. Sometimes that was the old vCenter and sometimes it was the new.
After I removed them from the inventory, the problem was solved.
Possible Solution 2: vCenter Patch is blocking, Remove the old bundle ID & use the VersionAlias file
So this one is much more trickier, but it is also use-case specific situation. When we did the upgrade to 3.10. our environment was at vCF version 3.9.1. However, in the meantime VMware released a patch for their vCenters because of a VMware Security Advisory (VMSA) report. Most of the vCF environments on version 3.9.1 got a patch update. Because of that patch update, the SDDC-Manager doesn’t recognize the version of the vCenters as starting point for the upgrade, even though it can still simply upgrade. To change this we need to remove the bundle id of vCenter patch within the SDDC-Manager. After that we have to create an alias for the version that the vCenters are using, so that the SDDC-Manager knows through the alias that it can upgrade to the latest version that it downloaded.
First things first
Before you starting using this, do make sure that you are effected by this, and creating a ticket for GSS is maybe always a good idea if you are having troubles with this. Our environment was effected by the following patch. Because of that we were not inline with the vCF versions, even though the patch was actually applied by the SDDC-Manager itself. So this can happen to anyone. To check version requirements for each vCF release, check this link.
Now, if you have verified that you are effected by the patch, make a snapshot & backup of the SDDC-Manager & vCenters (+PSCs) . Always check the bundle id that I’m removing in this procedure is the same that is also blocking your vCenter to be upgraded. If you are coming from a different vCF version, the bundle and patch could be different
Login to the SDDC-Manager with SSH (username vcf).
Steps:
- Snapshot SDDC Manager (without memory) – Do not skip this step.
- Revert the versionalias (please do not use “cp” command, because this will change the file permissions):
cat /home/vcf/backup/VersionAlias.yml > /opt/vmware/vcf/lcm/lcm-app/conf/VersionAlias.yml
- Remove the old bundle (vCenter 6.7u3f – 15976714):
/usr/bin/python bundle_cleanup.py dde897dd-d8d1-4e92-abf2-ce9c37785865
* Note be sure to make a backup VersionAlias file and a snapshot before you are going to start fiddling with it.
The file should look similar to this (we removed some of the components in the files, so yours could be bigger).
After the change and executing the script above that removes the bundle VC/PSC 15976714, you shoul be able to download and install the VC/PSC build 16075168 for vCF version 3.10 (as of the time of writing). If this is not the case, give the SDDC-Manager a quick reboot and check again.
I hope this could be of any help for you.
Samir is the author of vSAM.Pro & a Life enthusiast who works as a consultant in the field of IT. With a great passion for Tech & Personal Development, he loves to help people with their problems, but also inspire them with a positive outlook on life.
Besides that, he is also a big Sport & Music junky that loves to spend a big chunk of his time on producing music or physically stretching himself.
