vCenter Upgrade Error: `Exception Occurred in precheck phase`

Troubleshooting

VCSA 7.0 has moved the upgrade process logging to a new location — the log itself is now at /storage/log/vmware/applmgmt/update_microservice.log (actual) or /var/log/vmware/applmgmt/update_microservice.log (symlink)

This appears to be a rough order of operations with this new update process:

From there, it picks up the scope for the upgrade, and verifies against common upgrade issues:

In this case, everything looks good. I’m not really sure why it needs the SSO Administrator password, and there isn’t much on-line about this. We’re seeing three errors after we hit go time:

This is pretty odd, because it’s indicating a “resumable error” despite the fact that it cannot resume until a file lock is removed. Here are the errors I see:

• Empty Stage Location: Unsure what this means, given the context. Odds are the upgrade script cannot find out where to stage RPMs (Red Hat Package Manager).
• Package discrepancy error: It could be relating to the above, or it could be a failed checksum. No other logging is generated by the agent to indicate what’s wrong.
• Can't read JSON file /storage/core/software-update/stage/stageDir.json: This one's more actionable! It looks like there's no directory by this name.

Easter Egg: statsmoitor probably should be statsmonitor

VAMI saves the installation state as a file in /etc/applmgmt/appliance/software_update_state.conf:

{ "state": "INSTALL_FAILED", "version": "7.0.1.00200", "latest_query_time": "2020-12-21T00:19:32Z", "operation_id": "/storage/core/software-update/install_operation" }

VAMI will be stuck in a loop until you remove this file as root:

rm -rf /etc/applmgmt/appliance/software_update_state.conf

This will not necessarily resolve the issue that caused the failure, however, more work still needs to be done.

Install via ISO

We’re going to try a fallback method, attaching the upgrade ISO. The following snippet is from the vSphere UI, modifying vCenter’s VM Hardware:

From there, simply click “Check CD-ROM” and it will immediately appear.

This time, we know what directories to search, so I’m going to watch the logs:

tail -f /var/log/vmware/applmgmt/update_microservice.log | grep -i err

Attempt via Command-line with ISO

VMware documents the following method to update via the command line https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vcenter.upgrade.doc/GUID-8466F019-C57C-4344-9E15-8CFF74A6E4C2.html

Stage Packages

We’re going to try and clear the (empty) workspace and try fresh, auto-accepting EULAs:

Let’s take a look at the update:

Let’s run it!

Looks like the manual install worked for me — 7.0 U1c

TL;DR

rm -rf /etc/applmgmt/application/software_update_state grep -i error /var/log/vmware/applmgmt/update_microservice.log exit software-packages unstage software-packages stage software-packages list software-packages install shutdown reboot -r "patch reboot"

Originally published at https://blog.engyak.net.