Updated to 6u2 to fix an issue, maybe I shouldn't have. Rollback or wait?

[WhiskyTangoFoxt]

I've updated our VCenter and 4 hosts running VSAN from ESXi 6 (December 2015 build) to v6 update 2. I haven't updated to VSAN 6.2 disk format v3 yet, and hadn't planned to for some time.

I was surprised to see our new R730xd hosts are not on the 6u2 HCL, nor the H730p adapter on the VSAN 6.2 HCL.

Since updating VCenter and the hosts, whenever I navigate to the VSAN section of VCenter, I'm greeted with:

"TypeError: Error #1009

    at com.vmware.vsphere.client.model.vsan::VmComponentItem()

    at com.vmware.vsphere.client.views.vsan.monitor.virtualdisks::VirtualDisksViewMediator/onPropertyReceived()....

Under monitor->VSAN->virtual disks is empty (unpopulated)

I've got the yellow triangle next to the cluster with the following messages for VSAN health:

• Disk format version - prompting me to move disks from format 2 to 3
• Virtual SAN disk balance - shows 2 disks above median by 3% and 5%
• Controller Driver - Hosts1,2,3,4 -,vmhba0: Avago (LSI) MegaRAID SAS Invader Controller,lsi_mr3 (6.606.12.00-1OEM.600.0.0.2159203),Warning,
• Controller release support - warning
• Performance Service - Stats DB object doesn't exist

I'm tempted to roll back, but worried I may be in an even bigger mess with VCenter database versioning. I do have backups of the hosts and the VCSA pre-upgrade. I'm worried that my VSAN will be in jeopardy if I leave it as it is.

Any advice?

B

[WhiskyTangoFoxt]

OK, so I decided to roll it back, or rather try to...

The restore of the VCenter server from snapshot seems to have went well, the ESXi hosts not so much.

When I attempted to perform a rollback of the first host in maintenance mode I ran into my first roadblock. The current and previous profile are showing the same (3620759). So the easy rollback is out of the question.

Next I attempted a downgrade:

esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-6.0.0-20151004001-standard --allow-downgrades

but was greeted with these errors:

is claimed by multiple non-overlay VIBs:.. and vsan_6.0.0-2.34.3563498 requires esx-base >= 6.0.0-2.34, but the requirement cannot be satisfied within the ImageProfile.

So now I have 4 VSAN hosts running 6 update 2 and one VCenter running 6 update 1.

Is my only option to place the first host in maintenance mode, reboot, re-install build 3073146 clean, then restore from pre-upgrade backup? If so, would the re-connection to VDS and VSAN be seamless?

Thanks,

B

[Techie01]

To revert esxi to previous version, can you try VMware KB: Reverting to a previous version of ESXi  . That should work ( if you have patched/updated esxi 60 Update2 from a previous 60 release version) . 

I assume that your decision to go for downgrade was correct mainly because the storage controller being not supported in 60 Update 2. Storage controller compatibility is extremely important for vsan.

>>

The current and previous profile are showing the same (3620759). So the easy rollback is out of the question.

>>

Can you please elaborate what exactly is meant by profile showing same version.

[WhiskyTangoFoxt]

Good morning,

I tried the rollback already, as it has worked in the past on lab VSphere hosts, both 5.5 and 6, but this one isn't giving me any option to revert. The update was done through VUM. I'm wondering if it is a bug or a result of newer VSAN VIBs being implemented.

This is what I'm seeing when trying to rollback:

I have backups of the hosts before and after the update. I was thinking of doing a reinstall on one host at a time, and restoring the backup. I'd rather not though. I don't want to risk messing up the VSAN or the VDS.

[WhiskyTangoFoxt]

Asuming that I have no other choice but to restore each host from pre-update backup, what problems am I likely to encounter during the process? The current locations of vms residing on hosts are different now than they were when I performed the pre-update backup.

How will a restore affect their respective locations in the cluster?

[WhiskyTangoFoxt]

Anyone??

I really do want to go back but I'm reluctant to move forward with this being as they are production VSAN hosts.

[douglasarcidino]

This is overall a pretty big deal. Have you contacted VMware GSS and opened a ticket?

As far as the errors you were receiving in the UI, can I assume you were using a Windows vCenter server? A lot of the UI errors would have gone away after a reboot of the Windows vCenter server. This happened in my home lab. the support issue is something you can discuss with GSS but would prevent you from getting back to 6.0 U2 for the time being. They might just be behind on updating hosts for support.You should upgrade your vCenter back to U2, and then downgrade each of your hosts one at a time and then downgrade vCenter again. There are some KB articles about re-adding hosts to a VSAN cluster after doing a rebuild.

VMware KB: Adding a host back to a Virtual SAN cluster after an ESXi host rebuild

[WhiskyTangoFoxt]

Hi Douglas,

Yes I opened a ticket with VMWare today. I'm actually using the VCSA, and had done a couple of reboots. When I was running 6.0u2 it seemed that the errors were related to VSAN still running in 6.0 mode as well as lack of HCL compatibility. VSAN 6.2 being so new, I was not wanting to update the VSAN format right away. I'm in agreement about the downgrade of the hosts, it's just it doesn't seem so straight forward now that the rollback function doesn't work. I'm wondering if the host backup file contains the data needed to do a restore that would seamlessly integrate the host back into it's original cluster membership, restoring VSAN and VDS status as well.

Before this system went to production, I had removed the 4th host from VSAN and VDS, then added it in again. Now I was hoping that a valid host backup would mitigate the need for such an exercise, especially as I would have to do it for each host.

B

[douglasarcidino]

I feel for you buddy. This is why I built my home lab the way I did. It gives me the ability to really test things for my clients before I mess with their stuff. Been through a lot of VSAN issues and I have only had it up for about two months. Been through a ton of Horizon 7 issues too now. Now I feel really good about configuring it for clients though. The UI bugs like what you were seeing are very similar to the ones I saw and a reboot of my Windows vCenter cleared them. Sorry it has not been as graceful with yours. Still surprised VMware is this far along in only a few years. Please update your thread with the results of your support case.