VM_Helmut
Enthusiast
Enthusiast

Booting into maintenance mode and forgetting license information after upgrade to 7u1

Hi,

I upgraded 6.7 to 7.0u1 today on a Dell Poweredge T620 using

VMware-VMvisor-Installer-7.0.0.update01-17325551.x86_64-DellEMC_Customized-A01.iso

The upgrade went fine. I entered the license key, exited the maintenance mode and rebooted the host.
It always boots into maintenance mode and forgets about the license information.

Any ideas? Thank you!

0 Kudos
6 Replies
vXav
Expert
Expert

can you run "ls -al" on the host in / and post the output?

0 Kudos
VM_Helmut
Enthusiast
Enthusiast

~ # ls -la /
total 837
drwxr-xr-x 1 root root 512 Mar 5 13:28 .
drwxr-xr-x 1 root root 512 Mar 5 13:28 ..
-rw------- 1 root root 255 Mar 5 14:44 .ash_history
-r--r--r-- 1 root root 20 Dec 15 12:21 .mtoolsrc
lrwxrwxrwx 1 root root 49 Mar 5 12:59 altbootbank -> /vmfs/volumes/0c20686b-a04a5142-cd99-afdebde7ba67
drwxr-xr-x 1 root root 512 Mar 5 13:00 bin
lrwxrwxrwx 1 root root 49 Mar 5 12:59 bootbank -> /vmfs/volumes/b7c3f84e-424da4a9-4581-61df113a1220
-r--r--r-- 1 root root 348396 Dec 15 12:21 bootpart.gz
-r--r--r-- 1 root root 231484 Dec 15 12:21 bootpart4kn.gz
drwxr-xr-x 16 root root 512 Mar 5 14:44 dev
drwxr-xr-x 1 root root 512 Mar 5 14:00 etc
drwxr-xr-x 1 root root 512 Mar 5 12:59 lib
drwxr-xr-x 1 root root 512 Mar 5 12:59 lib64
-r--r--r-- 1 root root 44662 Mar 5 11:22 local.tgz
lrwxrwxrwx 1 root root 29 Mar 5 12:59 locker -> /var/lib/vmware/osdata/locker
drwxr-xr-x 1 root root 512 Mar 5 12:59 opt
drwxr-xr-x 1 root root 131072 Mar 5 14:44 proc
lrwxrwxrwx 1 root root 28 Mar 5 13:00 productLocker -> /locker/packages/vmtoolsRepo
lrwxrwxrwx 1 root root 4 Dec 15 12:22 sbin -> /bin
lrwxrwxrwx 1 root root 22 Mar 5 13:00 scratch -> /var/lib/vmware/osdata
lrwxrwxrwx 1 root root 28 Mar 5 12:59 store -> /var/lib/vmware/osdata/store
drwxr-xr-x 1 root root 512 Mar 5 12:59 tardisks
drwxr-xr-x 1 root root 512 Mar 5 12:59 tardisks.noauto
drwxrwxrwt 1 root root 512 Mar 5 14:44 tmp
drwxr-xr-x 1 root root 512 Mar 5 12:59 usr
drwxr-xr-x 1 root root 512 Mar 5 12:59 var
drwxr-xr-x 1 root root 512 Mar 5 12:59 vmfs
drwxr-xr-x 1 root root 512 Mar 5 12:59 vmimages
~ #

0 Kudos
vXav
Expert
Expert

There's no "red" in the output of the symbolic links?

Do you have access to /vmfs/volumes/b7c3f84e-424da4a9-4581-61df113a1220  ?

0 Kudos
VM_Helmut
Enthusiast
Enthusiast

"red" is sticky?! No.

~ # ls -la /vmfs/volumes/b7c3f84e-424da4a9-4581-61df113a1220
total 172625
drwxr-xr-x 1 root root 8 Jan 1 1970 .
drwxr-xr-x 1 root root 512 Mar 5 15:05 ..
-rwx------ 1 root root 136861 Mar 5 11:19 b.b00
-rwx------ 1 root root 219280 Mar 5 11:20 bnxtnet.v00
-rwx------ 1 root root 108549 Mar 5 11:20 bnxtroce.v00
-rwx------ 1 root root 1625 Mar 5 11:27 boot.cfg
-rwx------ 1 root root 593682 Mar 5 11:20 brcmfcoe.v00
-rwx------ 1 root root 28678 Mar 5 11:20 brcmnvme.v00
-rwx------ 1 root root 855194 Mar 5 11:21 btldr.v00
-rwx------ 1 root root 7473259 Mar 5 11:21 crx.v00
-rwx------ 1 root root 659 Mar 5 11:22 dell_con.v00
[...]

~ # ls -la /bootbank/
total 172625
drwxr-xr-x 1 root root 8 Jan 1 1970 .
drwxr-xr-x 1 root root 512 Mar 5 15:08 ..
-rwx------ 1 root root 136861 Mar 5 11:19 b.b00
-rwx------ 1 root root 219280 Mar 5 11:20 bnxtnet.v00
-rwx------ 1 root root 108549 Mar 5 11:20 bnxtroce.v00
-rwx------ 1 root root 1625 Mar 5 11:27 boot.cfg
-rwx------ 1 root root 593682 Mar 5 11:20 brcmfcoe.v00
-rwx------ 1 root root 28678 Mar 5 11:20 brcmnvme.v00
-rwx------ 1 root root 855194 Mar 5 11:21 btldr.v00
-rwx------ 1 root root 7473259 Mar 5 11:21 crx.v00
-rwx------ 1 root root 659 Mar 5 11:22 dell_con.v00
[...]

If "red" is sticky, does that help?

~ # dmesg | grep -i stickybit
2021-03-05T13:01:03.516Z cpu11:2100804)ALERT: Failed to complete filtering of stickybit files
2021-03-05T13:29:07.581Z cpu28:2101971)ALERT: Failed to complete filtering of stickybit files
2021-03-05T13:32:02.821Z cpu2:2102517)ALERT: Failed to complete filtering of stickybit files
2021-03-05T13:32:17.893Z cpu12:2102776)ALERT: Failed to complete filtering of stickybit files
2021-03-05T14:01:03.245Z cpu2:2103919)ALERT: Failed to complete filtering of stickybit files
2021-03-05T15:01:03.818Z cpu8:2105701)ALERT: Failed to complete filtering of stickybit files
~ #

0 Kudos
vXav
Expert
Expert

I was asking that because it could indicate broken links due to problems to access the bootbank backing device. Meaning it is running it ram so the changes are not persisted to disk. Some people have had issues with it in vSphere 7 but the u1c was the version that fixed it which is the one you're running but it never hurts to check. Although we didn't prove it is or isn't that..

Could you attach vmkernel.log and vobd.log? It would be a long shot but it could be a storage controller firmware problem, something remotely similar happened to me in the past.

0 Kudos
VM_Helmut
Enthusiast
Enthusiast

Almost forgot, nothing custom but:

~ # esxcli software vib list | grep ntp
fwenable-ntpd 1.2.0 V-Front.de CommunitySupported 2021-03-05
~ #

0 Kudos