Unable power on VM - ESXi 5.5.0

[ManoelH]

I have a RHEL 6.9 in a virtual machine, after I ran the "yum update" command, I can not connect to VM anymore.

In the BIOS it shows that it did not find the disk. Follow the attached log.

Can you help me?

[ManoelH]

daphnissov​ as far as I was with a RedHat engineer remotely, he asked me to change the SCSI control type from "LSI Logic Parallel" to "Vmware Paravirtual."

I do not know if this could have caused the name change of VMDK. But even changing the controller type continued with the same problem.

[ManoelH]

bluefirestorm​ thanks your time and patience, in which file should I change these parameters?

[daphnissov]

I asked when this VM first stopped booting correctly. Do you have that information?

Also, get to an SSH session and cd to the directory with your VM's files. Show output of ls -lh.

[daphnissov]

Also, I was mistaken here. You appear to have two VMDKs from the outset. The first is named VM-Linux-yspp0097-LDAP.vmdk and the second is VM-Linux-yspp0097-LDAP-PROD.vmdk. Note the addition of "PROD" on the end. It appears that the first file may have been renamed to _2 and the second file named to _1. Please show output in the VM's home directory of all files.

[ManoelH]

daphnissov​

Q: "I asked when this VM first stopped booting correctly. Do you have that information?"

A: I do not have the exact time of the first stop, but it was the first restart of yesterday (01/26).

Q: "Also, I was mistaken here. You appear to have two VMDKs from the outset. The first is named VM-Linux-yspp0097-LDAP.vmdk and the second is VM-Linux-yspp0097-LDAP-PROD.vmdk. Note the addition of "PROD" on the end. It appears that the first file may have been renamed to _2 and the second file named to _1. Please show output in the VM's home directory of all files."

A: I believe I know the answer, probably previously it should have the name:

VM-Linux-yspp0097-LDAP

And at some point in the past I must have renamed myself to:

VM-Linux-yspp0097-LDAP-PROD

With this change it only changes the label and not the name of the disk, when I did the vMotion of the machine it automatically changes the name of the disk equal to the name of the label.

[daphnissov]

Not a vMotion but a storage vMotion, yes, then that's probably what happened. Still, let's take a look at the files inside the home directory. After that, cat each .vmdk (not the *-flat.vmdk files though) and show the contents.

[ManoelH]

daphnissov​

command: ls -lh

command: cat VM-Linux-yspp0097-LDAP-PROD_1.vmdk

/vmfs/volumes/588def62-7ba161cc-4403-b4b52f6fd380/VM-Linux-yspp0097-LDAP-PROD # cat VM-Linux-yspp0097-LDAP-PROD_1.vmdk

# Disk DescriptorFile

version=1

encoding="UTF-8"

CID=e20e1be6

parentCID=ffffffff

isNativeSnapshot="no"

createType="vmfs"

# Extent description

RW 104857600 VMFS "VM-Linux-yspp0097-LDAP-PROD_1-flat.vmdk"

# The Disk Data Base

#DDB

ddb.adapterType = "lsilogic"

ddb.deletable = "true"

ddb.geometry.cylinders = "6527"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

ddb.longContentID = "3be786eff1e2b1e11c5c6907e20e1be6"

ddb.toolsVersion = "0"

ddb.uuid = "60 00 C2 93 df 57 ed 13-1b a2 53 1c f8 8b 4c db"

ddb.virtualHWVersion = "10"

/vmfs/volumes/588def62-7ba161cc-4403-b4b52f6fd380/VM-Linux-yspp0097-LDAP-PROD #

command: cat VM-Linux-yspp0097-LDAP-PROD_2.vmdk

/vmfs/volumes/588def62-7ba161cc-4403-b4b52f6fd380/VM-Linux-yspp0097-LDAP-PROD # cat VM-Linux-yspp0097-LDAP-PROD_2.vmdk

# Disk DescriptorFile

version=1

encoding="UTF-8"

CID=6979ae82

parentCID=ffffffff

isNativeSnapshot="no"

createType="vmfs"

# Extent description

RW 62914560 VMFS "VM-Linux-yspp0097-LDAP-PROD_2-flat.vmdk"

# The Disk Data Base

#DDB

ddb.adapterType = "lsilogic"

ddb.deletable = "true"

ddb.geometry.cylinders = "3916"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

ddb.longContentID = "5bf6fc0e0d3a43ae339282db6979ae82"

ddb.toolsVersion = "0"

ddb.uuid = "60 00 C2 99 9f 01 4f 4f-26 f0 37 04 ad f1 58 2e"

ddb.virtualHWVersion = "8"

/vmfs/volumes/588def62-7ba161cc-4403-b4b52f6fd380/VM-Linux-yspp0097-LDAP-PROD #

[daphnissov]

Ok, that appears to be consistent.

[ManoelH]

daphnissov​ is there anything else I'd like to try?

[daphnissov]

bluefirestorm​ is advising to comment out these masks in the VMX file directly by editing it manually. It's worth a shot.

[daphnissov]

What EVC mode is the cluster configured to use where the host on which this VM resides? Also, attach the latest copy of your VMX file, please.

[ManoelH]

daphnissov​ and bluefirestorm is advising to comment out these masks in the VMX file directly by editing it manually. It's worth a shot.

A: Already made a success for success. The same problem occurred.

[ManoelH]

daphnissov​

[daphnissov]

In your VMX you have PVSCSI set. Change that back to LSI Logic. Your VM won't be able to boot with the PVSCSI option. Once complete, verify both disks are using LSI Logic and reattach the new VMX (or just cat the file and show output).

[ManoelH]

daphnissov​ same problem. I have added all the logs and the current vmx for analysis.

[bluefirestorm]

From the latest log, looks like you are using a different server? Using an even older Westmere CPU instead of the previous Sandy Bridge.

Anyway, it is better to keep to a consistent environment and make few changes at a time.

I do not think it is a problem of VMware ESXi. I don't think it is a problem of the virtual disks either. If it were a problem of the virtual disks or controllers we should have seen a message along the lines of "unable to boot" or "cannot find operating system" or whatever is the normal message for a RHEL VM.

I really suspect it has to do with the yum updates (which may or may not have included a kernel update for Spectre/Meltdown). You need to find out what were in those yum updates before this boot failure.

Red Hat pulled the patches for Spectre/Meltdown because of some systems not booting after the updates.

I would suggest try to go back to previous kernel (if there was a change with the yum updates).

[ManoelH]

bluefirestorm​ yes, I had a kernel upgrade.

There is no way to go back to the previous kernel, it does not even appear the GRUB screen.

[bluefirestorm]

I am not a Linux expert. But I assume you should be able to boot from a LiveCD/ISO/USB and then recover from there. Since you have raised a support request from Red Hat perhaps they can guide you that.

Similar problems on Xen server have popped up for CentOS 6 VM (cannot even go to grub); interestingly, there is someone nicknamed "brasilworks" in the CentOS thread.

https://bugs.centos.org/view.php?id=14336

https://access.redhat.com/solutions/3312501

[daphnissov]

I agree with bluefirestorm here. I wanted to rule out the disk issue, which it sounds like we've done, for if it had been related to disks the boot process would go straight to trying other devices which it doesn't do. It seems pretty clear that whatever yum updates pulled down it hosed this VM and so you need to find out what those are. In the future, since this is a production VM you should probably plan to back it up for cases just like these.

[jefferson342]

Yeah there's nothing wrong there. If your VM was working fine one minute and then you performed a yum update and now it won't boot, it's a problem with packages/kernel that got upgraded, not the VM's configuration that changed.