VMware Cloud Community
JaapHoetmer
Enthusiast
Enthusiast

vSphere 6.0 to 6.5 upgrade - VMWare Tools installation not possible, symlink errors

Hello all.

I upgraded my standalone 6.0 to 6.5, without apparent problems, and I tried subsequently to update VMWare Tools in my Windows VM. This failed with an error message at the ESXi web interface:

The required vmware tools ISO image does not exist or is inaccessible.


The 6.0 installation I had before _did_ allow me to install the VMWare Tools package in this manner, so this apparently has changed due to the upgrade.

I then logged in to my host using SSH, and detected that a number of symlinks were broken.

I tried to find information on why these symlinks are broken, why the VMFS volumes were not there, but I can't find anything.

The system is rather simple, I use an SD card to boot the hypervisor from, the rest is on one volume using a mirrored raid configuration on a HP Proliant MicroServer.

pastedImage_1.png

When I listed the /vmfs/volumes contents, the actual volumes that are broken presented input/output errors. Often these i/o errors are associated with hard drive problems, however, there are no disk problems, all is healthy from a hardware level; the HP iLO storage status page tells me both RAID members are fine, no errors.So maybe something else is wrong?


[root@ESXI-HOME:/vmfs/volumes] ls -al

ls: ./24bb26ec-c9665d30-f22d-ac27f319a285: Input/output error

ls: ./539116af-b19c09ae-6407-3464a99acaf0: Input/output error

ls: ./946e92e8-49ccba1c-6064-11cfe06e29f4: Input/output error

total 1028

drwxr-xr-x    1 root     root           512 Jan 10 21:51 .

drwxr-xr-x    1 root     root           512 Jan 10 10:03 ..

drwxr-xr-t    1 root     root          1820 Nov 23 22:12 552fff39-7de2fd47-cc3b-3464a99acaf0

lrwxr-xr-x    1 root     root            35 Jan 10 21:51 datastore1 -> 552fff39-7de2fd47-cc3b-3464a99acaf0

If anyone has ideas on

  1. why this has happened
  2. how to fix these vmfs volume issues

that would be greatly appreciated.

Thanks,

Jaap

Reply
0 Kudos
5 Replies
vXav
Expert
Expert

Can you check your partitions and paste the result to see what they look like:

partedUtil getptbl "/vmfs/devices/disks/mpx.vmhbaXX:C0:T0:L0"

(Replace XX with the id of the disk on which ESXi is installed)

I ran into the same issue a couple months ago but it was because I messed with the partitions.

To fix it I had to recreate the partition with the right part number and copy the files from another host.

Reply
0 Kudos
JaapHoetmer
Enthusiast
Enthusiast

Hello vXav, thanks for your reply.

The output:

[root@ESXI-HOME:~] partedUtil getptbl "/vmfs/devices/disks/mpx.vmhba33:C0:T0:L0"

Unable to get device /vmfs/devices/disks/mpx.vmhba33:C0:T0:L0

A listing of the disks, as you can see I have a 64 GB SD card that is used as the vSphere boot disk, and a 2TB volume for virtual machines.

[root@ESXI-HOME:~] ls -alh /vmfs/devices/disks/

total 3971910872

drwxr-xr-x    2 root     root         512 Jan 11 16:17 .

drwxr-xr-x   13 root     root         512 Jan 11 16:17 ..

-rw-------    1 root     root       58.6G Jan 11 16:17 mpx.vmhba33:C0:T0:L0

-rw-------    1 root     root        4.0M Jan 11 16:17 mpx.vmhba33:C0:T0:L0:1

-rw-------    1 root     root      250.0M Jan 11 16:17 mpx.vmhba33:C0:T0:L0:5

-rw-------    1 root     root      250.0M Jan 11 16:17 mpx.vmhba33:C0:T0:L0:6

-rw-------    1 root     root      110.0M Jan 11 16:17 mpx.vmhba33:C0:T0:L0:7

-rw-------    1 root     root      286.0M Jan 11 16:17 mpx.vmhba33:C0:T0:L0:8

-rw-------    1 root     root        2.5G Jan 11 16:17 mpx.vmhba33:C0:T0:L0:9

-rw-------    1 root     root        1.8T Jan 11 16:17 naa.600508b1001c527374ffd275415f8788

-rw-------    1 root     root        1.8T Jan 11 16:17 naa.600508b1001c527374ffd275415f8788:1

lrwxrwxrwx    1 root     root          20 Jan 11 16:17 vml.0000000000766d68626133333a303a30 -> mpx.vmhba33:C0:T0:L0

lrwxrwxrwx    1 root     root          22 Jan 11 16:17 vml.0000000000766d68626133333a303a30:1 -> mpx.vmhba33:C0:T0:L0:1

lrwxrwxrwx    1 root     root          22 Jan 11 16:17 vml.0000000000766d68626133333a303a30:5 -> mpx.vmhba33:C0:T0:L0:5

lrwxrwxrwx    1 root     root          22 Jan 11 16:17 vml.0000000000766d68626133333a303a30:6 -> mpx.vmhba33:C0:T0:L0:6

lrwxrwxrwx    1 root     root          22 Jan 11 16:17 vml.0000000000766d68626133333a303a30:7 -> mpx.vmhba33:C0:T0:L0:7

lrwxrwxrwx    1 root     root          22 Jan 11 16:17 vml.0000000000766d68626133333a303a30:8 -> mpx.vmhba33:C0:T0:L0:8

lrwxrwxrwx    1 root     root          22 Jan 11 16:17 vml.0000000000766d68626133333a303a30:9 -> mpx.vmhba33:C0:T0:L0:9

lrwxrwxrwx    1 root     root          36 Jan 11 16:17 vml.0200000000600508b1001c527374ffd275415f87884c4f47494341 -> naa.600508b1001c527374ffd275415f8788

lrwxrwxrwx    1 root     root          38 Jan 11 16:17 vml.0200000000600508b1001c527374ffd275415f87884c4f47494341:1 -> naa.600508b1001c527374ffd275415f8788:1

Thanks again.

Jaap

Reply
0 Kudos
JaapHoetmer
Enthusiast
Enthusiast

And here's the info on the individual partitions.

[root@ESXI-HOME:~] partedUtil getptbl /vmfs/devices/disks/mpx.vmhba33:C0:T0:L0:1

unknown

0 255 63 8128

[root@ESXI-HOME:~] partedUtil getptbl /vmfs/devices/disks/mpx.vmhba33:C0:T0:L0:5

unknown

31 255 63 511968

[root@ESXI-HOME:~] partedUtil getptbl /vmfs/devices/disks/mpx.vmhba33:C0:T0:L0:6

unknown

31 255 63 511968

[root@ESXI-HOME:~] partedUtil getptbl /vmfs/devices/disks/mpx.vmhba33:C0:T0:L0:7

unknown

14 255 63 225248

[root@ESXI-HOME:~] partedUtil getptbl /vmfs/devices/disks/mpx.vmhba33:C0:T0:L0:8

unknown

36 255 63 585696

[root@ESXI-HOME:~] partedUtil getptbl /vmfs/devices/disks/mpx.vmhba33:C0:T0:L0:9

unknown

326 255 63 5242880

Reply
0 Kudos
JaapHoetmer
Enthusiast
Enthusiast

Continued my search, and I think it's related to a problem with the HP iLO 4 firmware and the SD card I use to boot ESXi off.

It may be a failing SD card, it may be the firmware. I'll first get a replacement SD card, and download the latest firmware, then I'll get started on testing.

Here's an article that touches similar issues I have:

http://serverfault.com/questions/549253/what-happens-when-the-usb-key-or-sd-card-ive-installed-vmwar...

but I have also found questions of other HP users and advisories from HP indicating similar issues due to iLO firmware bugs, with a recommendation to upgrade iLO 4 to a later version.

So I am narrowing it down, I'll keep posting as I find out more.

Thanks,

Jaap

Reply
0 Kudos
vXav
Expert
Expert

Have you tried to rollback to ESXi 6.0 to see if the issue is related to the upgrade (your components are on the HCL for 6.5?).

To rollback press Shift + R when you see the esxi progress bar at boot time.

You can even try the upgrade again after the rollback, maybe something went wrong the first time.

Reply
0 Kudos