ESXi 6.5U1 5969303 - Can't mount iSCSI VMFS 6 volume

[grantph]

I have 400GB VMFS6 volume hosted on FreeNAS 9.3 and connected via iSCSI. The device and partition are visible if I inspect via the ESXi Web UI on the host. I've tried UI, vim-cmd, and esxcli methods to rescan HBAs, refresh devices, etc. and the volume will NOT mount. The same volume mounts correctly on build 5310538.

In the old vSphere Client, I would select Add Storage, Keep Existing Signature, etc. to mount the missing volume. As of the 27 July 2017 update, this is no longer possible.

How do I mount an existing VMFS6 volume via the Web UI (can't find any feature), or via the command line?

The annoying thing is that I have multiple volumes, and some mounted automatically, and some didn't.

(Side rant: Who's idea was it to force users into a substandard experience via the Web UI? It's incomplete, buggy, crash prone, F-grade, poorly tested, bloatware, unworthy of "release" status, etc.  Whoever at VMware is responsible for these decisions should be FIRED).

[grantph]

On closer inspection it appears that ESXi 6.5U1 is NOT reading partitions properly. Below, I've dumped the partitions via UI and esxcli. The UI shows the partition as <blank> AND the diagram as being correct - bizarre. There's something SERIOUSLY WRONG with this update. esxcli shows the same partition tables on both builds, and the volume continues to work correctly on the '538 build.

Here's the output on both builds of ESXi 6.5 ...

ON BUILD 5969303 HOST

Web UI

FreeNAS iSCSI Disk (naa.6589cfc000000bcff79a34187f94af3a)

Type: Disk

Model: iSCSI Disk

Path: /vmfs/devices/disks/naa.6589cfc000000bcff79a34187f94af3a

Capacity: 400 GB

Partition Format: gpt

UUID: 010003000030303063323961373930363631360000695343534920

Partitions

<blank>

Partition diagram

1. VMFS (400 GB)

esxcli storage core device partition list

naa.6589cfc000000bcff79a34187f94af3a 0             0   838860800     0   429496729600

naa.6589cfc000000bcff79a34187f94af3a 1           128   838860767    fb   429496647168

ON BUILD 5310538 HOST

Web UI

Type: Disk

Model: iSCSI Disk

Path: /vmfs/devices/disks/naa.6589cfc000000bcff79a34187f94af3a

Capacity: 400 GB

Partition Format: gpt

UUID: 010003000030303063323961373930363631360000695343534920

Partitions

1: VMFS

Type VMFS

Start block 0

End block 409599

Block size 1 MB

Size 400 GB (409599 blocks)

Partition diagram

1. VMFS (400 GB)

esxcli storage core device partition list

naa.6589cfc000000bcff79a34187f94af3a 0             0   838860800     0   429496729600

naa.6589cfc000000bcff79a34187f94af3a 1           128   838860767    fb   429496647168

[continuum]

Wow - that really looks weired.
Can you please post the result of

partedUtil getptbl /dev/disks/naa.6589cfc000000bcff79a34187f94af3a

[grantph]

Sorry - but I ended up rolling back to 5310538 for now - which works perfectly fine with the same drives (yay!). Don't really have time to mess with this kind of fault. I can run partedUtil on that build - if you think it's of any value.

[grantph]

Okay - you got my curiosity - here is partedUtil on 5310538:

gpt

52216 255 63 838860800

1 128 838860766 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

[continuum · Accepted solution]

Interesting - I assumed that this problem had been fixed already.
The VMFS-partition you have is not properly alligned and will not perform well.
See Why does ESXi 6.5 GUI creates VMFS-volumes that are not alligned to 1MB blocks ?

[grantph]

I suspect you're right - and misaligned partitions are not being read properly by the latest version.

I'll need to perform further tests to confirm. Could be a while before I get back to this. Thanks for the insight!

[grantph]

Recommendation: Check for 1MB partition misalignment BEFORE upgrading to 5969303

I can confirm that we used earlier builds of 6.5 that incorrectly created partitions on 128K boundaries, instead of 1M. Unfortunately for us, that means 39 volumes that are incorrectly aligned - not happy.

A quick way to identify suspect alignment is with the following quick & nasty command:

esxcli storage core device partition list | grep "128"

We won't be upgrading to 5969303 until all our alignment issues are resolved.

[a_p_]

Some quick notes:

• The partition alignment has been fixed with Fling v1.17 of the Host Client (Build 5310538 contains v1.19).
• The latest version (I tested with the Fling version v1.21) indeed seems to have a bug with displaying the partitions in the GUI. However, to me it seems to be a display issue only.

@ grantph​

Are the datastores - which were mounted - aligned to 2048 blocks (1MB), and the ones which didn't show up to 128 blocks, i.e. was it related to the alignment?

André

[grantph]

I had already rolled back to '538 before I was aware of the buggy alignment issue. So, I can't confirm if it was the 128 blocks preventing mounting. However, I do know that the problem we experienced was both UI AND command line - the missing volumes would NOT mount - and in my original post I was looking for guidance on different methods to mount via command line.

I suspect that the volumes were 128, but I wasn't looking for that at the time. A working environment is far more important to us than identifying bugs/causes for VMware.

Until we fix the alignment problems created by the earlier buggy builds, we won't be trying '303 again.

[hostasaurus]

In case it helps anyone else, I wanted to resurrect this old thread with a me too and some additional information.  In my case, I was working with a copy of 5969303 as a fresh install that doesn't have internet connectivity yet, nor is it hooked to vcenter yet.  I wanted to test that some iSCSI LUN snapshots were replicating successfully from one data center to another, so all I wanted to do was boot this copy of 6.5 5969303, mount the snapshot LUN on the local array, boot a VM.  No matter what I did in the GUI, or CLI, I could not mount the iSCSI LUN.  It would show as a storage device in the web interface, but I could not do anything with it, and the actions only allow me to add a new vmfs datastore which would partition and format it.  The GUI did reflect the correct partition size.  esxcfg-volume -l showed nothing.

I got some internet for this host and updated to 7388607.  Reboot.  esxcfg-volume -l and now it shows as available; used -M and successfully mounted.

Perhaps there's some VMFS versioning issue that affects certain builds?