vMotion fails because host can't see datastore...but vCenter shows host with datastore mounted

[MattGoddard]

[Equipment: vCenter 6.7; ESXi 5.5 & 6.7; UCS B200-M4 blades; Nimble CS700 SAN serving Fibre Channel LUNs. The hosts have no local storage - they boot directly from the SAN.]

I ran into a problem adding a 5.5 host to a 6.7 vCenter that already has 6.7 hosts in it that use the same datastores. (I got the "datastore has the same URL..." error.) Since the 6.7 hosts were not yet hosting any VMs, I decided to resolve this by removing their datastores, figuring that I could just add them back later, once the 5.5 host was added. So I unmounted and detached them, removed their initiators from the SAN and rescanned. All good: the datastores disappeared from the hosts.

I added the 5.5 host to vCenter. Also good: no more "datastore URL" error.

I added the initiators back to the SAN and rescanned. Again, all good: the LUNs showed up in "Storage Devices" and I was able to attach them, rescan once more and then the datastores showed up. Unusually, they all showed with zero bytes free space. I did "Refresh Capacity Information" and then the free space showed up.

So far, perfect.

Then I then tried to vMotion some VMs from the 5.5 host to a 6.7 host. This failed every time as follows:

Unable to access the virtual machine configuration: Unable to access file [datastore & path]

Unable to access virtual disk 'Hard disk 1' from the host. Either the host is disconnected from the datastore or has insufficient privilege. Details: Unable to access file [datastore & path]

When I connect to the 6.7 host via SSH and do "ls -l /vmfs/volumes" I don't see any FC datastores except for the host's boot volume. However, if I do "esxcli storage nmp device list" I see all the LUNs present and correct. Finally, if I do "esxcli storage san fc list" I see both HBAs with a port state of "ONLINE" and no apparent problems.

I tried rebooting the host but that didn't make any difference. So now I'm stumped! Can you help?

[a_p_]

Does esxcli storage core device detached list show any results?

If it doesn't, run vmkfstools -V (note the upper case "-V") from the host's command line for a rescan, and check the vmkernel.log for related entries.

André

[MattGoddard]

There's no output from "esxcli storage core device detached list".

When I do "vmkfstools -V" and monitor "vmkernel.log" I only see this:

2020-05-27T22:33:12.595Z cpu27:2104127)VC: 4616: Device rescan time 41 msec (total number of devices 5)

2020-05-27T22:33:12.595Z cpu27:2104127)VC: 4619: Filesystem probe time 17 msec (devices probed 4 of 5)

2020-05-27T22:33:12.595Z cpu27:2104127)VC: 4621: Refresh open volume time 0 msec

There should be 15 datastores on this host.

[MattGoddard]

I removed the 5.5 host and now all the datastores say "inaccessible".

So from that I assume that their earlier, normal-looking appearance was just because they were accessible on the 5.5 host, but for whatever reason, vCenter showed them as accessible on the other host too, even though they weren't.

[MattGoddard]

Well, I was able to right-click each inaccessible datastore on the 6.7 host and do "Mount..." and successfully mount each one. But then when I tried to add the 5.5 host I again got the error, "[host] has a datastore that conflicts with an existing datastore in the datacenter". And unfortunately, this version of vCenter doesn't tell you which datastore it's referring to.

I feel like I'm in a Catch-22 scenario!

[MattGoddard]

This may be relevant: if I do this on the 6.7 host in PowerCLI...

$vmHost = Get-VMHost [host] | Get-Datastore

$vmHost.extensiondata.summary.multiplehostaccess

...I get "False" for all the datastores.

If I do the same thing on the 5.5 host I get "True" for all datastores except the boot vol. The same is true for other 6.7 hosts attached to the same SAN.

If that's the root of the issue then I'm not sure how to resolve it. I've never had to worry about setting anything for multiple access for a FC datastore - the SAN has no options for that. It's always simply worked that way by default.

[a_p_]

I'm afraid that you cannot add the v5.5. host to vCenter 6.7 ( see VMware Product Interoperability Matrices).

Although in some cases "not supported" may not necessarily mean that something doesn't work,but in this it looks like this.

André

[MattGoddard]

Apologies: I've been mistyping horribly! The host I'm adding is ESXi 6.5 and the existing hosts are 6.7.

This is all part of a complex, major upgrade from 5.5 to 6.7, keeping track of which is all kinds of fun!

[MattGoddard · Accepted solution]

I've figured out the problem and I think it may be a bug: basically, in vCenter 6.7, if you click a host and then click its 'Datastores' tab in the main pane, the fact that there are datastores in the list does not necessarily mean they're mounted to that host.

I can replicate the problem like this:

• Start with a host that has no mounted Fibre Channel datastores
• On the SAN, add the host's initiators to an existing LUN that has a datastore created in it
• In vCenter, click on the host, then go to Configure -> Storage devices
• Right-click the host -> Storage -> Storage Rescan...:
  • The LUN identifier now appears in 'Storage Devices', but it's italicized and grayed out
• Click the LUN identifier and click 'Attach':
  • It's no longer italicized or grayed out
• Click the 'Datastores' link for the host
• Perform a second storage rescan on the host:
  • The datastore now appears
• At the host's command line, do "ls -l /vmfs/volumes":
  • The datastore does not appear in the list (because it's not actually mounted, even though vCenter implies that it is)
• In vCenter, right-click the datastore and do "Mount Datastore...":
  • The task completes, but there no other visual indication that anything has changed
• Do "ls -l /vmfs/volumes" again:
  • The datastore now appears in the list

After that, everything works normally.

Weird!