ESXi 5.1 hosts unable to access iSCSI VMFS Luns

[dbonengel]

Hi, there,

a customer of our's, running three ESXi 5.1 hosts connecting to two HP storage systems via iSCSI, had some planned downtime this weekend.

Upon bringing back up storage and Servers, they found the had no access to the VMFS volumes.

The LUNs can bee seen via the devices tab all right - but they cannot be seen as datastores.If I connect manually, I am only asked to initialise the LUN.

One thing I noticed: In the vmkernel.log, I have lines stating that devices could not be mounted by any fs driver which sounds like a bad thing for me..

Any thoughts what could cause the trouble here? As far as I know no configuration changes have been made, the stuff has been  running nicely for months. I am at a loss....

Thanks in advance

Dirk

[Jayden56]

Hi

Welcome to the communities.

Have you disconnect & reconnected it ? if not please do that & let us know if still having problem.

"You must not lose faith in humanity".

[dbonengel]

Hi, Jayden56,

unmounting the LUNs throws an error.

'Aufruf von "HostStorageSystem.DetachScsiLun" für Objekt "storageSystem" auf ESXi "10.79.7.1" ist fehlgeschlagen.'

(Call of "HostStorageSystem.DetachScsiLun"  for object "storageSystem" failed).

[dbonengel]

On a sideline:

- The partitions seem to be OK to me:

Random example:

~ # partedUtil getptbl /vmfs/devices/disks/naa.6000eb31f4838959000000000000004e
gpt
130541 255 63 2097152000
1 2048 2097151966 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

I'm not sure what to make of this here, however:

~ # vmkfstools --queryfs /vmfs/devices/disks/naa.6000eb31f4838959000000000000004e:1
devfs-1.00 file system spanning 0 partitions.
File system label (if any):
Mode: private
Capacity 512 (1 file blocks * 512), 512 (1 blocks) avail
UUID: 00000000-00000000-0000-000000000000
Partitions spanned (on "notDCS"):
Is Native Snapshot Capable: NO

[a_p_]

Not sure what's causing this, but http://kb.vmware.com/kb/1011387 might help troubleshooting. In case you (your customer) have an active support agreement, I'd recommend to open a call with VMware.

Btw. which storage system do you have in place? Ensure there's no issue with the LUN presentation (e.g. read-only mode, ...).

André

[dbonengel]

We had VMWare support look into the problem. They were clueless as well.

The way it seems, LUN presentation etc. are OK, but the ESXi hypervisors somehow seem to have lost the vmfs filesystem driver/Kernel module. The support guy couldn't tell me how this could happen...

This is happening on the latest ESXi 5.1 GA build with HP customizations, upgraded from 5.0. The admin on site is installing one machine from scratch, support was confident the data would show up then.

[dbonengel]

Just a follow-up on this: Reinstalling an -configuring the ESXi servers did the trick.

VMware support are unable to say how this could have happened, but apparently the servers lost access to the vmfs kernel module/driver.

As I wrote earlier, they have been upgraded to 5.1 (from  5.0) according to the upgrade guidelines, using the HP provided ISO images.

I wonder wether there have been any upgrades to the hypervisor since then that could have caused this behaviour.

As upgrading the server does include a restart at the end (end the servers where working fine after that) I think there must have been an update that went awry somehow.

Most importantly: No data have been lost.