uninspired
Contributor
Contributor

iSCSI LUNs Connected but Datastore Not Visible/Available

Hey all-

We've got a cluster consisting of 3 ESXi hosts. All three are configured indentically, with three BroadCom ports all connecting to an Equallogic array with 8 datastores (7x 2TB LUNs for VMs, and 1x LUN for ISOs) via software iSCSI adapter (using Dell multi-pathing module). I installed the latest firmware and tools updates (from the end of January) today, and now two of the datastores are missing (not that it matters, but disks 1 and 5). The LUNs are visible on the Static Discovery tab of the iSCSI Software Adapter properties page (and in the Details pane in Configuration > Storage Adapters when the iSCSI Software Adapter is selected), and the Equallogic shows active connections from all three IP addresses.

I've tried removing all of the datastores and re-discovering them all, I've tried rescanning the adapters, and I've tried Add Storage (Configuration > Storage), but I can't get the datastores to appear in the inventory.

Any thoughts on how to get the datastore to re-appear?

Thanks

-Chris

0 Kudos
10 Replies
AndySimmons
Hot Shot
Hot Shot

Any chance a rogue Windows server stomped on your VMFS volume? If you can see the LUN, but the datastore is gone, the VMFS is likely wiped out.

That doesn't mean you can't recover it, sometimes you just need to change the partition type back to VMFS. See this post. You'll need to enable the shell if you haven't done so already, but this should still work in ESXi 5.

-Andy VCAP5-DCA, VCP-DV 4/5, MCSE, space camp graduate.
0 Kudos
uninspired
Contributor
Contributor

Thanks for the input. I forgot about a crucial piece of information: The other two ESXi hosts still see the datastore and there are active, operating, VMs on that datastore. It's only the single host (that was just updated) not seeing the datastore. So, I know the integrity of the vmdk is solid, it's just not popping into the inventory. And, of course, now I can't migrate any VMs on that datastore to that host.

So, long story long, the datastore is not gone, I just can't get this host to recognize it and add it to inventory.

Thanks again

-Chris

0 Kudos
christianZ
Champion
Champion

Have you check the vmkernel log file for any errors/warnings?

Reg

Christian

0 Kudos
AndySimmons
Hot Shot
Hot Shot

Just to be clear, if the partition type is inadvertently changed, any ESXi hosts that had already identified that datastore as VMFS will continue to be able to see that datastore. However, if those hosts are rebooted, they will no longer be able to see them.

If you check with fdisk -l from the problematic ESXi host, that will tell you for sure.

-Andy VCAP5-DCA, VCP-DV 4/5, MCSE, space camp graduate.
uninspired
Contributor
Contributor

The results of the fdisk show something different about the two affected volumes, but I'm unable to interpret.


~ # fdisk -l

Disk /dev/disks/naa.6090a04870aeb18352d28456f5771e0b: 2199.0 GB, 2199036887040 bytes
255 heads, 63 sectors/track, 267351 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

                                           Device Boot      Start     End      Blocks  Id System
/dev/disks/naa.6090a04870aeb18352d28456f5771e0bp1             1         2     13 195+  fb  VMFS

Disk /dev/disks/naa.6090a04870aea1d636d2a445b577ee74: 2199.0 GB, 2199036887040 bytes
255 heads, 63 sectors/track, 267351 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

                                           Device Boot      Start     End      Blocks  Id System
/dev/disks/naa.6090a04870aea1d636d2a445b577ee74p1             1         2     13 195+  fb  VMFS
/dev/disks/naa.6090a04870aea1d636d2a445b577ee74p2             3    267351 2147480779+  fb  VMFS

Disk /dev/disks/naa.6090a04870aea14371d4746b387b0e02: 2199.0 GB, 2199036887040 bytes
255 heads, 63 sectors/track, 267351 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

                                           Device Boot      Start     End      Blocks  Id System
/dev/disks/naa.6090a04870aea14371d4746b387b0e02p1             1         2     13 195+  fb  VMFS
/dev/disks/naa.6090a04870aea14371d4746b387b0e02p2             3    267351 2147480779+  fb  VMFS

Disk /dev/disks/naa.6090a04870ae81f7e3d224b8f378eeb3: 2199.0 GB, 2199036887040 bytes
255 heads, 63 sectors/track, 267351 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

                                           Device Boot      Start     End      Blocks  Id System
/dev/disks/naa.6090a04870ae81f7e3d224b8f378eeb3p1             1         2     13 195+  fb  VMFS
/dev/disks/naa.6090a04870ae81f7e3d224b8f378eeb3p2             3    267351 2147480779+  fb  VMFS

Disk /dev/disks/naa.6090a04870ae410990d2343a47783e53: 1099.5 GB, 1099526307840 bytes
255 heads, 63 sectors/track, 133676 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

                                           Device Boot      Start     End      Blocks  Id System
/dev/disks/naa.6090a04870ae410990d2343a47783e53p1             1    133676 1073752406   fb  VMFS

Disk /dev/disks/naa.6090a04870aea1f7b2e1742bef93ce2c: 2199.0 GB, 2199036887040 bytes
255 heads, 63 sectors/track, 267351 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

                                           Device Boot      Start     End      Blocks  Id System
/dev/disks/naa.6090a04870aea1f7b2e1742bef93ce2cp1             1         2     13 195+  fb  VMFS
/dev/disks/naa.6090a04870aea1f7b2e1742bef93ce2cp2             3    267351 2147480779+  fb  VMFS

Disk /dev/disks/naa.6090a04870ae710493e0f46d3c923e1d: 2199.0 GB, 2199036887040 bytes
255 heads, 63 sectors/track, 267351 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

                                           Device Boot      Start     End      Blocks  Id System
/dev/disks/naa.6090a04870ae710493e0f46d3c923e1dp1             1         2     13 195+  fb  VMFS
/dev/disks/naa.6090a04870ae710493e0f46d3c923e1dp2             3    267351 2147480779+  fb  VMFS

0 Kudos
AndySimmons
Hot Shot
Hot Shot

Those look a little strange to me. Every LUN that you haven't put in bold starts off with a 2-block VMFS partition. That's 512B useable after you lose 512B for the VMFS header. After that, there's a larger VMFS partition that takes up the remaining blocks, which is obviously where all of your VM data is stored. It sounds like those are the LUNs that are working right now. Out of curiosity, were these partitioned by hand? I'm just not sure what purpose those tiny partitions could serve.

At any rate, here's what's going on with the first device you put in bold text:

/dev/disks/naa.6090a04870aeb18352d28456f5771e0b

This just has a 2-block VMFS partition on it right now. If a working ESXi host shows a different partition layout, chances are you've lost the partition table, and need to rebuild it to match what the working hosts are showing before they reboot and re-read the new table.

Here's the 2nd device you put in bold:

/dev/disks/naa.6090a04870ae410990d2343a47783e53

This is a "normal" looking VMFS partition table, although it sounds like it's inaccurate. The entire LUN is partitioned as a VMFS volume. Again, I'd be curious to see what a working host shows for the partition table here.

Be extremely careful, but there is a VMKB article explaining how to use fdisk to restore VMFS partition tables. They recommend you call VMware support if you are uncomfortable with any part of the process.

It wouldn't surprise me if you created a 2nd VMFS partition on naa.6090a04870aeb18352d28456f5771e0b to use blocks 3 - 267351 and that one started working.

Likewise, I wouldn't be surprised if you overwrote the table for naa.6090a04870ae410990d2343a47783e53, creating VMFS partitions from blocks 1 - 2, and blocks 3 - 133676, and that one starts working. That is purely speculation, based off of what I see working on the other LUNs, and I would only act on that hunch if you can't get the correct layout from the other hosts, and you have no other option.

Good luck!

-Andy VCAP5-DCA, VCP-DV 4/5, MCSE, space camp graduate.
0 Kudos
uninspired
Contributor
Contributor

My guess is the vss-control partitions are the small blocks at the beginning of each volume.

Good idea to check the volumes on the *working* hosts. Results are interesting, if not particular useful. It looks the same on the ESXi hosts that are currently functional. It is, in fact, misreporting the size. The Config > Storage section shows it as 2TB.

Disk /dev/disks/naa.6090a04870ae410990d2343a47783e53: 1099.5 GB, 1099526307840 bytes
255 heads, 63 sectors/track, 133676 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

                                           Device Boot      Start         End      Blocks  Id System
/dev/disks/naa.6090a04870ae410990d2343a47783e53p1             1    133676 1073752406   fb  VMFS

I'm going to migrate all of the VMs to another datastore so I can safely try repairing the partition table without having to worry about losing anything.

Thanks for your input. I'll report back here with results once I've safely moved all of the VMs elsewhere.

0 Kudos
AndySimmons
Hot Shot
Hot Shot

Interesting -- good luck!

As a side note, VMFS partitions are typically aligned on a 64KB boundary for performance reasons. I just figured I'd mention that now since it sounds like you're pulling all of the VMs off of those datastores, and can afford to make some destructive partition changes on those LUNs. This alignment is done automatically when creating a new VMFS volume from the vSphere client, or this doc explains how to do it with fdisk.

-Andy VCAP5-DCA, VCP-DV 4/5, MCSE, space camp graduate.
0 Kudos
sunny1986
Enthusiast
Enthusiast

Hi What did  you do to get the datastore back?

0 Kudos
MaciekTX
Contributor
Contributor

I have the same issue. How was that resolved?

0 Kudos