VMware Cloud Community
tascheHyaene
Contributor
Contributor

iSCSI LUNs Vanish...

... without a trace.

I'm running two ESX 3.5 boxes hooked up to a number of EMC FC LUNs and four LUNs on a HP MSA2012i.

Came into work today, the four MSA2012i LUNs are offline.

A few things are sure:

The HP management utility sees the LUNs fine.

Reply
0 Kudos
31 Replies
Chamon
Commander
Commander

If you have to re add the LUN through the add storage wizard it will reformat the LUN and you will lose all data on it. Are there VMs on these LUNs? If not have they been formatted with a file system other then VMFS? If you do not have any data on these luns that you need you can delete the LUNs on your storage device and then re create them and present them back again. After rescanning the hosts so that the originals problem ones are gone.

Sorry I went back to the top of the thread and read it again. You do have data there. How much? Do you have local storage that the data (VMs I assume) that you can move them to through the service console while the LUNs are rebuilt?

Chamon
Commander
Commander

The iSCSI LUNs don't have the same LUN number ID as the FC LUNs do they? If so that is a problem.

Reply
0 Kudos
tascheHyaene
Contributor
Contributor

Looks like I misspoke before. I couldn't see the iSCSI LUNs, only the FC LUNs.

That being said, looks like 100% data loss. I can now add the iSCSI LUNs, but unfortunately they're blank.

Do you have any idea what would cause iSCSI LUNs to drop off like that? Firmware, maybe?

Reply
0 Kudos
Chamon
Commander
Commander

Are the connections going through a switch? Did that drop or reboot? If you have to use the add storage wizard it will reformat the LUN. if it is already a VMFS LUN and has been presented to the esx host a rescan should make it available.

Reply
0 Kudos
tascheHyaene
Contributor
Contributor

New wrinkle:

I added two of the four LUNs back to the ESX infrastructure, blank.

I've shut down the HP MSA2012i, rescanned the ESX boxes, then rebooted the ESX boxes and turned up the MSA2012i, then rescanned the storage devices.

There are two more LUNs which I still can't add from the VIC wizard. I still get the error about not being able to read the partition information.

I can't see the LUNs in the CLI in /vmfs/volumes/, but the advertisement for the LUN is showing up in the VIC and also in /vmfs/devices/disks.

You mentioned something before about masking the LUN - how do I do that?

Reply
0 Kudos
tascheHyaene
Contributor
Contributor

The connections are through a switch, yes, and it's been up and running.

Reply
0 Kudos
Chamon
Commander
Commander

When you shut down the MSA you are doing that. But you would need to make sure on the MSA that the LUNs are not shared to the hosts. Similar to unpresenting the LUN on your FC SAN. I am not sure how that is done on your device. The iSCSI LUNs don't have the same LUN ID or number on the iSCSI LUNs as are on the FC luns do they? If they are they need to be a higher number than the LUN ID for the FC LUNs. Were these formatted VMFS3 before?

Reply
0 Kudos
tascheHyaene
Contributor
Contributor

The iSCSI LUNs are different from the FC LUNs and have higher addresses than the FC LUNs.

Yes, they were formatted VMFS3 before.

I guess I can see about stopping the advertisement from the HP iSCSI device before I shut it down.

Reply
0 Kudos
Chamon
Commander
Commander

you shouldn't have to shut it down just un present and then rescan ---> present and rescan again. But it seams that you have a different problem because it wants you to add storage which will wipe out the data. After the rescan and it is seen by the adapter it should be available. Do the hosts have read and write permission to the LUNs?

Reply
0 Kudos
tascheHyaene
Contributor
Contributor

Ok, turned access off at the HP MSA2012i, rescanned the adapter; sure enough, it dropped.

Turned rw access back on again on the HP MSA2012i, rescanned the adapter, it showed up in Storage Adapters, but it didn't show up in Storage as it was before. Didn't show up at all.

Reply
0 Kudos
tascheHyaene
Contributor
Contributor

I'm going to close this one out.

Thanks for your time and energy, I appreciate it.

Reply
0 Kudos
drhagan
Contributor
Contributor

I had this same issue. Came in one morning and all the VM's on particular LUN accessed via iSCSI were unavailable.

At first I could still see the datastore name in the list of datastores, but I could not explore it and see if there was still data on it.

So I rescanned my hba and then it disappeared altogether. The only way seemed to add it back and then of course reformat and lose everything. This is what we did as the VM's were for testing. This would not have sufficed on our production network of course so I continue to troubleshoot.

Previously 4 ESX hosts could see this LUN and then I came back from a vacation and only one could. I dismissed it at the time but it seems obvious now the lun ID must have changed adn the datastore was resignatured. The one ESX host that resignatured it could see it, the others could not. I got out the ol' best practice guide for VMFS Volumes and made sure all my hosts have the same resignature default value.

I understand the resignaturing, but not sure why I lost the datastore completely.

Reply
0 Kudos