ESXi some datastore disappear after host reboot (EMC VMAX Volumes)

[No_Way]

Hi All,

We have a very strange behavior with our EMC(VMAX) iSCSI volumes the are connected to our ESXi hosts(5.1 and 5.5). Some hosts have Emulex cards for iSCSI Adapters, others have iSCSI Software adapters. In both type of hosts we have the same problems.

This happen after we create a new Storage Group in the EMC and use the same iqn Groups(with the same hosts that have already some other Volumes from another Storage Group), then when we reboot a host, we cannot see most of the datastores anymore(some have the new ones, some have the old ones, that are from different Storage Groups). I know when we reboot will clean or close iSCSI connections and with this some Datastores will not show, but the strange is that if we go to the Devices all Volumes are there(in all hosts). But if we try to add, no Volumes show to add to the host.

I even try to change the on some volumes Round Robin to Fixed to see if fix the issue, no luck. All path shows as active and are green, so no issues here.

I even double check the Volumes configuration and connection(the ones that are visible and the ones are not visible) using "esxcli storage core device world list" and "esxcli storage core device list -d" , but dont see anything different between them.

Trying to see anything in VMware, but did not find anything related to issues like this.

Any help or seen some similar behavior will be a great help.

Thank You

NW

[CoolRam]

i think from the ESXi front its or LUN is visible. can you check the size of LUN in the device and path . If the size is zero than problem is from the storage side .

[No_Way]

Hi CoolRam,

First thank you for your reply.

I don't think I follow you.

In the devices all LUNs are visible, and all have 4 paths(2 for each interface) and have the right size.

Honestly I never seen LUNs that present to host with zero size.

Thank You

NW

[a_p_ · Accepted solution]

A zero sized "LUN" may show up if you don't present a data LUN with a Host LUN ID 0 (zero). In this case the Management LUN (formerly LUN "Z") shows up as LUN 0.

Anyway, you said that you presented LUNs from two storage pools to the same hosts. How did you configure the "Host LUN IDs" for the different LUNs? Are these unique across both storage groups, or are some of the LUNs presented using the same Host LUN ID?

Note: I'm not familiar with the VMAX, but I assume it's similar to a VNX.

André

[No_Way]

Hi a.p,

Thank You for your reply.

OK understood the zero sized LUN. Not here, because all have their normal size and LUN ID.

Is a little bit different in VMAX. In my opinion VNX is easier.

But there is IQN Groups and Mask Views.

In simple terms

IQN Groups = hosts vs iqn

Mask Views = Were Storage Groups will be visible.

So there is 2 different IQN Groups(divided by hosts like are in clusters in the vCenter)

2 different Storage Groups(the old ones and new the ones that was created)

So each Storage Groups is set to each iqn groups(one Storage groups can have both iqn groups) or Mask View can have both iqn Groups.

Don't know if I explain in a way to understand.

NW

[No_Way]

Hi,

Strange or not, connecting directly to the host I can see all volumes and ca add

NW

[Sreejesh_D]

Just to check. Is storage array connected to the hosts only through iSCSI? or is there a connectivity through FC as well?

There are some known issues if the same array connected to the hosts through different protocols.

[No_Way]

Hi Yezdi,

Thank for your reply.

No all hosts have iSCSI only(some with software adapter, some using Emulex iSCSI cards)

[No_Way]

Hi all,

After some troubleshooting I have check the same LUNs were presented to other vCenter and hosts.

After removing and set the proper permissions and configurations, we can see the LUNs again in devices, and also if I try to add a Datastore the LUNs are visible to add again(only as new).

Normally the hosts should pick up the datastore and put online and available, but for some reason they did not, so if I try to add the LUNs again as a Datastore this will reformat and destroy all data(there is still VMs in it), so my question is, how to add the LUNs again without formating them??

One example:

/dev/disks # ls -l | grep naa.60000970000292603672533030333136

-rw-------    1 root     root     2319282339840 May 28 13:24 naa.60000970000292603672533030333136

-rw-------    1 root     root      25148928 May 28 13:24 naa.60000970000292603672533030333136:1

-rw-------    1 root     root     209715200 May 28 13:24 naa.60000970000292603672533030333136:2

-rw-------    1 root     root     209715200 May 28 13:24 naa.60000970000292603672533030333136:3

-rw-------    1 root     root     2318690942976 May 28 13:24 naa.60000970000292603672533030333136:4

lrwxrwxrwx    1 root     root            36 May 28 13:24 vml.02000400006000097000029260367253303033313653594d4d4554 -> naa.60000970000292603672533030333136

lrwxrwxrwx    1 root     root            38 May 28 13:24 vml.02000400006000097000029260367253303033313653594d4d4554:1 -> naa.60000970000292603672533030333136:1

lrwxrwxrwx    1 root     root            38 May 28 13:24 vml.02000400006000097000029260367253303033313653594d4d4554:2 -> naa.60000970000292603672533030333136:2

lrwxrwxrwx    1 root     root            38 May 28 13:24 vml.02000400006000097000029260367253303033313653594d4d4554:3 -> naa.60000970000292603672533030333136:3

lrwxrwxrwx    1 root     root            38 May 28 13:24 vml.02000400006000097000029260367253303033313653594d4d4554:4 -> naa.60000970000292603672533030333136:4

Also checking LUN

/dev/disks # voma -m vmfs -f check -d /vmfs/devices/disks/naa.60000970000292603672533030333136

Checking if device is actively used by other hosts

Running VMFS Checker version 0.9 in check mode

Initializing LVM metadata, Basic Checks will be done

         ERROR: Missing LVM Magic. Disk doesn't have valid LVM Device

         ERROR: Failed to Initialize LVM Metadata

   VOMA failed to check device : Not a Logical Volume

~ # esxcfg-mpath -L | grep naa.60000970000292603672533030333136

vmhba37:C1:T1:L4 state:active naa.60000970000292603672533030333136 vmhba37 1 1 4 NMP active san iqn.1998-01.com.vmware:sunblade-06-6c739bc6 00023d000002,iqn.1992-04.com.emc:50000974083961e4,t,1

vmhba37:C0:T1:L4 state:active naa.60000970000292603672533030333136 vmhba37 0 1 4 NMP active san iqn.1998-01.com.vmware:sunblade-06-6c739bc6 00023d000001,iqn.1992-04.com.emc:50000974083961e4,t,1

vmhba37:C1:T0:L4 state:active naa.60000970000292603672533030333136 vmhba37 1 0 4 NMP active san iqn.1998-01.com.vmware:sunblade-06-6c739bc6 00023d000002,iqn.1992-04.com.emc:50000974083961a0,t,1

vmhba37:C0:T0:L4 state:active naa.60000970000292603672533030333136 vmhba37 0 0 4 NMP active san iqn.1998-01.com.vmware:sunblade-06-6c739bc6 00023d000001,iqn.1992-04.com.emc:50000974083961a0,t,1

~ # esxcfg-mpath -b -d naa.60000970000292603672533030333136

naa.60000970000292603672533030333136 : EMC iSCSI Disk (naa.60000970000292603672533030333136)

   vmhba37:C0:T0:L4 LUN:4 state:active iscsi Adapter: iqn.1998-01.com.vmware:sunblade-06-6c739bc6  Target: IQN=iqn.1992-04.com.emc:50000974083961a0 Alias= Session=00023d000001 PortalTag=1

   vmhba37:C1:T0:L4 LUN:4 state:active iscsi Adapter: iqn.1998-01.com.vmware:sunblade-06-6c739bc6  Target: IQN=iqn.1992-04.com.emc:50000974083961a0 Alias= Session=00023d000002 PortalTag=1

   vmhba37:C0:T1:L4 LUN:4 state:active iscsi Adapter: iqn.1998-01.com.vmware:sunblade-06-6c739bc6  Target: IQN=iqn.1992-04.com.emc:50000974083961e4 Alias= Session=00023d000001 PortalTag=1

   vmhba37:C1:T1:L4 LUN:4 state:active iscsi Adapter: iqn.1998-01.com.vmware:sunblade-06-6c739bc6  Target: IQN=iqn.1992-04.com.emc:50000974083961e4 Alias= Session=00023d000002 PortalTag=1

LUN 4 and active, cant find any issues here. So host should add the LUN in the host as datastore automatically,

So how can I add this without formating the LUN?

Thank You

NW

[Spyingjack]

Hi Can you please check HLU id  in masking view of VMAX.

Can you post symacess -sid <last3digitofvmax > show view <maskingview name>

please check the host lun number  under storage group, may be 2 lun have same HLU

thanks