ESXi lost datastore (HP MSA P2000 G3)

[DTLN]

Hello all, I would like to discuss a problem that I encountered.

Configuration:

2x HP Proliant DL360 Gen8 / 112GB RAM / HBA HP H221 (Cluster)

1x HP MSA P2000 G3

ESXi version 5.1 Update2 HP Customized

Vcenter 5.1 Update2 (virtual appliance)

Everything supported firmware / bios versions up to date.

Both servers have 2paths to datastore. Only one LUN is provisioned to esx servers.

After istalling esxi on both servers I have deployed vcenter appliance, builded ha cluster and deployed some VMs everithing works perfectly.

After cca 1 week one server lost both paths to datastore (MSA). After restart host everything works like before few days and problem occures again.

I noticed, that failed host (lost datastore) is allways one on witch vcenter appliance is running. The worst thing is, ha agent not restarts vms on another host untill I physically turn that host off.

Workaround was, I reinstall vcenter on third host that is not part of the cluster, only member of datacenter and has only local storage (not msa).

Are you kidding me vmware ?

[MKguy]

I noticed, that failed host (lost datastore) is allways one on witch vcenter appliance is running.

Workaround was, I reinstall vcenter on third host that is not part of the cluster, only member of datacenter and has only local storage (not msa).

That is obviously unrelated from the storage issue and just a coincidence at best. HA works independent from vCenter, even if vCenter is down. You only need an available vCenter server to make changes to the HA configuration.

The worst thing is, ha agent not restarts vms on another host untill I physically turn that host off.

Well, that's exactly what HA is supposed to do: Restart VMs in case of a host failure. HA does not protect against and was never supposed to handle storage-side failures.

You need to dig down to the cause of the storage issues instead of blaming the wrong effect.

[DTLN]

No thats not a coincidence, now if I deploy new vcenter appliance on cluster and store vmdk on storage and run on esx1 for example, this host esx1 will lost his storage within one week or less. Trust me its absolutly crazy. If i turn on vcenter on esx2 the same thing happens on esx2. Event logs on MSA tell nothing interesting, just something like "Topology changes"

[DavidSancho]

We are having the same issue.

[DavidSancho]

Do you have v2 cpus and 1600 memory? if yes, try to update bios server to last one, it solves problem 

[jarekpogan]

DavidSancho To which version of Bios you updated ? I have exacly same problem but with MSA2040... and it drives me crazy we replaced every vital part of the server affected (mainboard, h221 card, and cables) and nothing helps...

[DavidSancho]

This one, http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=5226971&sp...

Regards,

[jarekpogan]

Thanks

which version of vmware image are you using ? Hp customized or vannilla one ?

Regards

[DavidSancho]

Customized, VMware-ESXi-5.5.0-1331820-HP-5.72.27-Feb2014.iso .

Regards,

[JRink]

I have a problem with my VMware ESXi 5.1 host servers losing access to the LUNs on my MSA 2040 dual controller-SAS too.  The HBA is a H221.   Can anyone tell me a solution here?   We have v2 CPUs used.    HP can't even tell me what is the proper SAS cable to use between the H221 and the MSA 2040.  What are others using for the cable (Part #??).   Anyone FIX this issue??  I'm dealing with support right now from HP, but they're clueless as to what's going on.  LUNs are constantly disappearing until a reboot the host, then it's fine for a few hours before losing the LUNs again.

[jarekpogan]

For dl380gen8 helped movng sas card to another pcie slot... For 360 on esxi 5.5 nothing helped even replacing mb and sas card, we downgraded to 5.1 and updated bios, card bios, and reinstalled esxi from hp 2014 february version.

The only stable config for me is HP customized esxi 5.1 from february 2014. and newest firmwares from hp site. and according to msa2040 compatibility chart support for esxi 5.5 is still pending...

[dkflov]

I found this.

Old environment HP BL460c G6, in enclosure with Virtual connect, ESXi 5.1 installed, everything works fine regarding connections to P2000 G3

New environment. HP BL460c Gen8 v2, in enclosure with Cisco FEX and brocade san switch. ESXi 5.1 installed, connections to P2000 goes on and off. I have 8 LUN's attached the hosts and 7 of those goes on and off (vmfs 5.58). 1 LUN with older vmfs i stable. The attached LUN's stays online if I do this. In vcenter, on the host, configuration, advanced settings, vmfs 3, Set "VMFS3.hardwareacceleratedlocking to 0.

I then tried to install ESXi 5.5. In this version I cannot see P2000 LUN's at all, until hardwareacceleratedlocking is disabled, then connections goes online.

Having P2000, P6500 and 3Par in our environment, I'm not sure yet whether this setting is relevant for us.

NB. we have an other retired P2000 G3, which I tried to firmware upgrade to latest level (new firmware came in march 2014).

After this firmware upgrade I can attach LUN's to ESXi 5.5 host, without disabling hardwareacceleratedlocking feature.

[JRink]

I found a fix for my issue with the LUNs dropping off the ESXi 5.1 hosts with the H221 cards and the MSA 2040 SAS.  Turned out the DRIVER for the H221 card (not firmware) needed to be updated in VMware 5.1.  Once running the driver update, everything has been great.

[ygsun]

I ran into the same issue yesterday.

HP DL380 Gen 8

ESXi 5.1 (HP Image Apr 2013, VAAI  included)

HP P2000 G3 (Firmware TS240)

After upgrading the host to ESXi 5.5.0 U1 (HP Image Mar 2014), host can't see existing VMFS 5.58 datastores on P2000. Googling around, I found the following:

The workaround is to install the HP P2000 Software Plug-in for VMware VAAI. https://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?spf_p.tpst=swdMain&... The solution is to do a firmware update on the P2000 G3. HP released TS251R004 with native VAAI support (T10-compliance for VAAI).  The  VAAI plug-in is no longer required. I just flashed mine to TS251P002. Host sees everything after reboot. Hope it helps.

[dkflov]

just word today from HP

Error should be corrected in bundle firmware version TS251R004. have not tested yet.

[ygsun]

I flashed from TS240P003 to TRS251P002 (May 2014), hosts see VMFS 5.58 datastores after reboot.

This should be a final fix, more preferable than installing VAAI plug-in (every time in the future upgrades???).

HP Release notes: Beginning with TS251R004, HP MSA VAAI plug-in is no longer required. The HP MSA controller firmware now use T10 compliance in an ESX Environment.

[snapfast]

We're having the same issue on MSA2040. Updated both controllers to latest firmware, updated disks to latest firmware, and still happens. Randomly the host will loose visibility to the MSA2040 controllers and we have to manually rescan all HBA from vmware or sometimes restarting the virtual connect does the trick.

Seems like this is a documented and resolved issue for the P2000. Anyone know of a solution for the MSA2040?

5.5u1 (hp image)

BL460c G6

Virtual connect Flex10

MSA2040 dual SAN controller - iscsi

[dkflov]

Jeg er på ferie og forventer at være tilbage d. 20. oktober

I'm on holiday. I will be available on october 20.

Hilsen/BR

Ole Vig

[Avon]

I had the same issue with the HP Build of 5.1 U1 & U2 with a Gen 8 DL360 & P2000 G3

After HP had replaced the mainboard & both the H221 HBAs it was traced to a driver fault (Applies to 5.0 as well)

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=5177954&sp...

[jarekpogan]

W dniu dzisiejszym przebywam poza firmą z utrudnnionym dostępem do poczty zastępuje mnie pan Maciej Miś m.mis@infomex.pl

Wiadomość NIE została przekierowana.

--

Jarek Pogan

Koordynator zespołu wdrożeniowego

INFOMEX Sp. z o.o.

ul. Wesola 19b

34-300 Zywiec

Tel.: (+48 33) 484 32 20

Fax: (+48 33) 475 11 06

www.infomex.pl

j.pogan@infomex.pl