Adding recovered esxi host to "datastores" and "heartbeating"

[nieca]

Dear all,

I'm new here and I greet you all.

There was a crash of motherboard of one of the blade at my VMWare infasturcture.

The blade has been repaired and VMWare host is back - esxi03.

There are 3 esxi hosts and several virtual machines.

Now when I try to migrate one of virtual machine from esxi02 to esxi03 (newly added host) I get error messages:

esxi03 also shows me:

"the number of vSphere HA heartbeat datastores for this host is 0, which is less than required: 2"

When I go to "cluster status" esxi03 is missing under "datastores":

How to add esxi03 host back to "datastores" and "heartbeating"?

Regards

GN

[aravinds3107]

Welcome to the Communities..

Now when I try to migrate one of virtual machine from esxi02 to esxi03 (newly added host) I get error messages

Is the LUN and Datastore which are visible on ESXi 01 and ESXi02 and shared to your third host ESXi 03?

What type of storage ar eyou using? FC,iSCSI, NFS

[nieca]

How can I check this/get this infos?

From vSphere client:

[aravinds3107]

Select your host on left pane , On right pane goto Configuration Tab -> Select Storage Adapters and check if you can see LUN ( If you are using FC or iSCSI )

OR

Select Host - > Configuration -> Storage (Check if the datastore which are visile on ESXi 01 and 02 )

For the VM04 screenshot look like the third host is missing connectivity to the datastore

Follow this article to fx the issue, the article is based on webclient but you should be able to follow and fix it through vSphere Client

http://frankdenneman.nl/2012/10/15/partially-connected-datastore-clusters-where-can-i-find-the-warni...

[nieca]

esxi01 and esxi02:

esxi03:

[nieca]

esxi02:

esxi03:

[Gkeerthy]

i think you are not seeing the other luns in the esx03, can you rescan the storage for the host in      question.

Also check the iscsi connctivity, and any pnics issues.. and the iscsi configuration.

refer the below

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100398...

http://www.youtube.com/watch?v=FzXYUzYTJVE

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&e...

[aravinds3107]

Looking at both ur screenshot LUN's are visible to he third host.. I assume you might be using Hardware based iSCSI and check the setting on other 2 host and check n ur third host

[lenzker]

Have you masked your esxi luns correctly to the new esx03 host?

you should do this on the storage level. Which storage are you using?

[nieca]

Still no luck.

I have go through this article:

http://frankdenneman.nl/2012/10/15/partially-connected-datastore-clusters-where-can-i-find-the-warni...

My situation is:

And I'm trying to add storage to host:

what should I do next then?

Regards

GN

[vGuy]

You do not need to go through "Add Datastore" wizard on every host. Just ensure all the LUNs are visible on the problem host, the Datastore will be added automatically.

[nieca]

How can I ensure that this LUNS are visible for esxi03 host?

Where to check/configure this?

I just want to metioned that before esxi03 motherboard crash everything has worked fine.

After repair I have to add this datastores somehow again...

Regards

GN

[vGuy]

Can you try to rescan the vmhba on the host3..better yet if you can reboot the host once and see if the LUNs are visible. If the LUNs are still not visible plz post the vmkernel logs from the host (/var/log/vmkernel). It would also help if you verify the Storage array configuration is same for all the hosts and nothing has changed. 

[nieca]

It would also help if you verify the Storage array configuration is same for all the hosts

how can I verify/check that?

Rescanning with no luck.

I have rebooted this esxi host several times - no luck.

I have also change this host to "maintenance mode" and restrated it - no luck there - still no LUNs visible.

Regards

GN

[lenzker]

Which storage are you using?

[aravinds3107]

It would also help if you verify the Storage array configuration is same for all the hosts

how can I verify/check that?

Check with your storage team if you have seperate team..

The issue currently you are facing is the LUN's are not visible to your ESXi host since you have already rebooted your ESXi host. Check from storage for any zoning/masking issue

[vGuy]

Can you attach the vmkernel logs from the host (/var/log/vmkernel)?

[nieca]

2013-02-11T08:13:14.302Z cpu20:8212)ScsiDeviceIO: 2316: Cmd(0x4124410d1940) 0x12, CmdSN 0x607b from world 0 to dev "naa.600508e000000000961ff8b8f4f2a607" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-02-11T08:13:14.308Z cpu20:8212)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124410d1940, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-02-11T08:13:14.308Z cpu20:8212)ScsiDeviceIO: 2316: Cmd(0x4124410d1940) 0x1a, CmdSN 0x607d from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2013-02-11T08:15:17.506Z cpu18:9196)FSS: 4972: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2013-02-11T08:15:17.521Z cpu18:9196)Vol3: 692: Couldn't read volume header from control: Not supported
2013-02-11T08:15:17.521Z cpu18:9196)Vol3: 692: Couldn't read volume header from control: Not supported
2013-02-11T08:15:17.521Z cpu18:9196)FSS: 4972: No FS driver claimed device 'control': Not supported
2013-02-11T08:15:17.573Z cpu18:9196)VC: 1547: Device rescan time 46 msec (total number of devices 7)
2013-02-11T08:15:17.573Z cpu18:9196)VC: 1550: Filesystem probe time 90 msec (devices probed 6 of 7)
2013-02-11T08:15:17.593Z cpu12:426492)usb-storage: detected SCSI revision number 0 on vmhba32
2013-02-11T08:15:17.593Z cpu12:426492)usb-storage: patching inquiry data to change SCSI revision number from 0 to 2 on vmhba32
2013-02-11T08:15:17.593Z cpu8:8378)ScsiUid: 273: Path 'vmhba32:C0:T0:L0' does not support VPD Device Id page.
2013-02-11T08:15:17.593Z cpu8:8378)VMWARE SCSI Id: Could not get disk id for vmhba32:C0:T0:L0
2013-02-11T08:15:17.621Z cpu18:9196)Vol3: 692: Couldn't read volume header from control: Not supported
2013-02-11T08:15:17.621Z cpu18:9196)Vol3: 692: Couldn't read volume header from control: Not supported
2013-02-11T08:15:17.621Z cpu18:9196)FSS: 4972: No FS driver claimed device 'control': Not supported
2013-02-11T08:15:17.654Z cpu18:9196)FSS: 4972: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2013-02-11T08:15:17.669Z cpu18:9196)VC: 1547: Device rescan time 13 msec (total number of devices 7)
2013-02-11T08:15:17.669Z cpu18:9196)VC: 1550: Filesystem probe time 59 msec (devices probed 6 of 7)
2013-02-11T08:15:20.159Z cpu6:10725)FSS: 4972: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2013-02-11T08:15:20.159Z cpu6:10725)Vol3: 692: Couldn't read volume header from control: Not supported
2013-02-11T08:15:20.159Z cpu6:10725)Vol3: 692: Couldn't read volume header from control: Not supported
2013-02-11T08:15:20.159Z cpu6:10725)FSS: 4972: No FS driver claimed device 'control': Not supported
2013-02-11T08:15:20.211Z cpu0:10725)VC: 1547: Device rescan time 24 msec (total number of devices 7)
2013-02-11T08:15:20.211Z cpu0:10725)VC: 1550: Filesystem probe time 75 msec (devices probed 6 of 7)

[para2ox]

having the same problem by ESXi 5.5

but ESXi 5.0 work fine.