Datastore problem vmware esxi not responding

[straioto]

I have installed VMware ESXi 4.1 in dell 410 with 8GB of ram and two 500GB disks in a raid 1, we have two machines running, a windows 2008server and a Debian Linux Firewall, but it happens that suddenly the datastore is no longer accessible, and the commands of vi client does not respond, I can not restart the server with no command, someone has any idea?

[Virtualinfra]

Datastore got disconnected, there are lot many problems to this because its NFS.. network disconnection would be the primary impact and also storage issue to..

OK for you query how to reboot the server,

you have option to enable SSH in ESXi 4.1. connecting the esxi server via vsphere client by right click the server you can reboot it or by starting the ssh services you will be able to take a putty to the ESXi host and you can reboot there.

If both are not woring you can reboot by taking DRAC connection to the server.

If your problem You can not restart the server refers to virtual machine then.

YOu wont be able to restart the virtual machine till the datastore issue is resolved.. because all your VM files are in datastore. if datastore disconnect.. you cant have access to virtual machines.So first resolve the datasotre issue and now you will be able to restart the virtual machine..

Even after you resolve the datastore issue you and not able to restart.. unregister the VM and register the VM again to the host will resolve the issue.

[gafain]

Hello,

I have the same problem, I have a dell R210 with SAS iR6 RAID controller with two SATA 500GB in RAID 1.

I have the same problem about every 10-14 days.

Did you solve problem ?

Thanks

[straioto]

My problem  continue, today it happened again,

My datastore 2 stop of working, and vmware esx 4.1 with all updates not reponding:

Look this log:

Feb  2 08:55:09 vmware vmkernel: 6:00:17:58.110 cpu9:4246)<6>mptscsih: ioc0: attempting task abort! (sc=0x4100b0043cc0)

Feb  2 08:55:09 vmware vmkernel: 6:00:17:58.110 cpu9:4246)MPT SAS Host:7:1:2:0 :: 

Feb  2 08:55:09 vmware vmkernel: <6>        command: Test Unit Ready: 00 00 00 00 00 00

Feb  2 08:55:09 vmware vmkernel: 6:00:17:58.110 cpu9:4246)<6>mptscsih: ioc0: task abort: FAILED (sc=0x4100b0043cc0)

Feb  2 08:55:09 vmware vmkernel: 6:00:17:58.110 cpu9:4246)WARNING: LinScsi: SCSILinuxAbortCommands: Failed, Driver MPT SAS Host, for vmhba0

Feb  2 08:55:09 vmware vmkernel: 6:00:17:58.110 cpu9:4246)WARNING: ScsiPath: 5209: Set retry timeout for failed TaskMgmt abort for CmdSN  0x0, status Failure, path vmhba0:C1:T2:L0

Feb  2 08:55:09 vmware vmkernel: 6:00:17:58.158 cpu4:4126)ScsiDeviceIO: 3270: Waited for completion for all issued commands for partition naa.600508e000000000e77e4e54f1ddd004:1, for 120 secs. 32 completions still awaited. Giving up !

Feb  2 08:55:09 vmware vmkernel: 6:00:17:58.158 cpu4:4126)FS3: 7068: Starting HB reclaim for [HB state abcdef02 offset 3637248 gen 319 stamp 503064612172 uuid 4f227e92-7ec01cf3-c2fe-842b2b5257ca jrnl <FB 719200> drv 8.46]

Feb  2 08:55:10 vmware vmkernel: 6:00:17:58.557 cpu9:4246)<6>mptscsih: ioc0: attempting task abort! (sc=0x4100b0036cc0)

Feb  2 08:55:10 vmware vmkernel: 6:00:17:58.557 cpu9:4246)MPT SAS Host:7:1:2:0 :: 

Feb  2 08:55:10 vmware vmkernel: <6>        command: Report luns: a0 00 00 00 00 00 00 00 08 08 00 00

Feb  2 08:55:10 vmware vmkernel: 6:00:17:58.557 cpu9:4246)<6>mptscsih: ioc0: task abort: FAILED (sc=0x4100b0036cc0)

Feb  2 08:55:10 vmware vmkernel: 6:00:17:58.557 cpu9:4246)WARNING: LinScsi: SCSILinuxAbortCommands: Failed, Driver MPT SAS Host, for vmhba0

Feb  2 08:55:10 vmware vmkernel: 6:00:17:58.557 cpu9:4246)WARNING: ScsiPath: 5209: Set retry timeout for failed TaskMgmt abort for CmdSN  0x0, status Failure, path vmhba0:C1:T2:L0

F

Feb  2 08:55:39 vmware vmkernel: 6:00:18:28.125 cpu11:4246)<6>mptscsih: ioc0: attempting task abort! (sc=0x4100b0043cc0)

Feb  2 08:55:39 vmware vmkernel: 6:00:18:28.125 cpu11:4246)MPT SAS Host:7:1:2:0 :: 

Feb  2 08:55:39 vmware vmkernel: <6>        command: Test Unit Ready: 00 00 00 00 00 00

Feb  2 08:55:39 vmware vmkernel: 6:00:18:28.125 cpu11:4246)<6>mptscsih: ioc0: task abort: FAILED (sc=0x4100b0043cc0)

Feb  2 08:55:39 vmware vmkernel: 6:00:18:28.125 cpu11:4246)WARNING: LinScsi: SCSILinuxAbortCommands: Failed, Driver MPT SAS Host, for vmhba0

Feb  2 08:55:39 vmware vmkernel: 6:00:18:28.125 cpu11:4246)WARNING: ScsiPath: 5209: Set retry timeout for failed TaskMgmt abort for CmdSN  0x0, status Failure, path vmhba0:C1:T2:L0

Feb  2 08:55:40 vmware vmkernel: 6:00:18:28.571 cpu11:4246)<6>mptscsih: ioc0: attempting task abort! (sc=0x4100b0036cc0)

Feb  2 08:55:40 vmware vmkernel: 6:00:18:28.571 cpu11:4246)MPT SAS Host:7:1:2:0 :: 

Feb  2 08:55:40 vmware vmkernel: <6>        command: Report luns: a0 00 00 00 00 00 00 00 08 08 00 00

Feb  2 08:55:40 vmware vmkernel: 6:00:18:28.571 cpu11:4246)<6>mptscsih: ioc0: task abort: FAILED (sc=0x4100b0036cc0)

Feb  2 08:55:40 vmware vmkernel: 6:00:18:28.571 cpu11:4246)WARNING: LinScsi: SCSILinuxAbortCommands: Failed, Driver MPT SAS Host, for vmhba0

Feb  2 08:55:40 vmware vmkernel: 6:00:18:28.571 cpu11:4246)WARNING: ScsiPath: 5209: Set retry timeout for failed TaskMgmt abort for CmdSN  0x0, status Failure, path vmhba0:C1:T2:L0

Feb  2 08:55:41 vmware vmkernel: 6:00:18:30.127 cpu12:4246)<6>mptscsih: ioc0: attempting task abort! (sc=0x4100b0043cc0)

Feb  2 08:55:41 vmware vmkernel: 6:00:18:30.127 cpu12:4246)MPT SAS Host:7:1:2:0 :: 

Feb  2 08:55:41 vmware vmkernel: <6>        command: Test Unit Ready: 00 00 00 00 00 00

Feb  2 08:55:41 vmware vmkernel: 6:00:18:30.127 cpu12:4246)<6>mptscsih: ioc0: task abort: FAILED (sc=0x4100b0043cc0)

Feb  2 08:55:41 vmware vmkernel: 6:00:18:30.127 cpu12:4246)WARNING: LinScsi: SCSILinuxAbortCommands: Failed, Driver MPT SAS Host, for vmhba0

Feb  2 08:55:41 vmware vmkernel: 6:00:18:30.127 cpu12:4246)WARNING: ScsiPath: 5209: Set retry timeout for failed TaskMgmt abort for CmdSN  0x0, status Failure, path vmhba0:C1:T2:L0

Feb  2 08:55:42 vmware vmkernel: 6:00:18:30.573 cpu8:4246)<6>mptscsih: ioc0: attempting task abort! (sc=0x4100b0036cc0)

Feb  2 08:55:42 vmware vmkernel: 6:00:18:30.573 cpu8:4246)MPT SAS Host:7:1:2:0 :: 

Feb  2 08:55:42 vmware vmkernel: <6>        command: Report luns: a0 00 00 00 00 00 00 00 08 08 00 00

Feb  2 08:55:42 vmware vmkernel: 6:00:18:30.573 cpu8:4246)<6>mptscsih: ioc0: task abort: FAILED (sc=0x4100b0036cc0)

Feb  2 08:55:42 vmware vmkernel: 6:00:18:30.573 cpu8:4246)WARNING: LinScsi: SCSILinuxAbortCommands: Failed, Driver MPT SAS Host, for vmhba0

Feb  2 08:55:42 vmware vmkernel: 6:00:18:30.573 cpu8:4246)WARNING: ScsiPath: 5209: Set retry timeout for failed TaskMgmt abort for CmdSN  0x0, status Failure, path vmhba0:C1:T2:L0

F

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)<6>mptscsih: ioc0: attempting task abort! (sc=0x4100b002fec0)

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)MPT SAS Host:7:1:2:0 :: 

Feb  2 08:56:10 vmware vmkernel: <6>        command: Write(10): 2a 00 6b 06 75 06 00 00 20 00

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)<6>mptscsih: ioc0: task abort: FAILED (sc=0x4100b002fec0)

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)WARNING: LinScsi: SCSILinuxAbortCommands: Failed, Driver MPT SAS Host, for vmhba0

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)<6>mptscsih: ioc0: attempting task abort! (sc=0x4100b00252c0)

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)MPT SAS Host:7:1:2:0 :: 

Feb  2 08:56:10 vmware vmkernel: <6>        command: Write(10): 2a 00 0e c6 e6 b1 00 00 60 00

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)<6>mptscsih: ioc0: task abort: FAILED (sc=0x4100b00252c0)

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)WARNING: LinScsi: SCSILinuxAbortCommands: Failed, Driver MPT SAS Host, for vmhba0

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)<6>mptscsih: ioc0: attempting task abort! (sc=0x4100b00362c0)

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)MPT SAS Host:7:1:2:0 :: 

Feb  2 08:56:10 vmware vmkernel: <6>        command: Write(10): 2a 00 6b 06 75 26 00 00 38 00

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)<6>mptscsih: ioc0: task abort: FAILED (sc=0x4100b00362c0)

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)WARNING: LinScsi: SCSILinuxAbortCommands: Failed, Driver MPT SAS Host, for vmhba0

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)<6>mptscsih: ioc0: attempting task abort! (sc=0x4100b001dcc0)

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)MPT SAS Host:7:1:2:0 :: 

Feb  2 08:56:10 vmware vmkernel: <6>        command: Write(10): 2a 00 0e c6 e7 11 00 00 48 00

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)<6>mptscsih: ioc0: task abort: FAILED (sc=0x4100b001dcc0)

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)WARNING: LinScsi: SCSILinuxAbortCommands: Failed, Driver MPT SAS Host, for vmhba0

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)<6>mptscsih: ioc0: attempting task abort! (sc=0x4100b00188c0)

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)MPT SAS Host:7:1:2:0 :: 

Feb  2 08:56:10 vmware vmkernel: <6>        command: Write(10): 2a 00 6b 06 75 5e 00 00 18 00

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)<6>mptscsih: ioc0: task abort: FAILED (sc=0x4100b00188c0)

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)WARNING: LinScsi: SCSILinuxAbortCommands: Failed, Driver MPT SAS Host, for vmhba0

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)<6>mptscsih: ioc0: attempting task abort! (sc=0x4100b000eec0)

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)MPT SAS Host:7:1:2:0 :: 

Feb  2 08:56:10 vmware vmkernel: <6>        command: Write(10): 2a 00 0e c6 e7 59 00 00 68 00

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)<6>mptscsih: ioc0: task abort: FAILED (sc=0x4100b000eec0)

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.372 cpu14:4127)WARNING: LinScsi: SCSILinuxAbortCommands: Failed, Driver MPT SAS Host, for vmhba0

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.588 cpu8:4246)<6>mptscsih: ioc0: attempting task abort! (sc=0x4100b0036cc0)

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.588 cpu8:4246)MPT SAS Host:7:1:2:0 :: 

Feb  2 08:56:10 vmware vmkernel: <6>        command: Report luns: a0 00 00 00 00 00 00 00 08 08 00 00

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.588 cpu8:4246)<6>mptscsih: ioc0: task abort: FAILED (sc=0x4100b0036cc0)

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.588 cpu8:4246)WARNING: LinScsi: SCSILinuxAbortCommands: Failed, Driver MPT SAS Host, for vmhba0

Feb  2 08:56:10 vmware vmkernel: 6:00:18:58.588 cpu8:4246)WARNING: ScsiPath: 5209: Set retry timeout for failed TaskMgmt abort for CmdSN  0x0, status Failure, path vmhba0:C1:T2:L0

Feb  2 08:56:11 vmware vmkernel: 6:00:19:00.143 cpu10:4246)<6>mptscsih: ioc0: attempting task abort! (sc=0x4100b0043cc0)

Feb  2 08:56:11 vmware vmkernel: 6:00:19:00.143 cpu10:4246)MPT SAS Host:7:1:2:0 :: 

Feb  2 08:56:11 vmware vmkernel: <6>        command: Test Unit Ready: 00 00 00 00 00 00

Feb  2 08:56:11 vmware vmkernel: 6:00:19:00.143 cpu10:4246)<6>mptscsih: ioc0: task abort: FAILED (sc=0x4100b0043cc0)

Feb  2 08:56:11 vmware vmkernel: 6:00:19:00.143 cpu10:4246)WARNING: LinScsi: SCSILinuxAbortCommands: Failed, Driver MPT SAS Host, for vmhba0

Feb  2 08:56:11 vmware vmkernel: 6:00:19:00.143 cpu10:4246)WARNING: ScsiPath: 5209: Set retry timeout for failed TaskMgmt abort for CmdSN  0x0, status Failure, path vmhba0:C1:T2:L0

F

[gafain]

Hello,

I think is incompatible with vmware.

I replace all two harddisk 2 month ago.

For first 20 day all works fine, but after I have a block every week.

Now I have a block every 2 days.

I chose to change all the server with NO DELL hardware!

[Shakaal]

Hi,

from the log's mentioned by you it seems to be an issue with Storage, check the lights on the Disk drives on the ESX host, seems to be an issue with Storage hardware. All the commands are just pilingup/queueing up.

"

eb  2 08:55:09 vmware vmkernel: <6>        command: Test Unit Ready: 00 00 00 00 00 00

Feb  2 08:55:09 vmware vmkernel: 6:00:17:58.110 cpu9:4246)<6>mptscsih: ioc0: task abort: FAILED (sc=0x4100b0043cc0)

Feb  2 08:55:09 vmware vmkernel: 6:00:17:58.110 cpu9:4246)WARNING: LinScsi: SCSILinuxAbortCommands: Failed, Driver MPT SAS Host, for vmhba0

Feb  2 08:55:09 vmware vmkernel: 6:00:17:58.110 cpu9:4246)WARNING: ScsiPath: 5209: Set retry timeout for failed TaskMgmt abort for CmdSN  0x0, status Failure, path vmhba0:C1:T2:L0

Feb  2 08:55:09 vmware vmkernel: 6:00:17:58.158 cpu4:4126)ScsiDeviceIO: 3270: Waited for completion for all issued commands for partition naa.600508e000000000e77e4e54f1ddd004:1, for 120 secs. 32 completions still awaited. Giving up !"

If it is a SAN based datastore then there is an issue with SAN Storage device. Check with Storage Vendor in that case.

Incase of Local Datastore check the lights on the disks also check with server vendor.

Regards

[straioto]

Dear Gafain , we have the same problem, however my others servers DELL doesn't show this problem.

This Server the configuration is:

R410 - 32 GB RAM - 2 Processors -

2 Disks 500GB RAID 1 - DATASTORE 1

2 Disks 1 TB - RAID 1 - DATASTORE 2

= 2 datastores in my vmware esx 4.1

but only the datastore 2 of 1TB stop work, the other datastore 1 whit 500GB continue working without problems....

The Vm's in datastore 1 continue working normally, but vm's of datastore 1 working....

What your configuration of your DELL ?

I think with this problem is the controller of disks.

a hug.

[straioto]

My server has local disks, i will call to Dell...

tks....

[gafain]

R210 with SAS iR6 RAID controller with two SATA 500GB in RAID 1.

I have the server in a webfarm and the provider say to me that there are no hardware problems.

Ha i write I replace the hard disk and for some time it works fine.

Now I have orderd a new server NON DELL and I transfer the VM.

[straioto]

I understood,

what the number VM'S in your server ?

In my other server the same model R410 is normal, but the disk are SAS with RAID 10.

Strange problem...

[gafain]

Hello,

At the begin I have 6 VM.

Now (I already start the migration) I have only 2 active but today I have another block.

I think is related to the controller, wich controller you have )

If you solve with DELL please tell me.

Regards

[straioto]

My controller is DELL SAS 6/IR Integrated

[straioto]

I found a problem, my servers with controller integrated are stopping and other server with off board controller are OK.

On board SAS 6ir are whith problem.

and my other server with the same controller but (PCI-E) off board is OK.

[gafain]

Oh intersting .

You find the problem or DELL tell to you ?

You contact DELL to solve the problem ?

Regards

[straioto]

Yes , but DELL does not want to change de Device.............. i have the same Server in other client and same controller, but HD SAS and the problem not happened.

Any news I tell you.

[AGS_JP]

Hi, same issue for me on an IBM x3550 M3, ESXi 4.1U2, IBM 6Gb SAS PCI HBA attached to a IBM DS3500...

I'm looking for help...

Thanks.

[dherbots]

Did anyone solve this issue?

I have the server 10000km from here so I hope there is a bug fix or something?

First it was every 2 weeks --> 1 week --> and now every few days.

Thanks

[gafain]

Hello,

I don't solve, I change provider and server.