ESXI 5.0 events (Lost access to volume?) on HP Proliant DL380p Gen8 with Smart Array 420i

[h202b3]

Hi All.

Anyone seen this before? We noticed some services stopping and restarting on the guest machines at 60 minute intervals. We found this events in the host vmkernel.log comes up every hour?

2017-06-12T05:00:29.632Z cpu8:3394587)<4>hpsa 0000:02:00.0: Abort request on C2:B0:T0:L2

2017-06-12T05:00:31.678Z cpu15:3397938)ScsiCore: 63: Starting taskmgmt handler world 3397938/2

2017-06-12T05:00:31.678Z cpu15:3397938)<4>hpsa 0000:02:00.0: Abort request on C2:B0:T0:L3

2017-06-12T05:00:33.680Z cpu15:3397939)ScsiCore: 63: Starting taskmgmt handler world 3397939/3

2017-06-12T05:00:33.681Z cpu15:3397939)<4>hpsa 0000:02:00.0: Abort request on C2:B0:T0:L8

2017-06-12T05:00:35.682Z cpu14:3397940)ScsiCore: 63: Starting taskmgmt handler world 3397940/4

2017-06-12T05:00:35.683Z cpu14:3397940)<4>hpsa 0000:02:00.0: Abort request on C2:B0:T0:L5

2017-06-12T05:00:37.686Z cpu16:3397944)ScsiCore: 63: Starting taskmgmt handler world 3397944/5

2017-06-12T05:00:37.686Z cpu16:3397944)<4>hpsa 0000:02:00.0: Abort request on C2:B0:T0:L7

2017-06-12T05:00:39.688Z cpu3:3397951)ScsiCore: 63: Starting taskmgmt handler world 3397951/6

2017-06-12T05:00:39.689Z cpu3:3397951)<4>hpsa 0000:02:00.0: Abort request on C2:B0:T0:L6

2017-06-12T05:00:41.690Z cpu0:3397952)ScsiCore: 63: Starting taskmgmt handler world 3397952/7

2017-06-12T05:00:41.690Z cpu15:3397952)<4>hpsa 0000:02:00.0: Abort request on C2:B0:T0:L3

2017-06-12T05:00:43.691Z cpu11:3397953)ScsiCore: 63: Starting taskmgmt handler world 3397953/8

2017-06-12T05:00:43.692Z cpu11:3397953)<4>hpsa 0000:02:00.0: Abort request on C2:B0:T0:L8

2017-06-12T05:00:45.693Z cpu3:3397956)ScsiCore: 63: Starting taskmgmt handler world 3397956/9

2017-06-12T05:00:45.693Z cpu3:3397956)<4>hpsa 0000:02:00.0: Abort request on C2:B0:T0:L8

2017-06-12T05:00:47.696Z cpu2:3397957)ScsiCore: 63: Starting taskmgmt handler world 3397957/10

2017-06-12T05:00:47.696Z cpu2:3397957)<4>hpsa 0000:02:00.0: Abort request on C2:B0:T0:L8

2017-06-12T05:00:49.699Z cpu16:3397958)ScsiCore: 63: Starting taskmgmt handler world 3397958/11

2017-06-12T05:00:49.699Z cpu12:3397958)<4>hpsa 0000:02:00.0: Abort request on C2:B0:T0:L1

2017-06-12T05:00:51.700Z cpu8:3397965)ScsiCore: 63: Starting taskmgmt handler world 3397965/12

2017-06-12T05:00:51.701Z cpu8:3397965)<4>hpsa 0000:02:00.0: Abort request on C2:B0:T0:L4

2017-06-12T05:00:58.094Z cpu7:4806)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3858432 gen 271259 stampUS 3726634025478 uuid 5905429a-463f63b5-xxxxxx-ac162d979050 jrnl <FB 13000> drv 14.54] on vol 'xxxxP1-DS07'

2017-06-12T05:00:58.099Z cpu1:4930)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3858432 gen 271259 stampUS 3726634025478 uuid 5905429a-463f63b5-xxxxxx-ac162d979050 jrnl <FB 13000> drv 14.54] on vol 'xxxxxP1-DS06'

2017-06-12T05:00:58.779Z cpu12:3397965)<3>hpsa 0000:02:00.0: ABORT REQUEST on C2:B0:T0:L4 Tag:0x00000000:00000230 Command:0x2a SN:0x14bbd1cc  (via RESET) FAILED, Command completed before reset was attempted after delaying 7 seconds.

2017-06-12T05:00:58.780Z cpu10:4133)HBX: 231: Reclaimed heartbeat for volume 51ff2100-af0dcb26-xxxxxx-d89d671a946d (xxxxP1-DS04): [Timeout] [HB state abcdef02 offset 3858432 gen 329 stampUS 3726673172798 uuid 5905429a-463f63b5-xxxxxx-ac162d979050 jrnl <F$

2017-06-12T05:00:58.803Z cpu9:3397958)<3>hpsa 0000:02:00.0: ABORT REQUEST on C2:B0:T0:L1 Tag:0x00000000:000002b0 Command:0x2a SN:0x14bbd1d0  (via RESET) FAILED, Command completed before reset was attempted after delaying 9 seconds.

2017-06-12T05:00:58.803Z cpu16:4132)HBX: 231: Reclaimed heartbeat for volume 51fef219-bbc70d96-xxxxxx-d89d671a946d (xxxxxx-DS01): [Timeout] [HB state abcdef02 offset 3858432 gen 231 stampUS 3726673196381 uuid 5905429a-463f63b5-xxxxxx-ac162d979050 jrnl <F$

2017-06-12T05:00:58.822Z cpu12:3397957)<3>hpsa 0000:02:00.0: ABORT REQUEST on C2:B0:T0:L8 Tag:0x00000000:000007f0 Command:0x2a SN:0x14bbd1fa  (via RESET) FAILED, Command completed before reset was attempted after delaying 11 seconds.

2017-06-12T05:00:58.843Z cpu9:3397956)<3>hpsa 0000:02:00.0: ABORT REQUEST on C2:B0:T0:L8 Tag:0x00000000:000007d0 Command:0x2a SN:0x14bbd1f9  (via RESET) FAILED, Command completed before reset was attempted after delaying 13 seconds.

2017-06-12T05:00:58.863Z cpu20:3397953)<3>hpsa 0000:02:00.0: ABORT REQUEST on C2:B0:T0:L8 Tag:0x00000000:00000730 Command:0x2a SN:0x14bbd1f4  (via RESET) FAILED, Command completed before reset was attempted after delaying 15 seconds.

2017-06-12T05:00:58.885Z cpu20:3397952)<3>hpsa 0000:02:00.0: ABORT REQUEST on C2:B0:T0:L3 Tag:0x00000000:00000530 Command:0x2a SN:0x14bbd1e4  (via RESET) FAILED, Command completed before reset was attempted after delaying 17 seconds.

2017-06-12T05:00:58.907Z cpu19:3397951)<3>hpsa 0000:02:00.0: ABORT REQUEST on C2:B0:T0:L6 Tag:0x00000000:00000210 Command:0x2a SN:0x14bbd1cb  (via RESET) FAILED, Command completed before reset was attempted after delaying 19 seconds.

2017-06-12T05:00:58.907Z cpu15:4131)HBX: 231: Reclaimed heartbeat for volume 54335da8-51810d4f-xxxxxx-ac162d979051 (xxxxxx-DS06): [Timeout] [HB state abcdef02 offset 3858432 gen 271259 stampUS 3726673300366 uuid 5905429a-463f63b5-xxxxxx-ac162d979050 jrnl$

2017-06-12T05:00:58.925Z cpu14:3397944)<3>hpsa 0000:02:00.0: ABORT REQUEST on C2:B0:T0:L7 Tag:0x00000000:000001f0 Command:0x2a SN:0x14bbd1ca  (via RESET) FAILED, Command completed before reset was attempted after delaying 21 seconds.

2017-06-12T05:00:58.925Z cpu10:4133)HBX: 231: Reclaimed heartbeat for volume 54335ddc-3ba7ef17-xxxxxx-ac162d979051 (xxxxxx-DS07): [Timeout] [HB state abcdef02 offset 3858432 gen 271259 stampUS 3726673318303 uuid 5905429a-463f63b5-bd30-ac162d979050 jrnl$

2017-06-12T05:00:58.925Z cpu10:4806)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3858432 gen 126987 stampUS 3726634025478 uuid 5905429a-463f63b5-xxxxxx-ac162d979050 jrnl <FB 61200> drv 14.54] on vol 'Jxxxxxx1-DS05'

2017-06-12T05:00:58.933Z cpu1:4930)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3858432 gen 227 stampUS 3726634025478 uuid 5905429a-463f63b5-xxxxxx-ac162d979050 jrnl <FB 660000> drv 14.54] on vol 'Jxxxxxx-DS08'

2017-06-12T05:00:58.951Z cpu3:3397940)<3>hpsa 0000:02:00.0: ABORT REQUEST on C2:B0:T0:L5 Tag:0x00000000:000001d0 Command:0x2a SN:0x14bbd1c9  (via RESET) FAILED, Command completed before reset was attempted after delaying 23 seconds.

2017-06-12T05:00:58.952Z cpu22:4132)HBX: 231: Reclaimed heartbeat for volume 5902ca80-859fb4db-xxxxxx-ac162d979050 (Jxxxxxx1-DS05): [Timeout] [HB state abcdef02 offset 3858432 gen 126987 stampUS 3726673344845 uuid 5905429a-463f63b5-xxxxxx-ac162d979050 jrnl$

2017-06-12T05:00:58.965Z cpu15:3397939)<3>hpsa 0000:02:00.0: ABORT REQUEST on C2:B0:T0:L8 Tag:0x00000000:00000270 Command:0x2a SN:0x14bbd1ce  (via RESET) FAILED, Command completed before reset was attempted after delaying 25 seconds.

2017-06-12T05:00:58.965Z cpu19:3394587)<3>hpsa 0000:02:00.0: ABORT REQUEST on C2:B0:T0:L2 Tag:0x00000000:00000290 Command:0x2a SN:0x14bbd1cf  (via RESET) FAILED, Command completed before reset was attempted after delaying 29 seconds.

2017-06-12T05:00:58.965Z cpu19:3394587)ScsiCore: 97: Stopping taskMgmt handler world 339458711

2017-06-12T05:00:58.966Z cpu15:4131)HBX: 231: Reclaimed heartbeat for volume 530e251c-3ebc7040-xxxxxx-ac162d979050 (xxxxxx-DS08): [Timeout] [HB state abcdef02 offset 3858432 gen 227 stampUS 3726673358944 uuid 5905429a-463f63b5-xxxxxx-ac162d979050 jrnl <F$

2017-06-12T05:00:58.966Z cpu7:4133)HBX: 231: Reclaimed heartbeat for volume 51ff1fa8-0e11619a-xxxxxx-d89d671a946d (Jxxxxxx1-DS02): [Timeout] [HB state abcdef02 offset 3858432 gen 337 stampUS 3726673359027 uuid 5905429a-463f63b5-xxxxxx-ac162d979050 jrnl <FB$

2017-06-12T05:00:58.989Z cpu2:3397938)<3>hpsa 0000:02:00.0: ABORT REQUEST on C2:B0:T0:L3 Tag:0x00000000:00000250 Command:0x2a SN:0x14bbd1cd  (via RESET) FAILED, Command completed before reset was attempted after delaying 27 seconds.

2017-06-12T05:00:58.990Z cpu22:4132)HBX: 231: Reclaimed heartbeat for volume 51ff20b8-ea3deb85-xxxxxx-d89d671a946d (xxxxxx1-DS03): [Timeout] [HB state abcdef02 offset 3858432 gen 355 stampUS 3726673382793 uuid 5905429a-463f63b5-xxxxxx-ac162d979050 jrnl <F$

[hussainbte]

Hello,

Check if there is a firmware/driver upgrade available for the smart array controller.

share the existing details like ESXi version , existing driver fimrware and I can help you search.

Moreover these are for local datastore so if you are keeping your VMs on SAN it should not affect your Storage connectivity.

[h202b3]

Hello hussainbte

Thanks for looking at this.

ESXi 5.0.0

This is local SAS Raid1 config.

I have multiple ESXi host with the same HP(Smart update manager 15.04) and ESXi firmware. This only seems to happy to 3 out of the 8 HP ESXi Hosts. Whats odd is the hourly event.

Some drivers:

Hewlett-Packard_bootbank_scsi-hpsa_5.0.0-28OEM.500.0.0.472560:

   Name: scsi-hpsa

   Version: 5.0.0-28OEM.500.0.0.472560

Bus Interface: PCI

   Slot: 0

   Serial Number: 001438026403A60

   Cache Serial Number: PBKUC0BRH4H8SA

   RAID 6 (ADG) Status: Enabled

   Controller Status: OK

  Hardware Revision: B

   Firmware Version: 6.34

Please let me know if you require more specific info.

Thanks