Highlighted
Enthusiast
Enthusiast

ESXi 5.1 vmnic's are down (Inte 82599 with 10 GB Port)

Hello,

We have ESXi 5.1 host (UCS C240 MS4) with Intel 82599 10 Gigabit Dual Port Network Connection and frequently vmnic's are down.

While checking the vmkernal logsl, following error message are getting.

2016-04-06T15:15:28.643Z cpu0:16931)<3>ixgbe 0000:06:00.0: Adapter removed

2016-04-06T15:15:29.011Z cpu0:16949)<3>ixgbe 0000:06:00.1: Adapter removed

2016-04-06T15:15:29.011Z cpu0:16949)<3>ixgbe 0000:02:00.1: Adapter removed

2016-04-06T15:15:29.011Z cpu0:16949)<3>ixgbe 0000:02:00.0: Adapter removed

2016-04-06T15:15:46.480Z cpu35:16419)StorageApdHandler: 248: APD Timer started for ident [ccb58c6b-19688665]

2016-04-06T15:15:46.480Z cpu35:16419)StorageApdHandler: 395: Device or filesystem with identifier [ccb58c6b-19688665] has entered the All Paths Down state.

2016-04-06T15:15:46.480Z cpu35:16419)StorageApdHandler: 846: APD Start for ident [ccb58c6b-19688665]!

2016-04-06T15:15:46.480Z cpu35:16419)StorageApdHandler: 248: APD Timer started for ident [44cb1195-7ada2d28]

2016-04-06T15:15:46.480Z cpu35:16419)StorageApdHandler: 395: Device or filesystem with identifier [44cb1195-7ada2d28] has entered the All Paths Down state.

2016-04-06T15:15:46.480Z cpu35:16419)StorageApdHandler: 846: APD Start for ident [44cb1195-7ada2d28]!

2016-04-06T15:16:02.507Z cpu0:16420)NFSLock: 610: Stop accessing fd 0x410019891cb8  3

2016-04-06T15:16:02.507Z cpu0:16420)NFSLock: 610: Stop accessing fd 0x41001988a5b8  3

2016-04-06T15:16:02.507Z cpu0:16420)NFSLock: 610: Stop accessing fd 0x41001a64c9e8  3

2016-04-06T15:16:02.507Z cpu0:16420)NFSLock: 610: Stop accessing fd 0x4100198866b8  3

2016-04-06T15:17:34.654Z cpu0:16422)WARNING: NFS: 322: Lost connection to the server 172.25.226.29 mount point /n6wvsvm001_ESX_d1, mounted as 44cb1195-7ada2d28-0000-000000000000 ("_6W-NU6W001")

2016-04-06T15:17:34.654Z cpu0:16422)WARNING: NFS: 322: Lost connection to the server 172.25.226.29 mount point /n6wvsvm001_ESX_d2, mounted as ccb58c6b-19688665-0000-000000000000 ("_6W-NU6W002")

2016-04-06T15:18:06.485Z cpu35:16419)StorageApdHandler: 411: Device or filesystem with identifier [ccb58c6b-19688665] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed.

2016-04-06T15:18:06.485Z cpu35:16419)StorageApdHandler: 320: APD Timeout for ident [ccb58c6b-19688665]!

2016-04-06T15:18:06.485Z cpu35:16419)StorageApdHandler: 411: Device or filesystem with identifier [44cb1195-7ada2d28] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed.

2016-04-06T15:18:06.485Z cpu35:16419)StorageApdHandler: 320: APD Timeout for ident [44cb1195-7ada2d28]!

2016-04-06T15:18:12.531Z cpu33:18352)BC: 4526: Failed to flush 1 buffers of size 8192 each for object 'host-1146-hb' b00f 52 44cb1195 7ada2d28 0 80000405 0 395b ee21ed0 80000405 4000000000 4001ea9edc 0 0: No connection

2016-04-06T15:18:12.531Z cpu0:18348)BC: 4526: Failed to flush 1 buffers of size 8192 each for object 'host-1146-hb' b00f 52 ccb58c6b 19688665 0 80000406 0 7c0b ab62b17 80000406 4000000000 406b129a72 0 0: No connection

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful
1 Reply
Highlighted
Enthusiast
Enthusiast

I would start with firmware / driver upgrade.

Martin Gavanda https://martingavanda.com https://learnvmware.online