ESXi 6 keeps randomly freezing

rwh23 · ‎04-12-2016

I'm currently in the process of setting up 2 ESXi hosts. I have one up and running with several VM's running on iscsi datastores.

Randomly since setup the host becomes unresponsive. I can't ping the host nor any of the VM's.

The host is accessible via BMC, but only enough to log in. After I login it doesn't do anything and I'm forced to reboot the server.

I attached the vmkernel log file in the hopes someone can see something I can't.

Edit:

Running on ESXi 6.0.0 Bild 3029758

cesprov · ‎04-12-2016

While I don't see the exact same error occurring in your vmkernel.log, it's possible you are running into this:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=21246...

The build number in your vmkernel.log shows as:

0:00:00:06.697 cpu0:32768)Init: 745: vmkernel build Number = 3029758

which is 6.0U1. The above link was a nasty bug with NICs and CPU interrupts that occurred on specific generations of processors if I remember correctly. I saw it most on servers that were equivalent in release dates as the x10 Dell series (i.e. R910). It tanked the whole host with circumstances that sound similar to yours as this is how this issue manifested itself from the outside when the problem was occurring:

"The host is accessible via BMC, but only enough to log in. After I login it doesn't do anything"

You might want to upgrade to at least 6.0U1a to take that out of the equation. If you go that route, make sure to upgrade your vCenter to the 6.0U1a version or higher before upgrading to the same version on the host or your host may not rejoin vCenter properly.

LucianoPatrão · ‎04-12-2016

Hi,

In the logs iSCSI volumes / Paths are disconnecting from the hosts and this can have that behavior in the hosts.

So check your Storage connections and also the Storage network(ESXi host side and Storage side).

Jail

Luciano Patrão

VCP-DCV, VCAP-DCV Design 2023, VCP-Cloud 2023
vExpert vSAN, NSX, Cloud Provider, Veeam Vanguard
Solutions Architect - Tech Lead for VMware / Virtual Backups

________________________________
If helpful Please award points
Thank You
Blog: https://www.provirtualzone.com | Twitter: @Luciano_PT

rwh23 · ‎04-12-2016

Thanks for the information. It does sound similar with the issue I am seeing. I will update both hosts as soon as I can.

ymolinar · ‎10-13-2017

Hi. i got the same problem, could you help me please

Finikiez · ‎10-13-2017

Hi!

you have the following issue

2017-10-13T12:44:00.020Z cpu2:33118)NMP: nmp_ResetDeviceLogThrottling:3339: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x2 0x3a 0x1 from dev "mpx.vmhba34:C0:T0:L0" occurred 4 times(of 4 commands)

0x2 0x3a 0x1 means

0x2 - NOT READY

0x3a 0x1 - MEDIUM NOT PRESENT - TRAY CLOSED

Something is wrong with your local disks.

It's hard to say something more without full log bundle.

Also I see very long scsi reservation time on datastore1, but I can't say which is a corret path for it

2017-10-04T12:38:59.844Z cpu3:35149 opID=6d3e4b78)FS3Misc: 1759: Long VMFS rsv time on 'datastore1' (held for 528 msecs). # R: 2, # W: 1 bytesXfer: 6 sectors

2017-10-04T12:39:06.698Z cpu0:35221)FS3Misc: 1759: Long VMFS rsv time on 'datastore1' (held for 487 msecs). # R: 2, # W: 1 bytesXfer: 6 sectors

What's your HW?

ymolinar · ‎10-16-2017

Hi thanks for your response, i'm a newbie in esxi servers.

What do you mean with full log bundle?

This is my machine hardware, i began to learn in a normal computer, not a professional server

Display Name: Local ASUS CD-ROM (mpx.vmhba34:C0:T0:L0)

Vendor: ASUS Model: DRW-24F1ST c Revis: 1.00

Display Name: Local ATA Disk (t10.ATA_____ST1000DM0032D1SB10C__________________________________Z9A27DA9)

Vendor: ATA Model: ST1000DM003-1SB1 Revis: CC43

Intel(R) Core(TM) i5-4460 CPU @ 3.20GHz

Gigabyte Technology Co., Ltd.

B85M-D3H

8Gb 4CPU x 3.192GHZ

CPU

CPU Packages: 1

CPU Cores: 4

CPU Threads: 4

Hyperthreading Active: false

Hyperthreading Supported: false

Hyperthreading Enabled: true

HV Support: 3

HV Replay Capable: true

PCI

0000:00:00.0

Address: 0000:00:00.0

Segment: 0x0000

Bus: 0x00

Slot: 0x00

Function: 0x0

VMkernel Name:

Vendor Name: Intel Corporation

Device Name: Haswell DRAM Controller

Configured Owner: Unknown

Current Owner: VMkernel

Vendor ID: 0x8086

Device ID: 0x0c00

SubVendor ID: 0x1458

SubDevice ID: 0x5000

Device Class: 0x0600

Device Class Name: Host bridge

Programming Interface: 0x00

Revision ID: 0x06

Interrupt Line: 0xff

IRQ: 255

Interrupt Vector: 0x00

PCI Pin: 0xff

Spawned Bus: 0x00

Flags: 0x0200

Module ID: -1

Module Name: None

Chassis: 0

Physical Slot: 4294967295

Slot Description:

Passthru Capable: false

Parent Device:

Dependent Device:

Reset Method: None

FPT Sharable: false

0000:00:02.0

Address: 0000:00:02.0

Segment: 0x0000

Bus: 0x00

Slot: 0x02

Function: 0x0

VMkernel Name:

Vendor Name: Intel Corporation

Device Name: Haswell Integrated Graphics Controller

Configured Owner: Unknown

Current Owner: VMkernel

Vendor ID: 0x8086

Device ID: 0x0412

SubVendor ID: 0x1458

SubDevice ID: 0xd000

Device Class: 0x0300

Device Class Name: VGA compatible controller

Programming Interface: 0x00

Revision ID: 0x06

Interrupt Line: 0x0b

IRQ: 11

Interrupt Vector: 0x2c

PCI Pin: 0x00

Spawned Bus: 0x00

Flags: 0x0221

Module ID: -1

Module Name: None

Chassis: 0

Physical Slot: 0

Slot Description: J6B2

Passthru Capable: true

Parent Device:

Dependent Device: PCI 0:0:2:0

Reset Method: Function reset

FPT Sharable: true

0000:00:03.0

Address: 0000:00:03.0

Segment: 0x0000

Bus: 0x00

Slot: 0x03

Function: 0x0

VMkernel Name:

Vendor Name: Intel Corporation

Device Name: Haswell HD Audio Controller

Configured Owner: Unknown

Current Owner: VMkernel

Vendor ID: 0x8086

Device ID: 0x0c0c

SubVendor ID: 0x8086

SubDevice ID: 0x2010

Device Class: 0x0403

Device Class Name: Audio device

Programming Interface: 0x00

Revision ID: 0x06

Interrupt Line: 0x0b

IRQ: 11

Interrupt Vector: 0x2c

PCI Pin: 0x00

Spawned Bus: 0x00

Flags: 0x0201

Module ID: -1

Module Name: None

Chassis: 0

Physical Slot: 3

Slot Description: J7B1

Passthru Capable: true

Parent Device:

Dependent Device: PCI 0:0:3:0

Reset Method: Function reset

FPT Sharable: true

0000:00:14.0

Address: 0000:00:14.0

Segment: 0x0000

Bus: 0x00

Slot: 0x14

Function: 0x0

VMkernel Name:

Vendor Name: Intel Corporation

Device Name: Lynx Point USB xHCI Host Controller

Configured Owner: Unknown

Current Owner: VMkernel

Vendor ID: 0x8086

Device ID: 0x8c31

SubVendor ID: 0x1458

SubDevice ID: 0x5007

Device Class: 0x0c03

Device Class Name: USB controller

Programming Interface: 0x30

Revision ID: 0x05

Interrupt Line: 0x0b

IRQ: 11

Interrupt Vector: 0x32

PCI Pin: 0x00

Spawned Bus: 0x00

Flags: 0x0201

Module ID: 4126

Module Name: xhci

Chassis: 0

Physical Slot: 4294967295

Slot Description:

Passthru Capable: false

Parent Device:

Dependent Device:

Reset Method: None

FPT Sharable: false

0000:00:16.0

Address: 0000:00:16.0

Segment: 0x0000

Bus: 0x00

Slot: 0x16

Function: 0x0

VMkernel Name:

Vendor Name: Intel Corporation

Device Name: Lynx Point MEI Controller #1

Configured Owner: Unknown

Current Owner: VMkernel

Vendor ID: 0x8086

Device ID: 0x8c3a

SubVendor ID: 0x1458

SubDevice ID: 0x1c3a

Device Class: 0x0780

Device Class Name: Communication controller

Programming Interface: 0x00

Revision ID: 0x04

Interrupt Line: 0x0b

IRQ: 11

Interrupt Vector: 0x2c

PCI Pin: 0x00

Spawned Bus: 0x00

Flags: 0x0201

Module ID: -1

Module Name: None

Chassis: 0

Physical Slot: 4294967295

Slot Description:

Passthru Capable: false

Parent Device:

Dependent Device:

Reset Method: None

FPT Sharable: false

0000:00:16.3

Address: 0000:00:16.3

Segment: 0x0000

Bus: 0x00

Slot: 0x16

Function: 0x3

VMkernel Name:

Vendor Name: Intel Corporation

Device Name: Lynx Point KT Controller

Configured Owner: Unknown

Current Owner: VMkernel

Vendor ID: 0x8086

Device ID: 0x8c3d

SubVendor ID: 0x1458

SubDevice ID: 0x1c3a

Device Class: 0x0700

Device Class Name: Serial controller

Programming Interface: 0x02

Revision ID: 0x04

Interrupt Line: 0x0a

IRQ: 10

Interrupt Vector: 0x2d

PCI Pin: 0x01

Spawned Bus: 0x00

Flags: 0x0201

Module ID: -1

Module Name: None

Chassis: 0

Physical Slot: 4294967295

Slot Description:

Passthru Capable: false

Parent Device:

Dependent Device:

Reset Method: None

FPT Sharable: false

0000:00:1a.0

Address: 0000:00:1a.0

Segment: 0x0000

Bus: 0x00

Slot: 0x1a

Function: 0x0

VMkernel Name:

Vendor Name: Intel Corporation

Device Name: Lynx Point USB Enhanced Host Controller #2

Configured Owner: Unknown

Current Owner: VMkernel

Vendor ID: 0x8086

Device ID: 0x8c2d

SubVendor ID: 0x1458

SubDevice ID: 0x5006

Device Class: 0x0c03

Device Class Name: USB controller

Programming Interface: 0x20

Revision ID: 0x05

Interrupt Line: 0x0b

IRQ: 11

Interrupt Vector: 0x2c

PCI Pin: 0x00

Spawned Bus: 0x00

Flags: 0x0201

Module ID: 4125

Module Name: ehci-hcd

Chassis: 0

Physical Slot: 4294967295

Slot Description:

Passthru Capable: true

Parent Device:

Dependent Device: PCI 0:0:26:0

Reset Method: Function reset

FPT Sharable: true

0000:00:1b.0

Address: 0000:00:1b.0

Segment: 0x0000

Bus: 0x00

Slot: 0x1b

Function: 0x0

VMkernel Name:

Vendor Name: Intel Corporation

Device Name: Lynx Point High Definition Audio Controller

Configured Owner: Unknown

Current Owner: VMkernel

Vendor ID: 0x8086

Device ID: 0x8c20

SubVendor ID: 0x1458

SubDevice ID: 0xa002

Device Class: 0x0403

Device Class Name: Audio device

Programming Interface: 0x00

Revision ID: 0x05

Interrupt Line: 0x03

IRQ: 3

Interrupt Vector: 0x2e

PCI Pin: 0x00

Spawned Bus: 0x00

Flags: 0x0201

Module ID: -1

Module Name: None

Chassis: 0

Physical Slot: 4294967295

Slot Description:

Passthru Capable: true

Parent Device:

Dependent Device: PCI 0:0:27:0

Reset Method: Function reset

FPT Sharable: true

0000:00:1c.0

Address: 0000:00:1c.0

Segment: 0x0000

Bus: 0x00

Slot: 0x1c

Function: 0x0

VMkernel Name: PCIe RP[0000:00:1c.0]

Vendor Name: Intel Corporation

Device Name: Lynx Point PCI Express Root Port #1

Configured Owner: Unknown

Current Owner: VMkernel

Vendor ID: 0x8086

Device ID: 0x8c10

SubVendor ID: 0x0000

SubDevice ID: 0x0000

Device Class: 0x0604

Device Class Name: PCI bridge

Programming Interface: 0x00

Revision ID: 0xd5

Interrupt Line: 0x0b

IRQ: 11

Interrupt Vector: 0x2c

PCI Pin: 0x00

Spawned Bus: 0x01

Flags: 0x0203

Module ID: 0

Module Name: vmkernel

Chassis: 0

Physical Slot: 1

Slot Description: J6B1

Passthru Capable: false

Parent Device:

Dependent Device:

Reset Method: None

FPT Sharable: false

0000:00:1c.2

Address: 0000:00:1c.2

Segment: 0x0000

Bus: 0x00

Slot: 0x1c

Function: 0x2

VMkernel Name: PCIe RP[0000:00:1c.2]

Vendor Name: Intel Corporation

Device Name: Lynx Point PCI Express Root Port #3

Configured Owner: Unknown

Current Owner: VMkernel

Vendor ID: 0x8086

Device ID: 0x8c14

SubVendor ID: 0x0000

SubDevice ID: 0x0000

Device Class: 0x0604

Device Class Name: PCI bridge

Programming Interface: 0x00

Revision ID: 0xd5

Interrupt Line: 0x0b

IRQ: 11

Interrupt Vector: 0x2f

PCI Pin: 0x02

Spawned Bus: 0x02

Flags: 0x0203

Module ID: 0

Module Name: vmkernel

Chassis: 0

Physical Slot: 1

Slot Description: J6B1

Passthru Capable: false

Parent Device:

Dependent Device:

Reset Method: None

FPT Sharable: false

0000:00:1d.0

Address: 0000:00:1d.0

Segment: 0x0000

Bus: 0x00

Slot: 0x1d

Function: 0x0

VMkernel Name:

Vendor Name: Intel Corporation

Device Name: Lynx Point USB Enhanced Host Controller #1

Configured Owner: Unknown

Current Owner: VMkernel

Vendor ID: 0x8086

Device ID: 0x8c26

SubVendor ID: 0x1458

SubDevice ID: 0x5006

Device Class: 0x0c03

Device Class Name: USB controller

Programming Interface: 0x20

Revision ID: 0x05

Interrupt Line: 0x0a

IRQ: 10

Interrupt Vector: 0x30

PCI Pin: 0x00

Spawned Bus: 0x00

Flags: 0x0201

Module ID: 4125

Module Name: ehci-hcd

Chassis: 0

Physical Slot: 4294967295

Slot Description:

Passthru Capable: true

Parent Device:

Dependent Device: PCI 0:0:29:0

Reset Method: Function reset

FPT Sharable: true

0000:00:1f.0

Address: 0000:00:1f.0

Segment: 0x0000

Bus: 0x00

Slot: 0x1f

Function: 0x0

VMkernel Name:

Vendor Name: Intel Corporation

Device Name: Lynx Point LPC Controller

Configured Owner: Unknown

Current Owner: VMkernel

Vendor ID: 0x8086

Device ID: 0x8c50

SubVendor ID: 0x1458

SubDevice ID: 0x5001

Device Class: 0x0601

Device Class Name: ISA bridge

Programming Interface: 0x00

Revision ID: 0x05

Interrupt Line: 0xff

IRQ: 255

Interrupt Vector: 0x00

PCI Pin: 0xff

Spawned Bus: 0x00

Flags: 0x0200

Module ID: -1

Module Name: None

Chassis: 0

Physical Slot: 4294967295

Slot Description:

Passthru Capable: false

Parent Device:

Dependent Device:

Reset Method: None

FPT Sharable: false

0000:00:1f.2

Address: 0000:00:1f.2

Segment: 0x0000

Bus: 0x00

Slot: 0x1f

Function: 0x2

VMkernel Name: vmhba0

Vendor Name: Intel Corporation

Device Name: Lynx Point AHCI Controller

Configured Owner: Unknown

Current Owner: VMkernel

Vendor ID: 0x8086

Device ID: 0x8c02

SubVendor ID: 0x1458

SubDevice ID: 0xb005

Device Class: 0x0106

Device Class Name: SATA controller

Programming Interface: 0x01

Revision ID: 0x05

Interrupt Line: 0x0a

IRQ: 10

Interrupt Vector: 0x2d

PCI Pin: 0x01

Spawned Bus: 0x00

Flags: 0x0201

Module ID: 4161

Module Name: ahci

Chassis: 0

Physical Slot: 4294967295

Slot Description:

Passthru Capable: false

Parent Device:

Dependent Device:

Reset Method: None

FPT Sharable: false

0000:00:1f.3

Address: 0000:00:1f.3

Segment: 0x0000

Bus: 0x00

Slot: 0x1f

Function: 0x3

VMkernel Name:

Vendor Name: Intel Corporation

Device Name: Lynx Point SMBus Controller

Configured Owner: Unknown

Current Owner: VMkernel

Vendor ID: 0x8086

Device ID: 0x8c22

SubVendor ID: 0x1458

SubDevice ID: 0x5001

Device Class: 0x0c05

Device Class Name: SMBus

Programming Interface: 0x00

Revision ID: 0x05

Interrupt Line: 0x0b

IRQ: 255

Interrupt Vector: 0x00

PCI Pin: 0x02

Spawned Bus: 0x00

Flags: 0x0201

Module ID: -1

Module Name: None

Chassis: 0

Physical Slot: 4294967295

Slot Description:

Passthru Capable: false

Parent Device:

Dependent Device:

Reset Method: None

FPT Sharable: false

0000:02:00.0

Address: 0000:02:00.0

Segment: 0x0000

Bus: 0x02

Slot: 0x00

Function: 0x0

VMkernel Name: vmnic0

Vendor Name: Realtek Semiconductor Co., Ltd.

Device Name: Motherboard

Configured Owner: Unknown

Current Owner: VMkernel

Vendor ID: 0x10ec

Device ID: 0x8168

SubVendor ID: 0x1458

SubDevice ID: 0xe000

Device Class: 0x0200

Device Class Name: Ethernet controller

Programming Interface: 0x00

Revision ID: 0x06

Interrupt Line: 0x0b

IRQ: 11

Interrupt Vector: 0x31

PCI Pin: 0x00

Spawned Bus: 0x00

Flags: 0x0201

Module ID: 4123

Module Name: r8168

Chassis: 0

Physical Slot: 4294967295

Slot Description: J6B1; relative bdf 00:00.0

Passthru Capable: true

Parent Device: PCI 0:0:28:2

Dependent Device: PCI 0:2:0:0

Reset Method: Bridge reset

FPT Sharable: true

ve3nsv · ‎01-02-2018

I seem to be having a similar issue with a VM locking up on ESXi 6.5 using a NUC Skull Canyon and was hoping somebody could help me out. This host does have a lot of USB peripherals attached to it and only this Windows VM with 2 sound cards and a USB to Serial adapter seems to be locking up.

I am running 3 Windows VM's and 2 Debian VM's.

All

ESXi 6 keeps randomly freezing