7 Replies Latest reply on Jan 2, 2018 10:46 AM by ve3nsv

    ESXi 6 keeps randomly freezing

    rwh23 Enthusiast

      I'm currently in the process of setting up 2 ESXi hosts. I have one up and running with several VM's running on iscsi datastores.

      Randomly since setup the host becomes unresponsive. I can't ping the host nor any of the VM's.

      The host is accessible via BMC, but only enough to log in. After I login it doesn't do anything and I'm forced to reboot the server.

       

      I attached the vmkernel log file in the hopes someone can see something I can't.

       

      Edit:

      Running on ESXi 6.0.0 Bild 3029758

        • 1. Re: ESXi 6 keeps randomly freezing
          cesprov Enthusiast

          While I don't see the exact same error occurring in your vmkernel.log, it's possible you are running into this:

           

          https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2124669

           

          The build number in your vmkernel.log shows as:

           

          0:00:00:06.697 cpu0:32768)Init: 745: vmkernel build Number = 3029758

           

          which is 6.0U1.  The above link was a nasty bug with NICs and CPU interrupts that occurred on specific generations of processors if I remember correctly.  I saw it most on servers that were equivalent in release dates as the x10 Dell series (i.e. R910).  It tanked the whole host with circumstances that sound similar to yours as this is how this issue manifested itself from the outside when the problem was occurring:

           

          "The host is accessible via BMC, but only enough to log in. After I login it doesn't do anything"

           

          You might want to upgrade to at least 6.0U1a to take that out of the equation.  If you go that route, make sure to upgrade your vCenter to the 6.0U1a version or higher before upgrading to the same version on the host or your host may not rejoin vCenter properly.

           

           

          • 2. Re: ESXi 6 keeps randomly freezing
            Luciano Patrao Hot Shot
            vExpert

            Hi,

             

            In the logs iSCSI volumes / Paths are disconnecting from the hosts and this can have that behavior in the hosts.

             

            So check your Storage connections and also the Storage network(ESXi host side and Storage side).

             

            Jail

            • 3. Re: ESXi 6 keeps randomly freezing
              rwh23 Enthusiast

              Thanks for the information. It does sound similar with the issue I am seeing. I will update both hosts as soon as I can.

              • 4. Re: ESXi 6 keeps randomly freezing
                ymolinar Lurker

                Hi. i got the same problem, could you help me please

                • 5. Re: ESXi 6 keeps randomly freezing
                  Finikiez Master
                  vExpert

                  Hi!

                   

                  you have the following issue

                   

                  2017-10-13T12:44:00.020Z cpu2:33118)NMP: nmp_ResetDeviceLogThrottling:3339: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x2 0x3a 0x1 from dev "mpx.vmhba34:C0:T0:L0" occurred 4 times(of 4 commands)

                   

                  0x2 0x3a 0x1 means

                  0x2 - NOT READY

                  0x3a 0x1 - MEDIUM NOT PRESENT - TRAY CLOSED

                   

                   

                  Something is wrong with your local disks.

                  It's hard to say something more without full log bundle.

                   

                  Also I see very long scsi reservation time on datastore1, but I can't say which is a corret path for it

                  2017-10-04T12:38:59.844Z cpu3:35149 opID=6d3e4b78)FS3Misc: 1759: Long VMFS rsv time on 'datastore1' (held for 528 msecs). # R: 2, # W: 1 bytesXfer: 6 sectors

                  2017-10-04T12:39:06.698Z cpu0:35221)FS3Misc: 1759: Long VMFS rsv time on 'datastore1' (held for 487 msecs). # R: 2, # W: 1 bytesXfer: 6 sectors

                   

                  What's your HW?

                  • 6. Re: ESXi 6 keeps randomly freezing
                    ymolinar Lurker

                    Hi thanks for your response, i'm a newbie in esxi servers.

                    What do you mean with full log bundle?

                    This is my machine hardware, i began to learn in a normal computer, not a professional server

                     

                    Display Name: Local ASUS CD-ROM (mpx.vmhba34:C0:T0:L0)

                       Vendor: ASUS      Model: DRW-24F1ST   c    Revis: 1.00

                       Display Name: Local ATA Disk (t10.ATA_____ST1000DM0032D1SB10C__________________________________Z9A27DA9)

                       Vendor: ATA       Model: ST1000DM003-1SB1  Revis: CC43

                    Intel(R) Core(TM) i5-4460  CPU @ 3.20GHz

                    Gigabyte Technology Co., Ltd.

                    B85M-D3H

                    8Gb 4CPU x 3.192GHZ

                     

                    CPU

                     

                    CPU Packages: 1

                       CPU Cores: 4

                       CPU Threads: 4

                       Hyperthreading Active: false

                       Hyperthreading Supported: false

                       Hyperthreading Enabled: true

                       HV Support: 3

                       HV Replay Capable: true

                      

                    PCI

                     

                    0000:00:00.0

                       Address: 0000:00:00.0

                       Segment: 0x0000

                       Bus: 0x00

                       Slot: 0x00

                       Function: 0x0

                       VMkernel Name:

                       Vendor Name: Intel Corporation

                       Device Name: Haswell DRAM Controller

                       Configured Owner: Unknown

                       Current Owner: VMkernel

                       Vendor ID: 0x8086

                       Device ID: 0x0c00

                       SubVendor ID: 0x1458

                       SubDevice ID: 0x5000

                       Device Class: 0x0600

                       Device Class Name: Host bridge

                       Programming Interface: 0x00

                       Revision ID: 0x06

                       Interrupt Line: 0xff

                       IRQ: 255

                       Interrupt Vector: 0x00

                       PCI Pin: 0xff

                       Spawned Bus: 0x00

                       Flags: 0x0200

                       Module ID: -1

                       Module Name: None

                       Chassis: 0

                       Physical Slot: 4294967295

                       Slot Description:

                       Passthru Capable: false

                       Parent Device:

                       Dependent Device:

                       Reset Method: None

                       FPT Sharable: false

                     

                    0000:00:02.0

                       Address: 0000:00:02.0

                       Segment: 0x0000

                       Bus: 0x00

                       Slot: 0x02

                       Function: 0x0

                       VMkernel Name:

                       Vendor Name: Intel Corporation

                       Device Name: Haswell Integrated Graphics Controller

                       Configured Owner: Unknown

                       Current Owner: VMkernel

                       Vendor ID: 0x8086

                       Device ID: 0x0412

                       SubVendor ID: 0x1458

                       SubDevice ID: 0xd000

                       Device Class: 0x0300

                       Device Class Name: VGA compatible controller

                       Programming Interface: 0x00

                       Revision ID: 0x06

                       Interrupt Line: 0x0b

                       IRQ: 11

                       Interrupt Vector: 0x2c

                       PCI Pin: 0x00

                       Spawned Bus: 0x00

                       Flags: 0x0221

                       Module ID: -1

                       Module Name: None

                       Chassis: 0

                       Physical Slot: 0

                       Slot Description: J6B2

                       Passthru Capable: true

                       Parent Device:

                       Dependent Device: PCI 0:0:2:0

                       Reset Method: Function reset

                       FPT Sharable: true

                     

                    0000:00:03.0

                       Address: 0000:00:03.0

                       Segment: 0x0000

                       Bus: 0x00

                       Slot: 0x03

                       Function: 0x0

                       VMkernel Name:

                       Vendor Name: Intel Corporation

                       Device Name: Haswell HD Audio Controller

                       Configured Owner: Unknown

                       Current Owner: VMkernel

                       Vendor ID: 0x8086

                       Device ID: 0x0c0c

                       SubVendor ID: 0x8086

                       SubDevice ID: 0x2010

                       Device Class: 0x0403

                       Device Class Name: Audio device

                       Programming Interface: 0x00

                       Revision ID: 0x06

                       Interrupt Line: 0x0b

                       IRQ: 11

                       Interrupt Vector: 0x2c

                       PCI Pin: 0x00

                       Spawned Bus: 0x00

                       Flags: 0x0201

                       Module ID: -1

                       Module Name: None

                       Chassis: 0

                       Physical Slot: 3

                       Slot Description: J7B1

                       Passthru Capable: true

                       Parent Device:

                       Dependent Device: PCI 0:0:3:0

                       Reset Method: Function reset

                       FPT Sharable: true

                     

                    0000:00:14.0

                       Address: 0000:00:14.0

                       Segment: 0x0000

                       Bus: 0x00

                       Slot: 0x14

                       Function: 0x0

                       VMkernel Name:

                       Vendor Name: Intel Corporation

                       Device Name: Lynx Point USB xHCI Host Controller

                       Configured Owner: Unknown

                       Current Owner: VMkernel

                       Vendor ID: 0x8086

                       Device ID: 0x8c31

                       SubVendor ID: 0x1458

                       SubDevice ID: 0x5007

                       Device Class: 0x0c03

                       Device Class Name: USB controller

                       Programming Interface: 0x30

                       Revision ID: 0x05

                       Interrupt Line: 0x0b

                       IRQ: 11

                       Interrupt Vector: 0x32

                       PCI Pin: 0x00

                       Spawned Bus: 0x00

                       Flags: 0x0201

                       Module ID: 4126

                       Module Name: xhci

                       Chassis: 0

                       Physical Slot: 4294967295

                       Slot Description:

                       Passthru Capable: false

                       Parent Device:

                       Dependent Device:

                       Reset Method: None

                       FPT Sharable: false

                     

                    0000:00:16.0

                       Address: 0000:00:16.0

                       Segment: 0x0000

                       Bus: 0x00

                       Slot: 0x16

                       Function: 0x0

                       VMkernel Name:

                       Vendor Name: Intel Corporation

                       Device Name: Lynx Point MEI Controller #1

                       Configured Owner: Unknown

                       Current Owner: VMkernel

                       Vendor ID: 0x8086

                       Device ID: 0x8c3a

                       SubVendor ID: 0x1458

                       SubDevice ID: 0x1c3a

                       Device Class: 0x0780

                       Device Class Name: Communication controller

                       Programming Interface: 0x00

                       Revision ID: 0x04

                       Interrupt Line: 0x0b

                       IRQ: 11

                       Interrupt Vector: 0x2c

                       PCI Pin: 0x00

                       Spawned Bus: 0x00

                       Flags: 0x0201

                       Module ID: -1

                       Module Name: None

                       Chassis: 0

                       Physical Slot: 4294967295

                       Slot Description:

                       Passthru Capable: false

                       Parent Device:

                       Dependent Device:

                       Reset Method: None

                       FPT Sharable: false

                     

                    0000:00:16.3

                       Address: 0000:00:16.3

                       Segment: 0x0000

                       Bus: 0x00

                       Slot: 0x16

                       Function: 0x3

                       VMkernel Name:

                       Vendor Name: Intel Corporation

                       Device Name: Lynx Point KT Controller

                       Configured Owner: Unknown

                       Current Owner: VMkernel

                       Vendor ID: 0x8086

                       Device ID: 0x8c3d

                       SubVendor ID: 0x1458

                       SubDevice ID: 0x1c3a

                       Device Class: 0x0700

                       Device Class Name: Serial controller

                       Programming Interface: 0x02

                       Revision ID: 0x04

                       Interrupt Line: 0x0a

                       IRQ: 10

                       Interrupt Vector: 0x2d

                       PCI Pin: 0x01

                       Spawned Bus: 0x00

                       Flags: 0x0201

                       Module ID: -1

                       Module Name: None

                       Chassis: 0

                       Physical Slot: 4294967295

                       Slot Description:

                       Passthru Capable: false

                       Parent Device:

                       Dependent Device:

                       Reset Method: None

                       FPT Sharable: false

                     

                    0000:00:1a.0

                       Address: 0000:00:1a.0

                       Segment: 0x0000

                       Bus: 0x00

                       Slot: 0x1a

                       Function: 0x0

                       VMkernel Name:

                       Vendor Name: Intel Corporation

                       Device Name: Lynx Point USB Enhanced Host Controller #2

                       Configured Owner: Unknown

                       Current Owner: VMkernel

                       Vendor ID: 0x8086

                       Device ID: 0x8c2d

                       SubVendor ID: 0x1458

                       SubDevice ID: 0x5006

                       Device Class: 0x0c03

                       Device Class Name: USB controller

                       Programming Interface: 0x20

                       Revision ID: 0x05

                       Interrupt Line: 0x0b

                       IRQ: 11

                       Interrupt Vector: 0x2c

                       PCI Pin: 0x00

                       Spawned Bus: 0x00

                       Flags: 0x0201

                       Module ID: 4125

                       Module Name: ehci-hcd

                       Chassis: 0

                       Physical Slot: 4294967295

                       Slot Description:

                       Passthru Capable: true

                       Parent Device:

                       Dependent Device: PCI 0:0:26:0

                       Reset Method: Function reset

                       FPT Sharable: true

                     

                    0000:00:1b.0

                       Address: 0000:00:1b.0

                       Segment: 0x0000

                       Bus: 0x00

                       Slot: 0x1b

                       Function: 0x0

                       VMkernel Name:

                       Vendor Name: Intel Corporation

                       Device Name: Lynx Point High Definition Audio Controller

                       Configured Owner: Unknown

                       Current Owner: VMkernel

                       Vendor ID: 0x8086

                       Device ID: 0x8c20

                       SubVendor ID: 0x1458

                       SubDevice ID: 0xa002

                       Device Class: 0x0403

                       Device Class Name: Audio device

                       Programming Interface: 0x00

                       Revision ID: 0x05

                       Interrupt Line: 0x03

                       IRQ: 3

                       Interrupt Vector: 0x2e

                       PCI Pin: 0x00

                       Spawned Bus: 0x00

                       Flags: 0x0201

                       Module ID: -1

                       Module Name: None

                       Chassis: 0

                       Physical Slot: 4294967295

                       Slot Description:

                       Passthru Capable: true

                       Parent Device:

                       Dependent Device: PCI 0:0:27:0

                       Reset Method: Function reset

                       FPT Sharable: true

                     

                    0000:00:1c.0

                       Address: 0000:00:1c.0

                       Segment: 0x0000

                       Bus: 0x00

                       Slot: 0x1c

                       Function: 0x0

                       VMkernel Name: PCIe RP[0000:00:1c.0]

                       Vendor Name: Intel Corporation

                       Device Name: Lynx Point PCI Express Root Port #1

                       Configured Owner: Unknown

                       Current Owner: VMkernel

                       Vendor ID: 0x8086

                       Device ID: 0x8c10

                       SubVendor ID: 0x0000

                       SubDevice ID: 0x0000

                       Device Class: 0x0604

                       Device Class Name: PCI bridge

                       Programming Interface: 0x00

                       Revision ID: 0xd5

                       Interrupt Line: 0x0b

                       IRQ: 11

                       Interrupt Vector: 0x2c

                       PCI Pin: 0x00

                       Spawned Bus: 0x01

                       Flags: 0x0203

                       Module ID: 0

                       Module Name: vmkernel

                       Chassis: 0

                       Physical Slot: 1

                       Slot Description: J6B1

                       Passthru Capable: false

                       Parent Device:

                       Dependent Device:

                       Reset Method: None

                       FPT Sharable: false

                     

                    0000:00:1c.2

                       Address: 0000:00:1c.2

                       Segment: 0x0000

                       Bus: 0x00

                       Slot: 0x1c

                       Function: 0x2

                       VMkernel Name: PCIe RP[0000:00:1c.2]

                       Vendor Name: Intel Corporation

                       Device Name: Lynx Point PCI Express Root Port #3

                       Configured Owner: Unknown

                       Current Owner: VMkernel

                       Vendor ID: 0x8086

                       Device ID: 0x8c14

                       SubVendor ID: 0x0000

                       SubDevice ID: 0x0000

                       Device Class: 0x0604

                       Device Class Name: PCI bridge

                       Programming Interface: 0x00

                       Revision ID: 0xd5

                       Interrupt Line: 0x0b

                       IRQ: 11

                       Interrupt Vector: 0x2f

                       PCI Pin: 0x02

                       Spawned Bus: 0x02

                       Flags: 0x0203

                       Module ID: 0

                       Module Name: vmkernel

                       Chassis: 0

                       Physical Slot: 1

                       Slot Description: J6B1

                       Passthru Capable: false

                       Parent Device:

                       Dependent Device:

                       Reset Method: None

                       FPT Sharable: false

                     

                    0000:00:1d.0

                       Address: 0000:00:1d.0

                       Segment: 0x0000

                       Bus: 0x00

                       Slot: 0x1d

                       Function: 0x0

                       VMkernel Name:

                       Vendor Name: Intel Corporation

                       Device Name: Lynx Point USB Enhanced Host Controller #1

                       Configured Owner: Unknown

                       Current Owner: VMkernel

                       Vendor ID: 0x8086

                       Device ID: 0x8c26

                       SubVendor ID: 0x1458

                       SubDevice ID: 0x5006

                       Device Class: 0x0c03

                       Device Class Name: USB controller

                       Programming Interface: 0x20

                       Revision ID: 0x05

                       Interrupt Line: 0x0a

                       IRQ: 10

                       Interrupt Vector: 0x30

                       PCI Pin: 0x00

                       Spawned Bus: 0x00

                       Flags: 0x0201

                       Module ID: 4125

                       Module Name: ehci-hcd

                       Chassis: 0

                       Physical Slot: 4294967295

                       Slot Description:

                       Passthru Capable: true

                       Parent Device:

                       Dependent Device: PCI 0:0:29:0

                       Reset Method: Function reset

                       FPT Sharable: true

                     

                    0000:00:1f.0

                       Address: 0000:00:1f.0

                       Segment: 0x0000

                       Bus: 0x00

                       Slot: 0x1f

                       Function: 0x0

                       VMkernel Name:

                       Vendor Name: Intel Corporation

                       Device Name: Lynx Point LPC Controller

                       Configured Owner: Unknown

                       Current Owner: VMkernel

                       Vendor ID: 0x8086

                       Device ID: 0x8c50

                       SubVendor ID: 0x1458

                       SubDevice ID: 0x5001

                       Device Class: 0x0601

                       Device Class Name: ISA bridge

                       Programming Interface: 0x00

                       Revision ID: 0x05

                       Interrupt Line: 0xff

                       IRQ: 255

                       Interrupt Vector: 0x00

                       PCI Pin: 0xff

                       Spawned Bus: 0x00

                       Flags: 0x0200

                       Module ID: -1

                       Module Name: None

                       Chassis: 0

                       Physical Slot: 4294967295

                       Slot Description:

                       Passthru Capable: false

                       Parent Device:

                       Dependent Device:

                       Reset Method: None

                       FPT Sharable: false

                     

                    0000:00:1f.2

                       Address: 0000:00:1f.2

                       Segment: 0x0000

                       Bus: 0x00

                       Slot: 0x1f

                       Function: 0x2

                       VMkernel Name: vmhba0

                       Vendor Name: Intel Corporation

                       Device Name: Lynx Point AHCI Controller

                       Configured Owner: Unknown

                       Current Owner: VMkernel

                       Vendor ID: 0x8086

                       Device ID: 0x8c02

                       SubVendor ID: 0x1458

                       SubDevice ID: 0xb005

                       Device Class: 0x0106

                       Device Class Name: SATA controller

                       Programming Interface: 0x01

                       Revision ID: 0x05

                       Interrupt Line: 0x0a

                       IRQ: 10

                       Interrupt Vector: 0x2d

                       PCI Pin: 0x01

                       Spawned Bus: 0x00

                       Flags: 0x0201

                       Module ID: 4161

                       Module Name: ahci

                       Chassis: 0

                       Physical Slot: 4294967295

                       Slot Description:

                       Passthru Capable: false

                       Parent Device:

                       Dependent Device:

                       Reset Method: None

                       FPT Sharable: false

                     

                    0000:00:1f.3

                       Address: 0000:00:1f.3

                       Segment: 0x0000

                       Bus: 0x00

                       Slot: 0x1f

                       Function: 0x3

                       VMkernel Name:

                       Vendor Name: Intel Corporation

                       Device Name: Lynx Point SMBus Controller

                       Configured Owner: Unknown

                       Current Owner: VMkernel

                       Vendor ID: 0x8086

                       Device ID: 0x8c22

                       SubVendor ID: 0x1458

                       SubDevice ID: 0x5001

                       Device Class: 0x0c05

                       Device Class Name: SMBus

                       Programming Interface: 0x00

                       Revision ID: 0x05

                       Interrupt Line: 0x0b

                       IRQ: 255

                       Interrupt Vector: 0x00

                       PCI Pin: 0x02

                       Spawned Bus: 0x00

                       Flags: 0x0201

                       Module ID: -1

                       Module Name: None

                       Chassis: 0

                       Physical Slot: 4294967295

                       Slot Description:

                       Passthru Capable: false

                       Parent Device:

                       Dependent Device:

                       Reset Method: None

                       FPT Sharable: false

                     

                    0000:02:00.0

                       Address: 0000:02:00.0

                       Segment: 0x0000

                       Bus: 0x02

                       Slot: 0x00

                       Function: 0x0

                       VMkernel Name: vmnic0

                       Vendor Name: Realtek Semiconductor Co., Ltd.

                       Device Name: Motherboard

                       Configured Owner: Unknown

                       Current Owner: VMkernel

                       Vendor ID: 0x10ec

                       Device ID: 0x8168

                       SubVendor ID: 0x1458

                       SubDevice ID: 0xe000

                       Device Class: 0x0200

                       Device Class Name: Ethernet controller

                       Programming Interface: 0x00

                       Revision ID: 0x06

                       Interrupt Line: 0x0b

                       IRQ: 11

                       Interrupt Vector: 0x31

                       PCI Pin: 0x00

                       Spawned Bus: 0x00

                       Flags: 0x0201

                       Module ID: 4123

                       Module Name: r8168

                       Chassis: 0

                       Physical Slot: 4294967295

                       Slot Description: J6B1; relative bdf 00:00.0

                       Passthru Capable: true

                       Parent Device: PCI 0:0:28:2

                       Dependent Device: PCI 0:2:0:0

                       Reset Method: Bridge reset

                       FPT Sharable: true

                    • 7. Re: ESXi 6 keeps randomly freezing
                      ve3nsv Lurker

                      I seem to be having a similar issue with a VM locking up on ESXi 6.5 using a NUC Skull Canyon and was hoping somebody could help me out. This host does have a lot of USB peripherals attached to it and only this Windows VM with 2 sound cards and a USB to Serial adapter seems to be locking up.

                       

                      I am running 3 Windows VM's and 2 Debian VM's.