Unkillable VM Guest on a ESXi 6 caused by pass-through PCI DVB-S Card (or INTEL USB host-controller ???)

[Sunnyflo84]

==== SEE UPDATE AT END OF POST ====

Hi Guys,

the last days i played a little with my esxi 6 system. i inserted an older pci dvb-s sat card  "B2C2 FlexCopII DVB chip / Technisat SkyStar2 DVB card" to stream Television via WiFi. I configured pass-through, created a new vm, installed debian and it works. But the problem starts if i try to shut down the guest. Two things happen:

1.) Massively Error on ESXi Kernel Log

2016-02-09T21:22:20.545Z cpu0:32820)WARNING: IOMMUIntel: 2720:  DMAR Fault IOMMU Unit #1: R/W=W, Device 0000:00:02.0 Faulting addr = 0xffffff000 Fault Reason = 0x5 -> PTE not set to allow Write.
2016-02-09T21:22:20.545Z cpu0:32820)WARNING: IOMMUIntel: 2777: IOMMU context entry dump for 0000:00:02.0 Ctx-Hi = 0x101 Ctx-Lo = 0x1121c2001
2016-02-09T21:22:20.545Z cpu3:32797)WARNING: IOMMU: 1765: IOMMU Fault detected for 0000:00:02.0 (unnamed/) IOaddr: 0xffffff000 Mask: 0x5 Domain: 0x4301b4487850.
2016-02-09T21:22:20.597Z cpu0:32820)WARNING: IOMMUIntel: 2720:  DMAR Fault IOMMU Unit #1: R/W=W, Device 0000:00:02.0 Faulting addr = 0xffffff000 Fault Reason = 0x5 -> PTE not set to allow Write.
2016-02-09T21:22:20.597Z cpu0:32820)WARNING: IOMMUIntel: 2777: IOMMU context entry dump for 0000:00:02.0 Ctx-Hi = 0x101 Ctx-Lo = 0x1121c2001
2016-02-09T21:22:20.597Z cpu2:32799)WARNING: IOMMU: 1765: IOMMU Fault detected for 0000:00:02.0 (unnamed/) IOaddr: 0xffffff000 Mask: 0x5 Domain: 0x4301b4487850.
2016-02-09T21:22:20.649Z cpu0:32820)WARNING: IOMMUIntel: 2720:  DMAR Fault IOMMU Unit #1: R/W=W, Device 0000:00:02.0 Faulting addr = 0xffffff000 Fault Reason = 0x5 -> PTE not set to allow Write.
2016-02-09T21:22:20.649Z cpu0:32820)WARNING: IOMMUIntel: 2777: IOMMU context entry dump for 0000:00:02.0 Ctx-Hi = 0x101 Ctx-Lo = 0x1121c2001
2016-02-09T21:22:20.649Z cpu2:32798)WARNING: IOMMU: 1765: IOMMU Fault detected for 0000:00:02.0 (unnamed/) IOaddr: 0xffffff000 Mask: 0x5 Domain: 0x4301b4487850.
2016-02-09T21:22:20.701Z cpu0:32820)WARNING: IOMMUIntel: 2720:  DMAR Fault IOMMU Unit #1: R/W=W, Device 0000:00:02.0 Faulting addr = 0xffffff000 Fault Reason = 0x5 -> PTE not set to allow Write.
2016-02-09T21:22:20.701Z cpu0:32820)WARNING: IOMMUIntel: 2777: IOMMU context entry dump for 0000:00:02.0 Ctx-Hi = 0x101 Ctx-Lo = 0x1121c2001
2016-02-09T21:22:20.701Z cpu3:32797)WARNING: IOMMU: 1765: IOMMU Fault detected for 0000:00:02.0 (unnamed/) IOaddr: 0xffffff000 Mask: 0x5 Domain: 0x4301b4487850.
2016-02-09T21:22:20.753Z cpu0:32820)WARNING: IOMMUIntel: 2720:  DMAR Fault IOMMU Unit #1: R/W=W, Device 0000:00:02.0 Faulting addr = 0xffffff000 Fault Reason = 0x5 -> PTE not set to allow Write.
2016-02-09T21:22:20.753Z cpu0:32820)WARNING: IOMMUIntel: 2777: IOMMU context entry dump for 0000:00:02.0 Ctx-Hi = 0x101 Ctx-Lo = 0x1121c2001
2016-02-09T21:22:20.753Z cpu2:32799)WARNING: IOMMU: 1765: IOMMU Fault detected for 0000:00:02.0 (unnamed/) IOaddr: 0xffffff000 Mask: 0x5 Domain: 0x4301b4487850.
2016-02-09T21:22:20.805Z cpu0:32820)WARNING: IOMMUIntel: 2720:  DMAR Fault IOMMU Unit #1: R/W=W, Device 0000:00:02.0 Faulting addr = 0xffffff000 Fault Reason = 0x5 -> PTE not set to allow Write.
2016-02-09T21:22:20.805Z cpu0:32820)WARNING: IOMMUIntel: 2777: IOMMU context entry dump for 0000:00:02.0 Ctx-Hi = 0x101 Ctx-Lo = 0x1121c2001

2.) I cannot power off the guest, or kill it either.

[root@esx:~] esxcli vm process list
[root@esx:~]

9:35:08pm up 24 min, 459 worlds, 0 VMs, 0 vCPUs; CPU load average: 0.62, 0.83, 0.80
PCPU USED(%): 151  76  40  40 AVG:  77
PCPU UTIL(%):  97  73  47  33 AVG:  63
View VM only
      ID      GID NAME             NWLD   %USED    %RUN    %SYS   %WAIT %VMWAIT    %RDY   %IDLE  %OVRLP   %CSTP  %MLMTD  %SWPWT

[root@esx:~] ps | grep vmx
35616  35616  vmx
35622  35616  vmx-vcpu-0:SatServer
[root@esx:~]

hostd.log

2016-02-09T21:44:30.425Z info hostd[FFEF0A80] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5609af24-3e2fae48-fdbd-b8ac6f41639f/SatServer/SatServer.vmx opID=c3eb8e61 user=root] Ticket issued for mks service to user: root

I tried to kill (even with -9 switch) the process, but nothing happens. all other guests are degraded in their performance. The only way is to shutdown the host. But the host doesn' shutdown completly. So i've to do it the really hard way. Push the button

As i said, this happens only when i try to shutdown the guest. I can use the dvb-s card in my linux guest without a problem.

My ESXi Host is a Dell Optiplex 780 System. Bios Revision A15. I know, I know. This is not a certified system, but it does its job.:-)

These are my vibs

[root@esx:~] esxcli software vib list
Name                           Version                               Vendor  Acceptance Level  Install Date
-----------------------------  ------------------------------------  ------  ----------------  ------------
mtip32xx-native                3.8.5-1vmw.600.0.0.2494585            VMWARE  VMwareCertified   2015-09-28
ata-pata-amd                   0.3.10-3vmw.600.0.0.2494585           VMware  VMwareCertified   2015-09-28
ata-pata-atiixp                0.4.6-4vmw.600.0.0.2494585            VMware  VMwareCertified   2015-09-28
ata-pata-cmd64x                0.2.5-3vmw.600.0.0.2494585            VMware  VMwareCertified   2015-09-28
ata-pata-hpt3x2n               0.3.4-3vmw.600.0.0.2494585            VMware  VMwareCertified   2015-09-28
ata-pata-pdc2027x              1.0-3vmw.600.0.0.2494585              VMware  VMwareCertified   2015-09-28
ata-pata-serverworks           0.4.3-3vmw.600.0.0.2494585            VMware  VMwareCertified   2015-09-28
ata-pata-sil680                0.4.8-3vmw.600.0.0.2494585            VMware  VMwareCertified   2015-09-28
ata-pata-via                   0.3.3-2vmw.600.0.0.2494585            VMware  VMwareCertified   2015-09-28
block-cciss                    3.6.14-10vmw.600.0.0.2494585          VMware  VMwareCertified   2015-09-28
cpu-microcode                  6.0.0-0.0.2494585                     VMware  VMwareCertified   2015-09-28
ehci-ehci-hcd                  1.0-3vmw.600.1.26.3380124             VMware  VMwareCertified   2016-02-08
elxnet                         10.2.309.6v-1vmw.600.0.0.2494585      VMware  VMwareCertified   2015-09-28
emulex-esx-elxnetcli           10.2.309.6v-0.0.2494585               VMware  VMwareCertified   2015-09-28
esx-base                       6.0.0-1.26.3380124                    VMware  VMwareCertified   2016-02-08
esx-dvfilter-generic-fastpath  6.0.0-0.0.2494585                     VMware  VMwareCertified   2015-09-28
esx-tboot                      6.0.0-0.0.2494585                     VMware  VMwareCertified   2015-09-28
esx-xserver                    6.0.0-0.0.2494585                     VMware  VMwareCertified   2015-09-28
ima-qla4xxx                    2.02.18-1vmw.600.0.0.2494585          VMware  VMwareCertified   2015-09-28
ipmi-ipmi-devintf              39.1-4vmw.600.0.0.2494585             VMware  VMwareCertified   2015-09-28
ipmi-ipmi-msghandler           39.1-4vmw.600.0.0.2494585             VMware  VMwareCertified   2015-09-28
ipmi-ipmi-si-drv               39.1-4vmw.600.0.0.2494585             VMware  VMwareCertified   2015-09-28
lpfc                           10.2.309.8-2vmw.600.0.0.2494585       VMware  VMwareCertified   2015-09-28
lsi-mr3                        6.605.08.00-7vmw.600.1.17.3029758     VMware  VMwareCertified   2015-09-28
lsi-msgpt3                     06.255.12.00-8vmw.600.1.17.3029758    VMware  VMwareCertified   2015-09-28
lsu-hp-hpsa-plugin             1.0.0-1vmw.600.0.0.2494585            VMware  VMwareCertified   2015-09-28
lsu-lsi-lsi-mr3-plugin         1.0.0-2vmw.600.0.11.2809209           VMware  VMwareCertified   2015-09-28
lsu-lsi-lsi-msgpt3-plugin      1.0.0-1vmw.600.0.0.2494585            VMware  VMwareCertified   2015-09-28
lsu-lsi-megaraid-sas-plugin    1.0.0-2vmw.600.0.11.2809209           VMware  VMwareCertified   2015-09-28
lsu-lsi-mpt2sas-plugin         1.0.0-4vmw.600.1.17.3029758           VMware  VMwareCertified   2015-09-28
lsu-lsi-mptsas-plugin          1.0.0-1vmw.600.0.0.2494585            VMware  VMwareCertified   2015-09-28
misc-cnic-register             1.78.75.v60.7-1vmw.600.0.0.2494585    VMware  VMwareCertified   2015-09-28
misc-drivers                   6.0.0-1.26.3380124                    VMware  VMwareCertified   2016-02-08
net-bnx2                       2.2.4f.v60.10-1vmw.600.0.0.2494585    VMware  VMwareCertified   2015-09-28
net-bnx2x                      1.78.80.v60.12-1vmw.600.0.0.2494585   VMware  VMwareCertified   2015-09-28
net-cnic                       1.78.76.v60.13-2vmw.600.0.0.2494585   VMware  VMwareCertified   2015-09-28
net-e1000                      8.0.3.1-5vmw.600.0.0.2494585          VMware  VMwareCertified   2015-09-28
net-e1000e                     3.2.2.1-1vmw.600.1.26.3380124         VMware  VMwareCertified   2016-02-08
net-enic                       2.1.2.38-2vmw.600.0.0.2494585         VMware  VMwareCertified   2015-09-28
net-forcedeth                  0.61-2vmw.600.0.0.2494585             VMware  VMwareCertified   2015-09-28
net-igb                        5.0.5.1.1-5vmw.600.0.0.2494585        VMware  VMwareCertified   2015-09-28
net-ixgbe                      3.7.13.7.14iov-20vmw.600.0.0.2494585  VMware  VMwareCertified   2015-09-28
net-mlx4-core                  1.9.7.0-1vmw.600.0.0.2494585          VMware  VMwareCertified   2015-09-28
net-mlx4-en                    1.9.7.0-1vmw.600.0.0.2494585          VMware  VMwareCertified   2015-09-28
net-nx-nic                     5.0.621-5vmw.600.0.0.2494585          VMware  VMwareCertified   2015-09-28
net-tg3                        3.131d.v60.4-2vmw.600.1.26.3380124    VMware  VMwareCertified   2016-02-08
net-vmxnet3                    1.1.3.0-3vmw.600.0.0.2494585          VMware  VMwareCertified   2015-09-28
nmlx4-core                     3.0.0.0-1vmw.600.0.0.2494585          VMware  VMwareCertified   2015-09-28
nmlx4-en                       3.0.0.0-1vmw.600.0.0.2494585          VMware  VMwareCertified   2015-09-28
nmlx4-rdma                     3.0.0.0-1vmw.600.0.0.2494585          VMware  VMwareCertified   2015-09-28
nvme                           1.0e.0.35-1vmw.600.1.17.3029758       VMware  VMwareCertified   2015-09-28
ohci-usb-ohci                  1.0-3vmw.600.0.0.2494585              VMware  VMwareCertified   2015-09-28
qlnativefc                     2.0.12.0-5vmw.600.0.0.2494585         VMware  VMwareCertified   2015-09-28
rste                           2.0.2.0088-4vmw.600.0.0.2494585       VMware  VMwareCertified   2015-09-28
sata-ahci                      3.0-22vmw.600.1.17.3029758            VMware  VMwareCertified   2015-09-28
sata-ata-piix                  2.12-10vmw.600.0.0.2494585            VMware  VMwareCertified   2015-09-28
sata-sata-nv                   3.5-4vmw.600.0.0.2494585              VMware  VMwareCertified   2015-09-28
sata-sata-promise              2.12-3vmw.600.0.0.2494585             VMware  VMwareCertified   2015-09-28
sata-sata-sil24                1.1-1vmw.600.0.0.2494585              VMware  VMwareCertified   2015-09-28
sata-sata-sil                  2.3-4vmw.600.0.0.2494585              VMware  VMwareCertified   2015-09-28
sata-sata-svw                  2.3-3vmw.600.0.0.2494585              VMware  VMwareCertified   2015-09-28
scsi-aacraid                   1.1.5.1-9vmw.600.0.0.2494585          VMware  VMwareCertified   2015-09-28
scsi-adp94xx                   1.0.8.12-6vmw.600.0.0.2494585         VMware  VMwareCertified   2015-09-28
scsi-aic79xx                   3.1-5vmw.600.0.0.2494585              VMware  VMwareCertified   2015-09-28
scsi-bnx2fc                    1.78.78.v60.8-1vmw.600.0.0.2494585    VMware  VMwareCertified   2015-09-28
scsi-bnx2i                     2.78.76.v60.8-1vmw.600.0.11.2809209   VMware  VMwareCertified   2015-09-28
scsi-fnic                      1.5.0.45-3vmw.600.0.0.2494585         VMware  VMwareCertified   2015-09-28
scsi-hpsa                      6.0.0.44-4vmw.600.0.0.2494585         VMware  VMwareCertified   2015-09-28
scsi-ips                       7.12.05-4vmw.600.0.0.2494585          VMware  VMwareCertified   2015-09-28
scsi-megaraid-mbox             2.20.5.1-6vmw.600.0.0.2494585         VMware  VMwareCertified   2015-09-28
scsi-megaraid-sas              6.603.55.00-2vmw.600.0.0.2494585      VMware  VMwareCertified   2015-09-28
scsi-megaraid2                 2.00.4-9vmw.600.0.0.2494585           VMware  VMwareCertified   2015-09-28
scsi-mpt2sas                   19.00.00.00-1vmw.600.0.0.2494585      VMware  VMwareCertified   2015-09-28
scsi-mptsas                    4.23.01.00-9vmw.600.0.0.2494585       VMware  VMwareCertified   2015-09-28
scsi-mptspi                    4.23.01.00-9vmw.600.0.0.2494585       VMware  VMwareCertified   2015-09-28
scsi-qla4xxx                   5.01.03.2-7vmw.600.0.0.2494585        VMware  VMwareCertified   2015-09-28
uhci-usb-uhci                  1.0-3vmw.600.0.0.2494585              VMware  VMwareCertified   2015-09-28
vsanhealth                     6.0.0-3000000.2.0.1.17.2972216        VMware  VMwareCertified   2015-09-28
xhci-xhci                      1.0-3vmw.600.1.26.3380124             VMware  VMwareCertified   2016-02-08
tools-light                    6.0.0-1.26.3380124                    VMware  VMwareCertified   2016-02-08

Well, so i have two questions

1.) How to kill a non-killable process ?

2.) What can i do, to make theese errors dissappear ?

Best Regards from Germany

Flo

Update 07.03.2016

Today i realized that this problem is grounded by my dvb-s pci card. The dvb-s card from my post above isn't connected since weeks anymore. Instead i "pass-throughed" one of the built-in usb controller to a windows 7 guest. at guest shutdown i get theese

2016-03-07T20:21:38.404Z cpu0:35474)WARNING: IOMMUIntel: 2777: IOMMU context entry dump for 0000:00:02.0 Ctx-Hi = 0x101 Ctx-Lo = 0x112195001
2016-03-07T20:21:38.455Z cpu0:35474)WARNING: IOMMUIntel: 2720:  DMAR Fault IOMMU Unit #1: R/W=W, Device 0000:00:02.0 Faulting addr = 0xf9ffff000 Fault Reason = 0x5 -> PTE not set to allow Write.
2016-03-07T20:21:38.455Z cpu0:35474)WARNING: IOMMUIntel: 2777: IOMMU context entry dump for 0000:00:02.0 Ctx-Hi = 0x101 Ctx-Lo = 0x112195001
2016-03-07T20:21:38.473Z cpu0:32798)WARNING: IOMMU: 1765: IOMMU Fault detected for 0000:00:02.0 (unnamed/) IOaddr: 0xf9ffff000 Mask: 0x5 Domain: 0x4301c5487850.
2016-03-07T20:21:38.473Z cpu0:32798)WARNING: IOMMU: 1765: IOMMU Fault detected for 0000:00:02.0 (unnamed/) IOaddr: 0xf9ffff000 Mask: 0x5 Domain: 0x4301c5487850.
2016-03-07T20:21:38.473Z cpu0:32798)WARNING: IOMMU: 1765: IOMMU Fault detected for 0000:00:02.0 (unnamed/) IOaddr: 0xf9ffff000 Mask: 0x5 Domain: 0x4301c5487850.
2016-03-07T20:21:38.585Z cpu0:467934)WARNING: IOMMUIntel: 2720:  DMAR Fault IOMMU Unit #1: R/W=W, Device 0000:00:02.0 Faulting addr = 0xf9ffff000 Fault Reason = 0x5 -> PTE not set to allow Write.
2016-03-07T20:21:38.585Z cpu0:467934)WARNING: IOMMUIntel: 2777: IOMMU context entry dump for 0000:00:02.0 Ctx-Hi = 0x101 Ctx-Lo = 0x112195001
2016-03-07T20:21:38.641Z cpu0:467934)WARNING: IOMMUIntel: 2720:  DMAR Fault IOMMU Unit #1: R/W=W, Device 0000:00:02.0 Faulting addr = 0xf9ffff000 Fault Reason = 0x5 -> PTE not set to allow Write.
2016-03-07T20:21:38.641Z cpu0:467934)WARNING: IOMMUIntel: 2777: IOMMU context entry dump for 0000:00:02.0 Ctx-Hi = 0x101 Ctx-Lo = 0x112195001
2016-03-07T20:21:38.707Z cpu0:467934)WARNING: IOMMUIntel: 2720:  DMAR Fault IOMMU Unit #1: R/W=W, Device 0000:00:02.0 Faulting addr = 0xf9ffff000 Fault Reason = 0x5 -> PTE not set to allow Write.
2016-03-07T20:21:38.707Z cpu0:467934)WARNING: IOMMUIntel: 2777: IOMMU context entry dump for 0000:00:02.0 Ctx-Hi = 0x101 Ctx-Lo = 0x112195001
2016-03-07T20:21:38.709Z cpu0:32798)WARNING: IOMMU: 1765: IOMMU Fault detected for 0000:00:02.0 (unnamed/) IOaddr: 0xf9ffff000 Mask: 0x5 Domain: 0x4301c5487850.
2016-03-07T20:21:38.709Z cpu0:32798)WARNING: IOMMU: 1765: IOMMU Fault detected for 0000:00:02.0 (unnamed/) IOaddr: 0xf9ffff000 Mask: 0x5 Domain: 0x4301c5487850.
2016-03-07T20:21:38.709Z cpu0:32798)WARNING: IOMMU: 1765: IOMMU Fault detected for 0000:00:02.0 (unnamed/) IOaddr: 0xf9ffff000 Mask: 0x5 Domain: 0x4301c5487850.
2016-03-07T20:21:38.859Z cpu0:493268)WARNING: IOMMUIntel: 2720:  DMAR Fault IOMMU Unit #1: R/W=W, Device 0000:00:02.0 Faulting addr = 0xf9ffff000 Fault Reason = 0x5 -> PTE not set to allow Write.
2016-03-07T20:21:38.859Z cpu0:493268)WARNING: IOMMUIntel: 2777: IOMMU context entry dump for 0000:00:02.0 Ctx-Hi = 0x101 Ctx-Lo = 0x112195001

The warnings looking quite similar to them mentioned above. Also the impacts on my system are the same. The difference to the case above is, that there is no process left after vm guest is closed.

Meanwhile i installed to latest update from begin of februar.

Interessting point is that a reboot don't end with any error, only the shutdown of the guest.

btw. this is my lspci

0000:00:02.0 VGA compatible controller Display controller: Intel Corporation 4 Series Chipset Integrated Graphics Controller
         Class 0300: 8086:2e12

0000:00:02.1 Display controller Display controller: Intel Corporation 4 Series Chipset Integrated Graphics Controller
         Class 0380: 8086:2e13

So why the heck vmware moan at the integrated vga controller ????

Maybe someone of you have a tip for me 🙂

greetings from germany

Florian

[Sunnyflo84]

Hi, new Update. This error occurs at shutdown a debian guest with a pass-throughed pci-e sata controller card. too. A temporary solution is to suspend the vm guest instead of shutdown it. This won't raise any errors.