Memnarch Tracker

Win 11 sound/audio stutter with 2023 Fusion preview

Memnarch — Sun, 24 Sep 2023 18:18:54 GMT

Hello all. Using Mac Pro M1 14, win 11 pro release 22621 fully patched, latest vmware 2023 tech preview (e x p).

Audio has considerable stutter/lag/popping. This is noticeable even when playing the test sounds in the windows control panel. Choosing higher bit depths/sampling rates makes it worse, but even present with the lowest 16k/44kHz. Very noticeable for example with 3dmark fire strike demo (which otherwise works well!).

I've searched the forums and don't see any recent fixes. A parallels demo install has no similar issues.

Any thoughts? Thanks, LT

Re: Windows 11 Arm sound driver and audio latency

Memnarch — Sun, 24 Sep 2023 16:05:29 GMT

Any update on this? I'm having marked audio stutter with the 2023 tech preview and win 11 insider edition.

Re: RTX 4090 GPU passthru, esxi 8.0

Memnarch — Wed, 09 Nov 2022 15:10:25 GMT

So, after considerable debugging...

TL;DR

I think it's a bug in how esxi releases the console-claimed GPU from the console to a VM, and there is a workaround.

Turn off boot display with

esxcli system settings kernel set -s vga -v FALSE (Can undo with: esxcli system settings kernel set -s vga -v TRUE)

VM is now stable including on first boot.

Long version:

Esxi had a "bug" in 7.0 where the display output claimed for the hypervisor console itself would have to be manually re-enabled for passthru with every reboot (in 6.7, this wasn't needed). This was reportedly later changed back to the 6.7 behavior. The above command line stops esxi from claiming any GPU for the console at all, and during the initial phases of 7.0 therefore removed the need to manual re-enable one gpu for passthru on every boot. The need for this command line was subsequently reportedly removed when the behavior was changed back to 6.7-style behavior, where passthru remained enabled on reboot and the VM would just take over the GPU automatically on its first boot.

Using this command line appears to totally fix the problems with the VM spontaneously combusting on first boot after host boot described in the initial post, and nothing else I tried does. It's probably not a coincidence that the 4090 is in the primary graphics slot and esxi does in fact claim it for the console unless I force it not to. (I never had to re-enable passthrough on boot, so didn't see any need for this command line until testing to see if it solved the problematic behavior.)

Source: https://williamlam.com/2020/06/passthrough-of-integrated-gpu-igpu-for-standard-intel-nuc.html

Beware-- the setting persists across boots. It is reversible, but you can run into trouble if your host loses network connectivity for some reason-- you will have no console AND no network access and this usually requires reinstalling esxi to fix. In particular, the combination of "no console" and an off-HCL network driver is a very dicey idea, since those often require resetting network configurations and host services for minor changes. You have been warned.

RTX 4090 GPU passthru, esxi 8.0

Memnarch — Wed, 02 Nov 2022 00:18:58 GMT

I'm working on passing an RTX 4090 GPU to a VM on an Intel 13900k system.

Things I did first-- enabled Vt-d, > 4G addressing in BIOS (no ACS option , appeared enabled by default.)

No problem turning on passthru and assigning it to a VM. Also passthru some NVME drives and a renesas USB controller which all seem to work. efi firmware mode. Windows 11. 48 GB RAM (out of 96 on the host.) 8 cores (out of 24).

VM won't poweron without Use64bitMMIO of at least 64 GB (as expected for a 24GB card.)

VM then powers on and works in windows. BUT, the first time after each host boot, the VM will spontaneously die (power off) within a few seconds of getting to the login screen. The VM log gives a message "attempted to map 65000 pages to host memory" and recommends setting a pciHole.start to 1536.

If I do that, the VM doesn't poweroff as above, BUT the windows OS inside of it dies at the same time and the system reboots (succesfully).

With or without pciHole.start, the VM can the be restarted and seems to work fine. I have seen it blow up spontaenously once in a few days of testing, otherwise rock solid. Only the first powerup after booting the physical host seems affected.

If I turn "resize BAR" off in the BIOS, the pciHole address changes but otherwise as above. reBAR works in the VM OS if enabled in BIOS.

Several other VM's on the same host , using 2070 GPU's, don't show this behavior (and also don't require 64bitMMIO.)

The same windows OS boots and 4090 on the same hardware (without esxi) and works fine.

Any ideas?

Thanks for thinking about it.

Re: ESXi 8.0 unable to add hard disk to VM

Memnarch — Sun, 30 Oct 2022 16:21:28 GMT

Another workaround I found -- create a new independent persistent disk, unregister the VM, manually edit the vmx file to replace the vmdk filename with that of the existing disk that you actually want, then re-register the VM. It's a PITA but doesn't require reinstalling esxi.

Re: ESXi 8.0 unable to add hard disk to VM

Memnarch — Sun, 30 Oct 2022 15:19:05 GMT

Same problem. Even if a create a new hard disk in 8.0 it won't attach. Existing Vm's from 7.0 with existing disks work fine, but if you delete the disk from the vm entry you can't re-add it.

Re: Threadripper 3970x GPU and USB Passthrough, esxi 6.7 U3, nvidia RTX 2080, TRX40

Memnarch — Wed, 23 Sep 2020 23:07:50 GMT

I think I've found the answer,and it looks like an issue that has nothing to do with vmware. I've found multiple other examples of non-virtualized users having same problem.

Proposed solutions:

https://www.nvidia.com/en-us/geforce/forums/geforce-graphics-cards/5/332481/type-c-port-on-my-2080ti-disconnects/

"TLDR: Make sure your PCI Express Link State Power Management savings mode is set to off! After changing this all my issues went away switching also by switching from Balanced to High Performance profile which changes this setting for you. https://www.sevenforums.com/tutorials/292971-pcie-link-state-power-management-turn-off-windows.html

Additionally just to ensure this I also set NVIDIA's root USB controller to uncheck "allow the computer to turn off this device to save power". I also moved the device to a powered USB hub. https://felixwong.com/2015/03/solved-microsoft-wired-keyboard-200-disconnects-and-makes-usb-connection-sound/ "

Neither of the above actually solved the issue for me. But the following did:

Nvidia control panel, Manage 3D settings, power management mode --> prefer maximum performance.

Presto, stutter gone. Including choices 1 and 2 because they seemed to help many other people with similar issue.

With this fix, the RTX gpu's are very convenient-- each VM has its own USB-c controller built in.

Another (no longer needed) workaround: https://www.virtuallyghetto.com/2020/05/how-to-passthrough-usb-keyboard-mouse-hid-and-ccid-devices-to-vm-in-esxi.html

https://www.virtuallyghetto.com/2020/05/how-to-passthrough-usb-keyboard-mouse-hid-and-ccid-devices-to-vm-in-esxi.html

but see all the comments to that article for the "fine print" that actually makes it work. I found that answer worked for mouse and keyboard but passing through a USB audio controller gave me massive lag. Can do audio over HDMI from the gpu instead though. I much prefer passing through a whole USB controller, much simpler to do.

Re: Threadripper 3970x GPU and USB Passthrough, esxi 6.7 U3, nvidia RTX 2080, TRX40

Memnarch — Tue, 28 Jan 2020 19:16:37 GMT

Some additional data--

Disabling XMP, updating BIOS to the latest beta version, shutting down all other vm's and changing the test vm to High latency sensitivity, pinned to a single ccx (4 cores) with full CPU reservation, and deleting all unused virtual devices on the VM did not fix (intermittent nvidia USB device freezing) problem.

No windows error messages when mouse freezes, device manager shows no errors.

No corresponding errors in VMware logs.

Still stumped.

Threadripper 3970x GPU and USB Passthrough, esxi 6.7 U3, nvidia RTX 2080, TRX40

Memnarch — Mon, 27 Jan 2020 17:48:30 GMT

Hi all-- wanted to describe progress on an update to my former threadripper system.

Starting point: 4 Vm's on a threadripper 1950, each with GPU passthrough (1 x 2080, 3 x 2070). 64 GB RAM, esxi 6.7 U3. System was quite stable (see prior thread).

Target: Threadripper 3970x (double the cores), 128 GB RAM, on an Asrock Creator TRX40 motherboard.

I started by validating the new hardware under a temporary (non-virtualized) windows build. Stuff worked.

BIOS settings used: defaults, except:

Changed some fan settings to make them quieter

Turned on XMP

Turned on SR-IOV

left PBO OFF (default, but I changed it to disabled. PBO sucks up huge amounts of power for little performance benefit, to say nothing of validation!)

Used current BIOS version, not the beta for 3990x.

esxi installation: Used previous installation. This had passthru.map entries for AMD and NVIDIA as detailed in my last post. It also had the epyc-recommended configuration change previously recommended (which I removed) and preinstalled aquantia NIC driver.

Moved 2 x m.2 SSd's from old into new system. System booted nicely into esxi. All hardware passthrough vanished as expected. Of note, *neither* of the NIC's on this board has a native driver. I used the aquantia driver and live off the 10Gb aquantia. I have no idea if there is a realtek Dragon 2.5g driver out there.

Redid the hardware passthrough. All GPU's passed back through to their VM's, yay. Only two of them would boot, boo. Eventually, after much gnashing of teeth, remade three VM's from scratch: they would all keep crashing immediately upon booting windows.

This was interesting. Esxi would report that the GPU's had violated memory access and advised adding a passthrough.map entry, which didn't fix the problem. Changing BIOS on the host to remove CSM support and enable 4G support, and enabling 64 bit MMIO in the vm didn't fix it either. A new vm with fresh windows install worked.

There were several other interesting changes from previous system:

disabling msi on the gpu made them keep crashing , unlike previously when it fixed stuttering :smileyalert:

No cpu pinning or NUMA settings were used or needed

The mystical cpuid.hypervisor setting remains required to avoid error 43

With these caveats, I got 4 bootable VM's each using the Nvidia card's own USB-c connector for keyboard/mouse. 8 cpus/vm. Which led to the next problem, which I haven't been able to solve:

The mice/keyboards would all intermittently freeze for moments to minutes, and sometimes not come back. Lots of testing inside of windows showed no cause. Interestingly, the problem was 1)worse with a highend G502 mouse, and 2) much worse inside of windows UI -- and never happened for example in demanding real time full screen apps. I was sure it was going to be some bizarre windows problem. Rarely (every few hours) systems would crash completely (while idle!) with the same memory access violation. Also rebooting one of the vm's would make other vm's momentarily stutter. None of this ever happened on the 1950x system where these controllers were reliable.

I eventually worked around the problem with the motherboard's USB controllers. There are 5: 2 x Matisse, 2 x Starship, and 1 x asmedia. The Matisse ones are lumped in the same IOMMU group and won't pass through (They are perpetually "reboot needed.") The Asmedia chip worked with no problems (usb-c port on the back of the motherboard). The Starship USB 3.0 controllers both worked IF you had a passthru.map entry moving them to d3d0 reset method. Otherwise, booting a VM with one of these controllers failed AND crashed a different VM with a GPU memory access violation :smileyalert: and the controller then permanently disappeared until the system was then powered down (not just a reboot). Wow, talk about bad crashes.

Using these 3 motherboard controllers an 3 vms appears rock stable (I haven't tested the fourth yet.) One of them has 64but mmo enabled which probably isn't needed.

Things I haven't gotten around to testing yet:

1. Does isolating the vm to one ccx fix anything?

2. If only one vm is running, does the usb-c nvidia controller become reliable?

3. Does turning off XMP or using the latest beta BIOS change anything?

Other advice -- I'm obviously waaay off of HCL here-- but don't even try DRAMless SSD's. Datastore *vanishes* under high load. Bad. Same thing happens with my OEM samsung until I updated the firmware, but that's another story well documented elsewhere.

I'm really puzzled by the nvidia usb-c thing. Would also be nice if the Matisse controllers worked. Otherwise mostly pleased-- many of the kludges needed on older esxi versions and the 1950x with its wacky NUMA configuration are no longer needed and the new system is *much* faster.

Hope this helps someone else. If anyone can tell me what's going on (or at least that it's not just me) would be much appreciated. I speculate it's a BIOS bug.

Thanks LT

Re: CPU scheduler "automatic relation removal" causing vm freeze, esxi 6.7U2, threadripper, gpu

Memnarch — Fri, 05 Jul 2019 16:00:26 GMT

Hi all-- in case anyone else ever runs into this--

the VM in question had bad USB driver that was causing it to hang. The vmware warnings above happened on rebooting the hanged vm, not and the crash itself.

Re: Threadripper ESXi 6.7 GPU and USB passthrough, experience progress and problems

Memnarch — Sun, 30 Jun 2019 12:07:42 GMT

Hi all--- a few more notes.

tl;dr Learn esxtop. Assigning more resources to vm's can result in lowered performance.

I *finally* pinned down what seemed to be the last "stutter" problem. To put it briefly, running 4 vm's with 8 vcpu's each on a 16core /32 thread CPU was a bad idea. Under load, I discovered up to 50% READY state in esxtop which corresponded to massive stutter (should be < 5%; this is a measure of how often the vm was ready to execute but forced to wait due to lack of host resources). I'm theorizing that the win10 vm's didn't realize (weren't told) that those "8 vcpus" were really "4 cores/ 8 thread." REDUCING the number of vcpu's on each vm from 8 down to 4 massively IMPROVED performance, with ready state < 0.5%.. It also fixed the latencymon testing (now the vm's pass, yay!).

vm's with GPU's all have RAM reserved. If you're running in NUMA mode, you probably want all that RAM in one NUMA domain. The order in which you poweron your vm's, and how much ram they have, can affect esxi's ability to do this. Again, more can be less. I haven't investigated UMA vs. NUMA comprehensively.

Moved back to the asrock x399 Professional Gaming MB, works. Again, only 2 MB USB pass through successfully. There seems to be a non-vmware driver available for the 10G NIC (aquantia).

All vm's moved to 6.5 NVME controllers successfully.

CPU scheduler "automatic relation removal" causing vm freeze, esxi 6.7U2, threadripper, gpu

Memnarch — Fri, 28 Jun 2019 14:58:44 GMT

Interesting problem, I can't find another example of it listed anywhere.

4 win 10 vm's on esxi 6.7U2 host, each with a GPU pass through and therefore reserved memory. NUMA enabled in BIOS. 8 vcpu's per vm on a 16 core threadripper 1950x.

Each vm gets occasional (every few hours of use) freezes where the running application becomes unresponsive. Task manager is still accessible but you can't actually do anything from it. If the vm is power cycled from the host it reboots normally. Host does not freeze or crash and other vm's unaffected.

The host error log always gives something like this:

2019-06-26T23:51:42.491Z cpu26:2097396)WARNING: CpuSched: 996: Automatic relation removal from 2100712(vmx-vcpu-0:VM3, VM1) to 2100713(PVSCSI-2100577:1)

2019-06-26T23:51:51.491Z cpu22:2097396)WARNING: CpuSched: 996: Automatic relation removal from 2100566(vmx-vcpu-0:VM2, VM1) to 2100567(LSI-2100430:0)

where "VM1" is the name of a vm.

VM1 is always involved, others may or may not be; and the second listing is always PVSCI or LSI (which are virtual SCSI adapters.) The freezes can happen on ANY of the 4 vm's, but vm1 is the least stable. Freezes only occur when the VM (that freezes) is under load, and not at idle. VM1 load doesn't seem to freeze the other vm's, only itself, but with freezes this intermittent it's hard to be sure of.

I haven't changed the default vm cpu scheduler. Latest BIOS / AGESA.

Since these warnings always accompany a freeze I assume they are related, but I can't find any documentation of what they mean let alone how to resolve them.

Anyone have a clue?

Thanks,

Re: Threadripper ESXi 6.7 GPU and USB passthrough, experience progress and problems

Memnarch — Mon, 15 Apr 2019 15:08:37 GMT

Hi all! OP here, a few more notes that I don't think I included above--

pciPassthru0.msiEnabled (where 0 is replaced with whichever device is your GPU) must be set to FALSE in advanced vm settings for the GPU closest to the CPU, otherwise you get constant crashing of the video driver (often with immediate restart and no bluescreen, looks like a slow computer with a screen that occasionally blinks). Beware, because if you change or delete the device and then re-add, you need to redo this.

Very important post on changing the esxi host configuration to prevent random core memory hopping. This was on reddit and made a substantial difference, reducing or eliminating microstutter on all the VM's. I also no longer needed to pin them to specific ccx's. This is in the reddit post titled AMD EPYC on ESXi 6.5-6.7 NUMA issues: Mostly Resolved

Using 6.7 U1 now, can use EFI in VM's instead of BIOS (used to hang if USB passhtru).

Ran into a problem where any VM would slam to 100% disk usage under heavy load (like say a file download) and become almost unresponsive for > 10 minutes after download stopped. Eventually tracked it down to meta corruption on the underlying VMFS. VOMA doesn't like vmfs 6 (didn't last week) so I migrated to a new vmfs, problem solved.

Other notes:

The motherboard appears to retrain its BIOS (and renumber PCI devices ) if you take any out or put any in, potentially breaking more vm's until you re-activate for passthru. You also need to reset lots of bios settings.

Very interestingly, I discovered my system overheating from a dying Enermax Liqtech TR4 AIO. This caused really "interesting" features like the system abruptly (failing to POST) after changing memory settings, even though it had been working fine a minute before. Even worse, if it DID memory training when hot, it would set to super low speeds. This sounds easy, but with not way to check CPU temps outside of BIOS, it was slightly less straightforward (anyone know another way to check CPU temp under esxi with a threadripper? or any of the vm's?)

Asmedia / X399 xHCI 3.1 USB passthrough on Asus Zenith Extreme (Threadripper 1950x) Esxi 6.7 U1

Memnarch — Sun, 17 Mar 2019 21:05:16 GMT

Attempting to passthrough USB controllers to VM's under esxi 6.7 U1. These are built into an Asus Zenith Extreme motherboard with latest BIOS. Two of the controllers (AMD 3.0 ) work fine.

One of them is an Asmedia device with info copied at the end: This perpetually needs a reboot when I attempt to activate passthrough. Disabling the ACS check in esxi.conf results in a nonbootable host, and in any event ACS is enabled in BIOS. Since it never becomes active for passthrough, I can't even attempt to pass it through to a vm.

The other is an "...X399 Series Chipset USB 3.1 xHCI controller." Unlike the Asmedia device, this can be made active for passthrough-- BUT if I attempt to boot a VM (version 14) with this, it fails with the error message below. This is unrelated to the previous "need to have a BIOS mode VM" problem ( that was fixed with U1, and DID poweron even when buggy-- just wouldn't boot windows.) I can't power the VM at all.

Anyone have any ideas on either of these bugs?

Thanks!

ERROR MESSAGE WITH X399 USB 3.1 FOLLOWS:

Power On VM

Key

haTask-5-vim.VirtualMachine.powerOn-1558496026

Description

Power On this virtual machine

Virtual machine

EfiBoot

State

Failed - Module 'DevicePowerOn' power on failed.

Errors

Module 'DevicePowerOn' power on failed.
Device 1:0.0 is not a passthrough device.
Failed to start the virtual machine.

ASMEDIA DESCRIPTION FOLLOWS:

<class> USB controller

Re: Threadripper ESXi 6.7 GPU and USB passthrough, experience progress and problems

Memnarch — Tue, 09 Oct 2018 15:11:26 GMT

Hi--

I did start this thread on the x399 prof, but I switched to the Asus Zenith Extreme partway (as above). My recollection from the x399 is that the aquantia did show up on the hardware list but I never did much with it, because I have no 10G switch nor anything else with a 10G port.

I'm only passing through whole controllers at this point. I'd like to have 1 controller for each of the 4 vm's with GPU's, and there are 4 usb controllers on the the Zenith. 2 of the 3 AMD controllers work; the AMD 3.1 controller won't allow boot when passed through and the ASMEDIA is stuck in perpetual "activated needs reboot": no matter how many times I reboot-- so I only have 2 useful passthrough controllers for 4 vm's. I'm using software to pass individual devices from one of the vm's to the other 2 but this is not idea.

Re: Threadripper ESXi 6.7 GPU and USB passthrough, experience progress and problems

Memnarch — Mon, 08 Oct 2018 19:24:50 GMT

The 1gb nic on this board works fine. Never tried to pass it through.

i dont have the included Aquantia Pcie board so I couldn’t tell you.

using 6.7.

Im embarrassed to say that a large chunk of problems disappeared with a cmos clear, probably had bad ram training. Freezes and crashes have cleared up. Also , strangely , turning memory interleaving back to auto DOUBLED my applixation Performance (remember I’ve got my vms pinned to one ccx each).

System works very nicely now. Would still like to figure out how to pas through the 3.1 usb controller (VM won’t boot , “not a passthrough device ”) and the. Asmedia usb controller (perpetually needs a reboot to enable passthrough).

Re: Threadripper ESXi 6.7 GPU and USB passthrough, experience progress and problems

Memnarch — Wed, 03 Oct 2018 19:20:19 GMT

Further notes--

The dreaded "error 43" after random lockups did NOT go away switching MB to an Asus zenith extreme. However, after that board was updated to the latest bios (for the new threadrippers, although I'm using a 1950x) I have gotten zero error 43: random lockups still happen but the VM is rebootable from the host.

I am (still) getting random audio freezes periodically on all the vm's, which occasional seem to result in these crashes. The freezes became much less frequent (but did not resolve entirely) when setting each VM to prefer CPU cores from a single die on the threadripper (eg, affinity for core 0-7). Still having occasional random crashes.

Disabling SMT in BIOS strangely did not work :smileyalert:---esxi still reports 32 cores with hyperthreading active. STrange. WIll try disabling hyperthreading in esxi itself, but would have thought the bios would be the definitive way to do that.

Re: Threadripper ESXi 6.7 GPU and USB passthrough, experience progress and problems

Memnarch — Fri, 06 Jul 2018 22:52:44 GMT

4 GPU passthrough devices, 2 USB controllers, 1 audio device, split amongs 4 VM's.

Using only motherboard USB now. Seems to work with d3d0; would you suggest changing it to something else?

Also, I'm getting random VM lockups when under GPU load (not often, on the order of once in a few hours.) VM locks hard, host is OK, but rebooting the vm puts it into Error 43 with no gpu: need to reboot the host to fix this. The VM with the GPU furthest on the motherboard from the CPU is most often affected. Don't know if this is a pciHole, msiEnable, or agesa problem... grrr.

I looked into the log files when rebooting the VM (into error 43). "Bad" reboots have this at the end: (truncated). There's some earlier stuff about vmware tools not loading either. Anyone know that this stuff means? (This text doesn't appear on normal boots, only after a lockup that yields error 43 on VM reboot).

Thanks LT

2018-07-05T21:49:31.734Z| vcpu-2| I125: Guest MSR write (0x49: 0x1)

2018-07-05T21:49:31.734Z| vcpu-4| I125: Guest MSR write (0x49: 0x1)

2018-07-05T21:49:31.734Z| vcpu-6| I125: Guest MSR write (0x49: 0x1)

2018-07-05T21:49:31.734Z| vcpu-3| I125: Guest MSR write (0x49: 0x1)

2018-07-05T21:49:31.734Z| vcpu-0| I125: Guest MSR write (0x49: 0x1)

2018-07-05T21:49:31.734Z| vcpu-5| I125: Guest MSR write (0x49: 0x1)

2018-07-05T21:49:32.836Z| svga| I125: SVGA disabling SVGA

2018-07-05T21:49:33.277Z| vcpu-7| I125: LSI: Invalid PageType [21] pageNo 0 Action 0

2018-07-05T21:49:43.510Z| vcpu-0| I125: Destroying virtual dev for scsi0:0 vscsi=8197

2018-07-05T21:49:43.510Z| vcpu-0| I125: VMMon_VSCSIStopVports: No such target on adapter

2018-07-05T21:49:43.511Z| vcpu-0| I125: DEVICE: Resetting device 'ALL'.

2018-07-05T21:49:43.511Z| vcpu-0| I125: USB: Per-Device Resetting device 0x200000050e0f0003

2018-07-05T21:49:43.511Z| vcpu-0| I125: Tools: ToolsRunningStatus_Reset, delayedRequest is 0x91DA8A4270

2018-07-05T21:49:43.511Z| vcpu-0| I125: Tools: Changing running status: 1 => 0.

2018-07-05T21:49:43.511Z| vcpu-0| I125: GuestLib Generated SessionId 11821741703570650877

2018-07-05T21:49:43.511Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 3527 us

2018-07-05T21:49:43.511Z| vcpu-0| I125: CPU reset: hard (mode 0)

2018-07-05T21:49:43.512Z| vcpu-1| I125: CPU reset: hard (mode 0)

2018-07-05T21:49:43.512Z| vcpu-2| I125: CPU reset: hard (mode 0)

2018-07-05T21:49:43.512Z| vcpu-3| I125: CPU reset: hard (mode 0)

2018-07-05T21:49:43.512Z| vcpu-7| I125: CPU reset: hard (mode 0)

2018-07-05T21:49:43.512Z| vcpu-6| I125: CPU reset: hard (mode 0)

2018-07-05T21:49:43.512Z| vcpu-5| I125: CPU reset: hard (mode 0)

2018-07-05T21:49:43.512Z| vcpu-4| I125: CPU reset: hard (mode 0)

2018-07-05T21:49:43.518Z| vcpu-0| I125: SVGA: Unregistering IOSpace at 0x1070

2018-07-05T21:49:43.518Z| vcpu-0| I125: SVGA: Unregistering MemSpace at 0xe8000000(0xe8000000) and 0xfe000000(0xfe000000)

2018-07-05T21:49:43.519Z| vcpu-0| I125: SCSI: switching scsi0 to push completion mode

2018-07-05T21:49:43.519Z| vcpu-0| I125: Creating virtual dev for 'scsi0:0'.

2018-07-05T21:49:43.519Z| vcpu-0| I125: DumpDiskInfo: scsi0:0 createType=11, capacity = 536870912, numLinks = 1, allocationType = 2

2018-07-05T21:49:43.519Z| vcpu-0| I125: SCSIDiskESXPopulateVDevDesc: Using FS backend

2018-07-05T21:49:43.519Z| vcpu-0| I125: DISKUTIL: scsi0:0 : geometry=33418/255/63

2018-07-05T21:49:43.519Z| vcpu-0| I125: SCSIFilterESXAttachCBRCInt: CBRC not enabled or opened without filters,skipping CBRC filter attach.

2018-07-05T21:49:43.519Z| vcpu-0| I125: SCSIFilterSBDAttachCBRC: device scsi0:0 is not SBD. Skipping CBRC attach SBD way.

2018-07-05T21:49:43.567Z| vcpu-0| I125: PCIXHCI: Interrupt type changed from MSIX to INTX

2018-07-05T21:49:43.578Z| vcpu-0| I125: PCIBridge4: ISA/VGA decoding enabled (ctrl 0004)

2018-07-05T21:49:43.578Z| vcpu-0| I125: pciBridge4:1: ISA/VGA decoding enabled (ctrl 0004)

Re: Threadripper ESXi 6.7 GPU and USB passthrough, experience progress and problems

Memnarch — Mon, 25 Jun 2018 21:27:51 GMT

And yet further progress--

The 1070 Ti with the error 43 was also dead in a single board system (board problem).

Was able to passthrough Asus USB Bluetooth dongles as USB devices (not as PCI devices; done through a non-passthrough USB controller-- the very same 3.1 controller that can't pass through anyway) to VM's, which could then attach Bluetooth keyboards and mice. However, I'm going to stick with Virtualhere because I'd rather not deal with the input lag and batteries, and can live with making some vm's dependent on others.

Passthrough of the onboard audio worked fine despite not passing through other items in the IOMMU grouping (like a SATA controller). I've never actually tested or used the sata controllers, NVME works fine.

The on-board Bluetooth was eventually discovered appearing as a USB device and could be passed through ("Intel USB device") but not actually used in the VM (recognized as Bluetooth controller but had an error, "wouldn't start.")

Re: Threadripper ESXi 6.7 GPU and USB passthrough, experience progress and problems

Memnarch — Thu, 21 Jun 2018 22:44:10 GMT

Further notes:

The Aquantia 10GB controller, the Intel wireless controller, an add-in PCIE 3.0 board in the PCIE x1 slot, and the USB 3.1 controller (as compared to the 2 x USB 3.0 controllers, which do passthrough) all won't passthrough-- most of these perpetually "need reboot". I've read this can be caused by being behind a non-ACS compatible PCIE switch. Setting the parameter to disable ACS check to TRUE results in a non-bootable host. All BIOS parameters that I can identify (including "enable ACS") are enabled, and I am using the beta bios with AGESA 1.0.0.6. Stumped, although none of this is absolutely essential to me. Another possibility (at least for the 3.1 controller) is that it seems to be in the same IOMMU group as a PCIE bridge that can't be enabled for passthrough (greyed out.) The 3.1 controller is slightly different, VM won't power up with a message that a device isn't passthrough capable.

of 4 GPU's in the system (1080 FE, 2 x 1070Ti FE, 1 x 1060 3Gb), all can be passed through. One of the 1070 Ti's gives the dreaded error 43, even with the hypervisor.cpuid flag set to FALSE. The others work very nicely.

I can try replacing one of the video boards with an AMD, and using software USB passthrough from the two working controllers, but it would be really nice not to have to do that (passing through USB between VM's means that VM's become dependent on eachother for their keyboards to work). And the error 43 just gets my goat because it is so poorly documented on the NVidia side.

Will post further updates, would appreciate any advice. Thanks