topic Re: Unable to start a VM with Nvidia A40 vGPU profile in Horizon Desktops and Apps

Unable to start a VM with Nvidia A40 vGPU profile

tomtomek — Wed, 23 Mar 2022 15:47:35 GMT

Hi all,

Until yesterday I had a fully running test VMware Horizon Environment based on Dell R740 host with Nvidia A40, running on ESXI 7.02U1a and vSphere 7. Since I was seeing an issue of expanding the Full Clone VMs hard disks, with the below error:
The disk extend operation failed: A digest operation has failed.
I decided to update the test environment. ESXI host was updated to the latest 7.02U2e, vSphere updated to 7.0.3.00300. I had to also update the Nvidia VIB since this was not seeing the profiles. The latest VIB of the following flavor has been installed. VMware_ESXi_7.0.2_Driver_510.47.03-1OEM.702.0.0.17630552.vib.

Issue:
When powering on a VM with added Nvidia card it fails with the below generic message. If the card is removed, the machine starts fine.
Error: Disconnected from virtual machine. Remote connection failure Failed to establish transport connection.

Any suggestions on how to resolve this would be much appreciated.

Regards,
Tom

Re: Unable to start a VM with Nvidia A40 vGPU profile

fabio1975 — Wed, 23 Mar 2022 16:45:55 GMT

Ciao

can you send me the screenshot of the VM configuration (edit...) where I can see the detail of the vGPU settings?

Re: Unable to start a VM with Nvidia A40 vGPU profile

tomtomek — Wed, 23 Mar 2022 17:18:31 GMT

Hey Fabio,

Thanks for your response! You helped me last year when I was playing with this environment

Just to note, I have updated to 7.03C

Requested details below:

Regards,

Tom

Re: Unable to start a VM with Nvidia A40 vGPU profile

fabio1975 — Wed, 23 Mar 2022 18:13:59 GMT

Ciao

Can you send me the vmware.log file that you should find in the directory with the VM name in the datastore where the VM is located?
The file contains the VM startup logs.

This is an example:

Re: Unable to start a VM with Nvidia A40 vGPU profile

tomtomek — Wed, 23 Mar 2022 19:04:11 GMT

Hi Fabio,

Can I upload it somewhere safe for you to download? Do not want to post full log, unless I go in and edit some IPs etc.

Last portion of log where it seems to fail is below, can send the full log if needs be.

2022-03-23T18:58:16.368Z In(05) vmx - VMIOP: Loaded Vmx plugin APIs
2022-03-23T18:58:16.368Z In(05) vmx - VMIOP: Loaded plugin libvmx-vmiop.so:VMIOP_InitModule
2022-03-23T18:58:16.368Z In(05) vmx - DeviceIfAlloc: setting up Async IO stuff.
2022-03-23T18:58:16.369Z In(05) vthread-2158390 - VTHREAD 350971553536 "vthread-2158390" wid 2158390
2022-03-23T18:58:16.369Z In(05) vmx - VMIOP: config /usr/share/nvidia/vgx/nvidia_a40-8q.conf
2022-03-23T18:58:16.370Z In(05) vmx - VMIOP: Using VER_4_0 symbol /usr/lib64/vmware/plugin/libnvidia-vgx.so:vmiop_display_vmiop_plugin
2022-03-23T18:58:16.370Z No(00) vmx - ConfigDB: Unsetting "vmiop.guestVgpuVersion"
2022-03-23T18:58:17.394Z In(05) vmx - VMIOP: Registered VF device 0000:3b:00.5
2022-03-23T18:58:17.439Z No(00) vmx - ConfigDB: Setting pciPassthru0.pgpu = "2235145A0606060606060600000003"
2022-03-23T18:58:17.439Z In(05) vmx - VMIOP: Enabling checkpoint support
2022-03-23T18:58:17.440Z In(05) vmx - VMIOP: Initializing plugin vmiop-display
2022-03-23T18:58:17.440Z In(05) vmx - vmiop_log: (0x0): detected a VF at 0:3b:0.5
2022-03-23T18:58:17.442Z In(05) vmx - vmiop_log: (0x0): gpu-pci-id : 0x3b00
2022-03-23T18:58:17.442Z In(05) vmx - vmiop_log: (0x0): vgpu_type : Quadro
2022-03-23T18:58:17.442Z In(05) vmx - vmiop_log: (0x0): Framebuffer: 0x1dc000000
2022-03-23T18:58:17.442Z In(05) vmx - vmiop_log: (0x0): Virtual Device Id: 0x2235:0x14dc
2022-03-23T18:58:17.442Z In(05) vmx - vmiop_log: (0x0): FRL Value: 60 FPS
2022-03-23T18:58:17.442Z In(05) vmx - vmiop_log: ######## vGPU Manager Information: ########
2022-03-23T18:58:17.442Z In(05) vmx - vmiop_log: Driver Version: 510.47.03
2022-03-23T18:58:17.444Z In(05) vmx - vmiop_log: (0x0): BAR0 has regions with different mapping types!!
2022-03-23T18:58:17.444Z In(05) vmx - vmiop_log: (0x0): vGPU supported range: (0x70001, 0xd0001)
2022-03-23T18:58:17.444Z In(05) vmx - vmiop_log: (0x0): Detected ECC enabled on physical GPU.
2022-03-23T18:58:17.444Z In(05) vmx - vmiop_log: (0x0): Guest usable FB size is reduced due to ECC.
2022-03-23T18:58:17.450Z Er(02) vmx - vmiop_log: NVOS status 0x1f
2022-03-23T18:58:17.450Z Er(02) vmx - vmiop_log: Assertion Failed at 0xb79ecb36:251
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: 17 frames returned by backtrace
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(_nv007173vgpu+0x35) [0x51b79727e5]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(_nv002992vgpu+0xe0) [0x51b79eba90]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(_nv002990vgpu+0x96) [0x51b79ecb36]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(_nv003007vgpu+0x54) [0x51b79eccd4]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(+0xb3554) [0x51b7942554]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(+0xb4c0e) [0x51b7943c0e]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libvmx-vmiop.so(+0x9234) [0x51b7486234]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x3d4874) [0x516f695874]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x2eda84) [0x516f5aea84]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x2ed5d4) [0x516f5ae5d4]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x2ee74b) [0x516f5af74b]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x2f946b) [0x516f5ba46b]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x26a0a5) [0x516f52b0a5]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x26a8d2) [0x516f52b8d2]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x25e731) [0x516f51f731]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /lib64/libc.so.6(__libc_start_main+0xed) [0x51b29f9c8d]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x25f105) [0x516f520105]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: NVOS status 0x1f
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: Assertion Failed at 0xb79ecb36:251
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: 17 frames returned by backtrace
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(_nv007173vgpu+0x35) [0x51b79727e5]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(_nv002992vgpu+0xe0) [0x51b79eba90]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(_nv002990vgpu+0x96) [0x51b79ecb36]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(_nv003007vgpu+0x74) [0x51b79eccf4]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(+0xb3554) [0x51b7942554]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(+0xb4c0e) [0x51b7943c0e]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libvmx-vmiop.so(+0x9234) [0x51b7486234]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x3d4874) [0x516f695874]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x2eda84) [0x516f5aea84]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x2ed5d4) [0x516f5ae5d4]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x2ee74b) [0x516f5af74b]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x2f946b) [0x516f5ba46b]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x26a0a5) [0x516f52b0a5]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x26a8d2) [0x516f52b8d2]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x25e731) [0x516f51f731]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /lib64/libc.so.6(__libc_start_main+0xed) [0x51b29f9c8d]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: /bin/vmx(+0x25f105) [0x516f520105]
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: NVOS status 0x1f
2022-03-23T18:58:17.451Z Er(02) vmx - vmiop_log: Assertion Failed at 0xb79ecb36:251
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: 17 frames returned by backtrace
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(_nv007173vgpu+0x35) [0x51b79727e5]
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(_nv002992vgpu+0xe0) [0x51b79eba90]
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(_nv002990vgpu+0x96) [0x51b79ecb36]
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(_nv003007vgpu+0x94) [0x51b79ecd14]
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(+0xb3554) [0x51b7942554]
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(+0xb4c0e) [0x51b7943c0e]
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libvmx-vmiop.so(+0x9234) [0x51b7486234]
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: /bin/vmx(+0x3d4874) [0x516f695874]
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: /bin/vmx(+0x2eda84) [0x516f5aea84]
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: /bin/vmx(+0x2ed5d4) [0x516f5ae5d4]
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: /bin/vmx(+0x2ee74b) [0x516f5af74b]
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: /bin/vmx(+0x2f946b) [0x516f5ba46b]
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: /bin/vmx(+0x26a0a5) [0x516f52b0a5]
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: /bin/vmx(+0x26a8d2) [0x516f52b8d2]
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: /bin/vmx(+0x25e731) [0x516f51f731]
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: /lib64/libc.so.6(__libc_start_main+0xed) [0x51b29f9c8d]
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: /bin/vmx(+0x25f105) [0x516f520105]
2022-03-23T18:58:17.452Z Cr(01) vmx - PANIC: VMIOP: Failed to find MSI-X capability for VF device.
2022-03-23T18:58:17.452Z In(05) vmx - VMIOP: informing the plugin vmiop-display of Panic
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: (0x0): Thread for engine 0x0 could not join with error 0x1
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: (0x0): Thread for engine 0x4 could not join with error 0x1
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: (0x0): Failed to free thread event for engine 0x4. Error: 0x1
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: (0x0): Thread for engine 0x5 could not join with error 0x1
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: (0x0): Failed to free thread event for engine 0x5. Error: 0x1
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: (0x0): Thread for engine 0x6 could not join with error 0x1
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: (0x0): Failed to free thread event for engine 0x6. Error: 0x1
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: (0x0): Failed to free edid_arrived_event. Error: 0x1
2022-03-23T18:58:17.452Z Er(02) vmx - vmiop_log: (0x0): Staging Buffer Not Allocated
2022-03-23T18:58:17.458Z Er(02) vmx - vmiop_log: (0x0): BAR1 VA heap is not completely empty (0x1 still used)
2022-03-23T18:58:18.005Z Wa(03) vmx - A core file is available in "/var/core/vmx-zdump.000"
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace:
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[0] 000003494da9ee80 rip=000000516f52b769 rbx=000003494da9eea0 rbp=000003494da9f380 r12=00000051707bb080 r13=0000000000000000 r14=0000005171020f10 r15=00000051b792bb60
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[1] 000003494da9f390 rip=000000516f698136 rbx=000000517101b750 rbp=000003494da9f3d0 r12=0000005171047ec8 r13=0000000000001000 r14=0000005171020f10 r15=00000051b792bb60
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[2] 000003494da9f3e0 rip=00000051b74885e4 rbx=00000051b792bb60 rbp=000003494da9f430 r12=0000005171047ec8 r13=0000005171020f10 r14=0000000000001000 r15=00000051b768d0b0
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[3] 000003494da9f440 rip=00000051b793459b rbx=000003494da9f580 rbp=000003494da9f450 r12=0000005171020f10 r13=0000000000000000 r14=00000051b7d0e390 r15=0000005171027198
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[4] 000003494da9f460 rip=00000051b79425d1 rbx=000003494da9f580 rbp=000003494da9f550 r12=0000005171020f10 r13=0000000000000000 r14=00000051b7d0e390 r15=0000005171027198
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[5] 000003494da9f560 rip=00000051b7943c0e rbx=0000005170feae00 rbp=000003494da9f600 r12=00000051b7a9dd81 r13=0000005171020f10 r14=000003494da9f588 r15=000003494da9f5c8
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[6] 000003494da9f610 rip=00000051b7486234 rbx=0000005170fec5e0 rbp=000003494da9f660 r12=0000005170fc78d0 r13=000003494da9f680 r14=0000000000000001 r15=000003494da9f628
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[7] 000003494da9f670 rip=000000516f695874 rbx=000003494da9f960 rbp=000003494da9f8a0 r12=0000005170feae00 r13=0000005170fc78d0 r14=000003494da9f680 r15=00000051b768d060
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[8] 000003494da9f8b0 rip=000000516f5aea84 rbx=000003494da9f960 rbp=000003494da9f920 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=00000051705249b0
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[9] 000003494da9f930 rip=000000516f5ae5d4 rbx=0000005170fc78d0 rbp=000003494da9fa20 r12=00000051705249b0 r13=0000000000000000 r14=0000005170525040 r15=000003494da9f960
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[10] 000003494da9fa30 rip=000000516f5af74b rbx=00000000000007e8 rbp=000003494da9fb10 r12=00000051705247a0 r13=2e8ba2e8ba2e8ba3 r14=000000517062be40 r15=00000051705263a8
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[11] 000003494da9fb20 rip=000000516f5ba46b rbx=0000000000000000 rbp=000003494da9fb60 r12=0000000000000000 r13=0000005170525580 r14=aaaaaaaaaaaaaaab r15=00000051705263a8
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[12] 000003494da9fb70 rip=000000516f52b0a5 rbx=0000005170deb900 rbp=000003494da9fba0 r12=00000051707bb0c0 r13=0000000000000000 r14=00000051707bb080 r15=0000000000000000
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[13] 000003494da9fbb0 rip=000000516f52b8d2 rbx=00000051707bb0c0 rbp=000003494da9fbf0 r12=0000000000000000 r13=0000005170000127 r14=000000516f520470 r15=0000000000000000
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[14] 000003494da9fc00 rip=000000516f51f731 rbx=0000000000000003 rbp=000003494da9fca0 r12=0000000000000000 r13=0000005170000127 r14=000000516f520470 r15=0000005170479ba0
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[15] 000003494da9fcb0 rip=00000051b29f9c8d rbx=0000000000000000 rbp=0000000000000000 r12=000000516f5200dc r13=000003494da9fd78 r14=0000000000000000 r15=0000000000000000
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[16] 000003494da9fd70 rip=000000516f520105 rbx=0000000000000000 rbp=0000000000000000 r12=000000516f5200dc r13=000003494da9fd78 r14=0000000000000000 r15=0000000000000000
2022-03-23T18:58:18.005Z In(05) vmx - Backtrace[17] 000003494da9fd78 rip=0000000000000000 rbx=0000000000000000 rbp=0000000000000000 r12=000000516f5200dc r13=000003494da9fd78 r14=0000000000000000 r15=0000000000000000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[0] 000003494da9ee80 rip=000000516f52b769 in function Panic in object /bin/vmx loaded at 000000516f2c1000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[1] 000003494da9f390 rip=000000516f698136 in function (null) in object /bin/vmx loaded at 000000516f2c1000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[2] 000003494da9f3e0 rip=00000051b74885e4 in function vmiop_register_emul_pci in object /usr/lib64/vmware/plugin/libvmx-vmiop.so loaded at 00000051b747d000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[3] 000003494da9f440 rip=00000051b793459b in function _nv009211vgpu in object /usr/lib64/vmware/plugin/libnvidia-vgx.so loaded at 00000051b788f000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[4] 000003494da9f460 rip=00000051b79425d1 in function (null) in object /usr/lib64/vmware/plugin/libnvidia-vgx.so loaded at 00000051b788f000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[5] 000003494da9f560 rip=00000051b7943c0e in function (null) in object /usr/lib64/vmware/plugin/libnvidia-vgx.so loaded at 00000051b788f000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[6] 000003494da9f610 rip=00000051b7486234 in function (null) in object /usr/lib64/vmware/plugin/libvmx-vmiop.so loaded at 00000051b747d000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[7] 000003494da9f670 rip=000000516f695874 in function (null) in object /bin/vmx loaded at 000000516f2c1000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[8] 000003494da9f8b0 rip=000000516f5aea84 in function (null) in object /bin/vmx loaded at 000000516f2c1000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[9] 000003494da9f930 rip=000000516f5ae5d4 in function (null) in object /bin/vmx loaded at 000000516f2c1000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[10] 000003494da9fa30 rip=000000516f5af74b in function (null) in object /bin/vmx loaded at 000000516f2c1000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[11] 000003494da9fb20 rip=000000516f5ba46b in function (null) in object /bin/vmx loaded at 000000516f2c1000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[12] 000003494da9fb70 rip=000000516f52b0a5 in function (null) in object /bin/vmx loaded at 000000516f2c1000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[13] 000003494da9fbb0 rip=000000516f52b8d2 in function (null) in object /bin/vmx loaded at 000000516f2c1000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[14] 000003494da9fc00 rip=000000516f51f731 in function (null) in object /bin/vmx loaded at 000000516f2c1000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[15] 000003494da9fcb0 rip=00000051b29f9c8d in function __libc_start_main in object /lib64/libc.so.6 loaded at 00000051b29d8000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[16] 000003494da9fd70 rip=000000516f520105 in function (null) in object /bin/vmx loaded at 000000516f2c1000
2022-03-23T18:58:18.005Z In(05) vmx - SymBacktrace[17] 000003494da9fd78 rip=0000000000000000
2022-03-23T18:58:18.005Z In(05) vmx - Printing loaded objects
2022-03-23T18:58:18.005Z In(05) vmx - [0x516F2C1000-0x51704775DC): /bin/vmx
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B0C90000-0x51B0C96630): /lib64/librt.so.1
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B0E98000-0x51B0E99E90): /lib64/libdl.so.2
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B109C000-0x51B1338C9C): /lib64/libcrypto.so.1.0.2
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B156B000-0x51B15D437C): /lib64/libssl.so.1.0.2
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B17DE000-0x51B18F2964): /lib64/libX11.so.6
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B1AF9000-0x51B1B0801C): /lib64/libXext.so.6
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B1D0A000-0x51B1D23890): /lib64/libconfigstorec.so
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B1F26000-0x51B2095CDA): /lib64/libstdc++.so.6
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B22A6000-0x51B23A1B94): /lib64/libm.so.6
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B25A4000-0x51B25B920C): /lib64/libgcc_s.so.1
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B27BB000-0x51B27D2680): /lib64/libpthread.so.0
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B29D8000-0x51B2B7D3B8): /lib64/libc.so.6
2022-03-23T18:58:18.005Z In(05) vmx - [0x5170A6C000-0x5170A8BCC0): /lib64/ld-linux-x86-64.so.2
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B2D88000-0x51B2DA2634): /lib64/libxcb.so.1
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B2FA3000-0x51B302D31A): /lib64/libconfigstore.so
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B3234000-0x51B323B118): /lib64/libconfigstorecommon.so
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B343E000-0x51B343F95C): /lib64/libXau.so.6
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B3640000-0x51B38E3E3C): /lib64/libvmlibs.so
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B3BBE000-0x51B3BBEFC8): /lib64/libvtl.so
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B3DC2000-0x51B3DF96CC): /lib64/libjsonvalidator.so
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B3FFC000-0x51B4025223): /lib64/libconfigstoredocument.so
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B4227000-0x51B4256C5C): /lib64/libconfigstoredb.so
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B445B000-0x51B4470AE4): /lib64/libconfigstoreupgrade.so
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B4673000-0x51B469F990): /lib64/libschemastore.so
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B48A3000-0x51B48BC65C): /lib64/libz.so.1
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B4ABF000-0x51B4BA696C): /usr/lib64/libsqlite3.so.0
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B4DAC000-0x51B4DC416D): /usr/lib64/libboost_filesystem-gcc64-mt-x64-1_67.so.1.67.0
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B4FC7000-0x51B4FCA5EF): /usr/lib64/libboost_system-gcc64-mt-x64-1_67.so.1.67.0
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B5236000-0x51B53F6CF4): /usr/lib64/vmware/plugin/objLib/upitObjBE.so
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B56B8000-0x51B58AEBB4): /usr/lib64/vmware/plugin/objLib/vsanObjBE.so
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B5D83000-0x51B5D8E758): /lib64/libnss_files.so.2
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B747D000-0x51B748C024): /usr/lib64/vmware/plugin/libvmx-vmiop.so
2022-03-23T18:58:18.005Z In(05) vmx - [0x51B788F000-0x51B7AB09AC): /usr/lib64/vmware/plugin/libnvidia-vgx.so
2022-03-23T18:58:18.005Z In(05) vmx - End printing loaded objects
2022-03-23T18:58:18.005Z In(05) vmx - Panic: monitor not available, skipping monitor coredump.
2022-03-23T18:58:18.005Z In(05) vmx - Exiting

Re: Unable to start a VM with Nvidia A40 vGPU profile

fabio1975 — Wed, 23 Mar 2022 21:48:19 GMT

Ciao

You can use a cloud share (OneDrive or Google Drive) and send the link in a personal message.

Can you confirm that the ECC memory is disabled ( nvidia-smi -q )?

Can you confirm that the SR-IOV is enabled in BIOS?

Can you confirm that the configuration of the vGPU cards on the host are Shared Direct?

how:

Re: Unable to start a VM with Nvidia A40 vGPU profile

tomtomek — Thu, 24 Mar 2022 09:38:20 GMT

Hi Fabio,

Will send the log shortly. One will be for startup with ECC on a card enabled and one with ECC disabled.

ECC was enabled. I have now disabled it, I take it this is preferred? The machines still would not start,

SR-IOV is enabled in BIOS.

vGPU is shared direct.

Regards,

Tom

Re: Unable to start a VM with Nvidia A40 vGPU profile

fabio1975 — Thu, 24 Mar 2022 12:42:59 GMT

Ciao

Set ECC memory disable.

I'll check the logs shortly

Re: Unable to start a VM with Nvidia A40 vGPU profile

tomtomek — Thu, 24 Mar 2022 17:10:31 GMT

Hi Fabio,

Managed to fix this. For some reason the devices were set to Passthrough enabled. Now after toggling passthrough all is back to normal.

I will post another thread relating to my HDD extension issue. Hope you may help with this. Thanks for your support!