VM Crash Dump Analysis - Ubuntu 20.04 on ESX 8.0u1 - Is this a tech support only thing?

[IamTHEvilONE]

I'm looking for a tool/page to debug the following ... unless it's a support only thing then I've gotta submit a TSA net for it (potentially)

I recently updated to:

- VMware ESXi, 8.0.1, 21495797

Since then, exactly one of my VMs has been crashing on boot.

Single user mode does not reproduce the same results.

So it's something after that point in the boot process.

All of the rip info is different each time the VM boots, but here is the first recorded crash that was in the vmware log that produced vmware-zdump.000 (I omitted that one line from the log)

2023-04-21T20:26:12Z[+1.191] Wa(03) vcpu-3 - Caught signal 11 -- tid 1210066 (addr 0)
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 - SIGNAL: rip 0x94d2e76fb3 rsp 0x9519c6e758 rbp 0x9519c6e7c0
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 - SIGNAL: rax 0x1c rbx 0x95180451a0 rcx 0x2 rdx 0x0 rsi 0x94d485efd0 rdi 0x0
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 -         r8 0x95180451b8 r9 0x95180451b8 r10 0x94d485efd0 r11 0x202 r12 0x94d485efd0 r13 0xf2 r14 0x9518016000 r15 0x9518f34000
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 - SIGNAL: stack 9519C6E758 : 0x00000094d2e68f99 0x0000009518014f80
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 - SIGNAL: stack 9519C6E768 : 0x0000009519c75700 0x0000009519c6e7a0
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 - SIGNAL: stack 9519C6E778 : 0x00000094d31c6df3 0x0000009518014f80
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 - SIGNAL: stack 9519C6E788 : 0x0000009519c75700 0x00000094d46392b0
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 - SIGNAL: stack 9519C6E798 : 0x00000095180451a0 0x0000000000208070
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 - SIGNAL: stack 9519C6E7A8 : 0x00000094d46392b0 0x00000094d4866990
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 - SIGNAL: stack 9519C6E7B8 : 0x0000000000000000 0x0000009519c6e830
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 - SIGNAL: stack 9519C6E7C8 : 0x00000094d2d807d0 0x0000009518014f68
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 - Backtrace:
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 - Backtrace[0] 0000009519c6d570 rip=00000094d34e9c94 rbx=0000009519c6e7d8 rbp=0000009519c6d760 r12=000000000000000b r13=0000000000000000 r14=00000094d39b6268 r15=0000009519c6e7d8
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 - Backtrace[1] 0000009519c6d770 rip=00000094d34ea4f1 rbx=00000094d4016c08 rbp=0000009519c6d7b0 r12=0000009519c6d7c0 r13=000000000000000b r14=0000009519c6f3a8 r15=0000000000000010
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 - Backtrace[2] 0000009519c6d7c0 rip=0000000000281002 rbx=00000095180451a0 rbp=0000009519c6e6c0 r12=0000009519c6d840 r13=00000000000000f2 r14=0000009518016000 r15=0000009518f34000
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 - SymBacktrace[0] 0000009519c6d570 rip=00000094d34e9c94 in function (null) in object /bin/vmx loaded at 00000094d299d000
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 - SymBacktrace[1] 0000009519c6d770 rip=00000094d34ea4f1 in function (null) in object /bin/vmx loaded at 00000094d299d000
2023-04-21T20:26:12Z[+1.192] In(05) vcpu-3 - SymBacktrace[2] 0000009519c6d7c0 rip=0000000000281002 
2023-04-21T20:26:12Z[+1.192] Cr(01) vcpu-3 - PANIC: Unexpected signal: 11.
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - Backtrace:
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - Backtrace[0] 0000009519c6d060 rip=00000094d2c1a2e4 rbx=0000000000000001 rbp=0000009519c6d560 r12=00000094d401db80 r13=0000009519c6d080 r14=00000094d39b6268 r15=00000
09519c6e7d8
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - Backtrace[1] 0000009519c6d570 rip=00000094d34e9b94 rbx=0000009519c6e7d8 rbp=0000009519c6d760 r12=000000000000000b r13=0000009519c6d690 r14=00000094d39b6268 r15=0000009519c6e7d8
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - Backtrace[2] 0000009519c6d770 rip=00000094d34ea4f1 rbx=00000094d4016c08 rbp=0000009519c6d7b0 r12=0000009519c6d7c0 r13=000000000000000b r14=0000009519c6f3a8 r15=0000000000000010
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - Backtrace[3] 0000009519c6d7c0 rip=0000000000281002 rbx=00000095180451a0 rbp=0000009519c6e6c0 r12=0000009519c6d840 r13=00000000000000f2 r14=0000009518016000 r15=0000009518f34000
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - SymBacktrace[0] 0000009519c6d060 rip=00000094d2c1a2e4 in function Panic in object /bin/vmx loaded at 00000094d299d000
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - SymBacktrace[1] 0000009519c6d570 rip=00000094d34e9b94 in function (null) in object /bin/vmx loaded at 00000094d299d000
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - SymBacktrace[2] 0000009519c6d770 rip=00000094d34ea4f1 in function (null) in object /bin/vmx loaded at 00000094d299d000
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - SymBacktrace[3] 0000009519c6d7c0 rip=0000000000281002 
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - Printing loaded objects
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x94D299D000-0x94D3C4D1A8): /bin/vmx
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x9514416000-0x951441C630): /lib64/librt.so.1
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x951461E000-0x951461FE90): /lib64/libdl.so.2
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x9514822000-0x9514C52988): /lib64/libcrypto.so.3
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x9514EB4000-0x9514F50664): /lib64/libssl.so.3
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x951515E000-0x9515272964): /lib64/libX11.so.6
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x9515479000-0x951548801C): /lib64/libXext.so.6
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x951568A000-0x95156A6AFF): /lib64/libconfigstorec.so
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x95156AA000-0x95157473B8): /lib64/libstdc++.so.6
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x951588C000-0x9515987BB4): /lib64/libm.so.6
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x9515B8A000-0x9515B8CD18): /lib64/libgcc_s.so.1
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x9515BA4000-0x9515BBB680): /lib64/libpthread.so.0
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x9515DC1000-0x9515F69150): /lib64/libc.so.6
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x94D41F2000-0x94D4211F08): /lib64/ld-linux-x86-64.so.2
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x9516174000-0x951618E634): /lib64/libxcb.so.1
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x951638F000-0x95164F3DE5): /lib64/libconfigstorecommon.so
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x9516502000-0x9516586E24): /lib64/libconfigstore.so
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x951658E000-0x951658F95C): /lib64/libXau.so.6
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x9516790000-0x9516A5BE60): /lib64/libvmlibs.so
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x9516B3D000-0x9516B3DF44): /lib64/libvtl.so
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x9516B42000-0x9516CA8D9D): /lib64/libjsonvalidator.so
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x9516CB7000-0x9516D27D4A): /lib64/libconfigstoredocument.so
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x9516D2C000-0x9516D60CD0): /lib64/libconfigstoredb.so
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x9516D66000-0x9516F01431): /lib64/libschemastore.so
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x9516F10000-0x95170136E4): /usr/lib64/libsqlite3.so.0
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x951721A000-0x951723471C): /lib64/libz.so.1
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x95174A3000-0x9517623598): /usr/lib64/ossl-modules/fips.so
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x951783C000-0x9517A459BC): /usr/lib64/vmware/plugin/objLib/vsanObjBE.so
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - [0x9517D21000-0x9517D2C758): /lib64/libnss_files.so.2
2023-04-21T20:26:12Z[+1.882] In(05) vcpu-3 - End printing loaded objects

I have many similar VMs running Ubuntu 20.04 or similar variants.

The Guest OS was working fine and rebooted many times on previous builds of ESXi.  This VM has been around since the same hardware was running ESXi 6.7 and patched/upgraded through 7.0 to 8.0u1 where it is now.

I will note that I think it's the `only` one that used SATA disks rather than SCSI for Ubuntu, but that is supported according to the Guest OS compatibility chart.

2023-04-21T20:29:08.347Z In(05) vmx - DICT --- GLOBAL SETTINGS /usr/lib/vmware/settings
2023-04-21T20:29:08.347Z In(05) vmx - DICT --- NON PERSISTENT (null)
2023-04-21T20:29:08.347Z In(05) vmx - DICT --- HOST DEFAULTS /etc/vmware/config
2023-04-21T20:29:08.347Z In(05) vmx - DICT                    libdir = "/usr/lib/vmware"
2023-04-21T20:29:08.347Z In(05) vmx - DICT --- NONPERSISTENT
2023-04-21T20:29:08.347Z In(05) vmx - DICT               sched.group = "host/user"
2023-04-21T20:29:08.347Z In(05) vmx - DICT --- COMMAND LINE
2023-04-21T20:29:08.347Z In(05) vmx - DICT               sched.group = "host/user"
2023-04-21T20:29:08.347Z In(05) vmx - DICT --- RECORDING
2023-04-21T20:29:08.347Z In(05) vmx - DICT               sched.group = "host/user"
2023-04-21T20:29:08.347Z In(05) vmx - DICT --- CONFIGURATION /vmfs/volumes/643b5b20-30c46151-a887-38eaa717cd38/edge-01/edge-01.vmx 
2023-04-21T20:29:08.347Z In(05) vmx - DICT               displayName = "edge-01-prod"
2023-04-21T20:29:08.347Z In(05) vmx - DICT            config.version = "8"
2023-04-21T20:29:08.347Z In(05) vmx - DICT         virtualHW.version = "20"
2023-04-21T20:29:08.347Z In(05) vmx - DICT             vmci0.present = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT           floppy0.present = "FALSE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT                  numvcpus = "4"
2023-04-21T20:29:08.347Z In(05) vmx - DICT                   memSize = "24576"
2023-04-21T20:29:08.347Z In(05) vmx - DICT      tools.upgrade.policy = "manual"
2023-04-21T20:29:08.347Z In(05) vmx - DICT           sched.cpu.units = "mhz"
2023-04-21T20:29:08.347Z In(05) vmx - DICT             vm.createDate = "1570463734018000"
2023-04-21T20:29:08.347Z In(05) vmx - DICT             sata0.present = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT             sata1.present = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT          sata1:0.fileName = "edge-01_1-000002.vmdk"
2023-04-21T20:29:08.347Z In(05) vmx - DICT      sched.sata1:0.shares = "normal"
2023-04-21T20:29:08.347Z In(05) vmx - DICT sched.sata1:0.throughputCap = "off"
2023-04-21T20:29:08.347Z In(05) vmx - DICT           sata1:0.present = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT          sata0:0.fileName = "edge-01-000002.vmdk"
2023-04-21T20:29:08.347Z In(05) vmx - DICT      sched.sata0:0.shares = "normal"
2023-04-21T20:29:08.347Z In(05) vmx - DICT sched.sata0:0.throughputCap = "off"
2023-04-21T20:29:08.347Z In(05) vmx - DICT           sata0:0.present = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT      ethernet0.virtualDev = "vmxnet3"
2023-04-21T20:29:08.347Z In(05) vmx - DICT    ethernet0.dvs.switchId = "50 35 17 5f aa 88 8e 8a-6e 5a 4f 16 8c 71 93 34"
2023-04-21T20:29:08.347Z In(05) vmx - DICT      ethernet0.dvs.portId = "2"
2023-04-21T20:29:08.347Z In(05) vmx - DICT ethernet0.dvs.portgroupId = "dvportgroup-2003"
2023-04-21T20:29:08.347Z In(05) vmx - DICT ethernet0.dvs.connectionId = "2112906521"
2023-04-21T20:29:08.347Z In(05) vmx - DICT          ethernet0.shares = "normal"
2023-04-21T20:29:08.347Z In(05) vmx - DICT     ethernet0.addressType = "vpx"
2023-04-21T20:29:08.347Z In(05) vmx - DICT ethernet0.generatedAddress = "00:50:56:85:49:83"
2023-04-21T20:29:08.347Z In(05) vmx - DICT ethernet0.uptCompatibility = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT         ethernet0.present = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT                annotation = <not printed>
2023-04-21T20:29:08.347Z In(05) vmx - DICT                   guestOS = "ubuntu-64"
2023-04-21T20:29:08.347Z In(05) vmx - DICT chipset.motherboardLayout = "i440bx"
2023-04-21T20:29:08.347Z In(05) vmx - DICT  toolScripts.afterPowerOn = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT   toolScripts.afterResume = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT toolScripts.beforeSuspend = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT toolScripts.beforePowerOff = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT            tools.syncTime = "FALSE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT                 uuid.bios = "42 05 e6 7b e5 1d 99 8c-fc 60 14 70 68 de b4 46"
2023-04-21T20:29:08.347Z In(05) vmx - DICT                   vc.uuid = "50 05 04 6c 33 e9 72 27-d8 05 3a 2b ec 15 cf cb"
2023-04-21T20:29:08.347Z In(05) vmx - DICT sched.cpu.latencySensitivity = "normal"
2023-04-21T20:29:08.347Z In(05) vmx - DICT tools.guest.desktop.autolock = "FALSE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT                     nvram = "edge-01.nvram"
2023-04-21T20:29:08.347Z In(05) vmx - DICT        pciBridge0.present = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT              svga.present = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT        pciBridge4.present = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT     pciBridge4.virtualDev = "pcieRootPort"
2023-04-21T20:29:08.347Z In(05) vmx - DICT      pciBridge4.functions = "8"
2023-04-21T20:29:08.347Z In(05) vmx - DICT        pciBridge5.present = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT     pciBridge5.virtualDev = "pcieRootPort"
2023-04-21T20:29:08.347Z In(05) vmx - DICT      pciBridge5.functions = "8"
2023-04-21T20:29:08.347Z In(05) vmx - DICT        pciBridge6.present = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT     pciBridge6.virtualDev = "pcieRootPort"
2023-04-21T20:29:08.347Z In(05) vmx - DICT      pciBridge6.functions = "8"
2023-04-21T20:29:08.347Z In(05) vmx - DICT        pciBridge7.present = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT     pciBridge7.virtualDev = "pcieRootPort"
2023-04-21T20:29:08.347Z In(05) vmx - DICT      pciBridge7.functions = "8"
2023-04-21T20:29:08.347Z In(05) vmx - DICT             hpet0.present = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT       sata1.pciSlotNumber = "34"
2023-04-21T20:29:08.347Z In(05) vmx - DICT       sata0.pciSlotNumber = "33"
2023-04-21T20:29:08.347Z In(05) vmx - DICT   ethernet0.pciSlotNumber = "192"
2023-04-21T20:29:08.347Z In(05) vmx - DICT RemoteDisplay.maxConnections = "1"
2023-04-21T20:29:08.347Z In(05) vmx - DICT isolation.tools.diskShrink.disable = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT isolation.tools.memSchedFakeSampleStats.disable = "true"
2023-04-21T20:29:08.347Z In(05) vmx - DICT isolation.tools.copy.disable = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT isolation.device.connectable.disable = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT     guest.command.enabled = "false"
2023-04-21T20:29:08.347Z In(05) vmx - DICT isolation.device.edit.disable = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT isolation.tools.unity.push.update.disable = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT isolation.tools.diskWiper.disable = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT isolation.tools.getCred.disable = "true"
2023-04-21T20:29:08.347Z In(05) vmx - DICT           serialX.present = "false"
2023-04-21T20:29:08.347Z In(05) vmx - DICT            ideX:Y.present = "false"
2023-04-21T20:29:08.347Z In(05) vmx - DICT isolation.tools.ghi.launchmenu.change = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT isolation.bios.bbs.disable = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT              usbX.present = "false"
2023-04-21T20:29:08.347Z In(05) vmx - DICT isolation.tools.setGUIOptions.enable = "FALSE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT isolation.tools.hgfsServerSet.disable = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT isolation.tools.paste.disable = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT           floppyX.present = "false"
2023-04-21T20:29:08.347Z In(05) vmx - DICT   tools.setInfo.sizeLimit = "104857"
2023-04-21T20:29:08.347Z In(05) vmx - DICT         parallelX.present = "false"
2023-04-21T20:29:08.347Z In(05) vmx - DICT isolation.tools.dnd.disable = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT     snapshot.maxSnapshots = "2"
2023-04-21T20:29:08.347Z In(05) vmx - DICT isolation.tools.ghi.autologon.change = "true"
2023-04-21T20:29:08.347Z In(05) vmx - DICT      numa.autosize.cookie = "40001"
2023-04-21T20:29:08.347Z In(05) vmx - DICT numa.autosize.vcpu.maxPerVirtualNode = "4"
2023-04-21T20:29:08.347Z In(05) vmx - DICT  pciBridge0.pciSlotNumber = "17"
2023-04-21T20:29:08.347Z In(05) vmx - DICT  pciBridge4.pciSlotNumber = "21"
2023-04-21T20:29:08.347Z In(05) vmx - DICT  pciBridge5.pciSlotNumber = "22"
2023-04-21T20:29:08.347Z In(05) vmx - DICT  pciBridge6.pciSlotNumber = "23"
2023-04-21T20:29:08.347Z In(05) vmx - DICT  pciBridge7.pciSlotNumber = "24"
2023-04-21T20:29:08.347Z In(05) vmx - DICT       vmci0.pciSlotNumber = "32"
2023-04-21T20:29:08.347Z In(05) vmx - DICT    monitor.phys_bits_used = "40"
2023-04-21T20:29:08.347Z In(05) vmx - DICT  vmotion.checkpointFBSize = "4194304"
2023-04-21T20:29:08.347Z In(05) vmx - DICT              softPowerOff = "FALSE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT       tools.remindInstall = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT    time.synchronize.allow = "TRUE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT      cpuid.coresPerSocket = "2"
2023-04-21T20:29:08.347Z In(05) vmx - DICT toolsInstallManager.updateCounter = "2"
2023-04-21T20:29:08.347Z In(05) vmx - DICT   guestInfo.detailed.data = <not printed>
2023-04-21T20:29:08.347Z In(05) vmx - DICT           migrate.hostLog = "edge-01-3957695f.hlog"
2023-04-21T20:29:08.347Z In(05) vmx - DICT             sched.cpu.min = "0"
2023-04-21T20:29:08.347Z In(05) vmx - DICT          sched.cpu.shares = "normal"
2023-04-21T20:29:08.347Z In(05) vmx - DICT             sched.mem.min = "0"
2023-04-21T20:29:08.347Z In(05) vmx - DICT         sched.mem.minSize = "0"
2023-04-21T20:29:08.347Z In(05) vmx - DICT          sched.mem.shares = "normal"
2023-04-21T20:29:08.347Z In(05) vmx - DICT    migrate.encryptionMode = "opportunistic"
2023-04-21T20:29:08.347Z In(05) vmx - DICT    ftcpt.ftEncryptionMode = "ftEncryptionOpportunistic"
2023-04-21T20:29:08.347Z In(05) vmx - DICT guestinfo.rubrik.platform = <not printed>
2023-04-21T20:29:08.347Z In(05) vmx - DICT guestinfo.vmtools.buildNumber = <not printed>
2023-04-21T20:29:08.347Z In(05) vmx - DICT guestinfo.vmtools.description = <not printed>
2023-04-21T20:29:08.347Z In(05) vmx - DICT guestinfo.vmtools.versionNumber = <not printed>
2023-04-21T20:29:08.347Z In(05) vmx - DICT guestinfo.vmtools.versionString = <not printed>
2023-04-21T20:29:08.347Z In(05) vmx - DICT    sched.swap.derivedName = "/vmfs/volumes/643b5b20-30c46151-a887-38eaa717cd38/edge-01/edge-01-c12d3f41.vswp"
2023-04-21T20:29:08.347Z In(05) vmx - DICT              sata0:0.redo = ""
2023-04-21T20:29:08.347Z In(05) vmx - DICT              sata1:0.redo = ""
2023-04-21T20:29:08.347Z In(05) vmx - DICT                  vmci0.id = "1759425606"
2023-04-21T20:29:08.347Z In(05) vmx - DICT           config.readOnly = "FALSE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT         vmxstats.filename = "edge-01.scoreboard"
2023-04-21T20:29:08.347Z In(05) vmx - DICT         guestinfo.appInfo = <not printed>
2023-04-21T20:29:08.347Z In(05) vmx - DICT        extendedConfigFile = "edge-01.vmxf"
2023-04-21T20:29:08.347Z In(05) vmx - DICT                  viv.moid = "447dd44b-8568-4e30-9109-530f7d163801:vm-95:1m64eEpL8Ci9bSa4FJ2EVmd04x2hLN1aG2lGuLkHLYE="
2023-04-21T20:29:08.347Z In(05) vmx - DICT             cleanShutdown = "FALSE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT     acpi.smbiosVersion2.7 = "FALSE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT    vmxnet3.serialNumberV2 = "FALSE"
2023-04-21T20:29:08.347Z In(05) vmx - DICT --- HOST DEFAULTS /etc/vmware/config 
2023-04-21T20:29:08.347Z In(05) vmx - DICT                    libdir = "/usr/lib/vmware"
2023-04-21T20:29:08.347Z In(05) vmx - DICT --- SITE DEFAULTS /usr/lib/vmware/config 
2023-04-21T20:29:08.347Z In(05) vmx - DICT --- GLOBAL SETTINGS /usr/lib/vmware/settings 

[IamTHEvilONE]

So there are 2 things I have found thus far:

1. Single user mode lets the system boot and the SATA HDDs are mountable.

2. Changing to a SCSI controller seems to avoid the issue (the SATA controller has to be removed)

Without knowing an RCA on the RIP/Backtrace, it feels like something to do with the above two in some combination of drivers/etc.