lxylxy123456's Posts

Thanks for letting me know. Interesting, I posted this bug here because I saw https://www.vmware.com/support/policies/defect.html . Do I need something special (e.g. active support agreement) to open... See more...
Thanks for letting me know. Interesting, I posted this bug here because I saw https://www.vmware.com/support/policies/defect.html . Do I need something special (e.g. active support agreement) to open a service request? When I am trying to post a service request I do not see anything under "Supported Products".
Thanks for the reply. I have edited the bug report to include my host CPU model: Intel(R) Core(TM) i5-7600 CPU @ 3.50GHz Since this bug is relevant to non-maskable interrupts (NMIs), I think virtual... See more...
Thanks for the reply. I have edited the bug report to include my host CPU model: Intel(R) Core(TM) i5-7600 CPU @ 3.50GHz Since this bug is relevant to non-maskable interrupts (NMIs), I think virtualised interrupt delivery and posted process interrupts are likely unrelated. According to SDM, these features are only related to maskable interrupts. Just FYI, here are the guest VMX capabilities printed in vmware.log: Guest VT-x Capabilities: Basic VMX Information (0x00d8100000000001) VMCS revision ID 1 VMCS region length 4096 VMX physical-address width natural SMM dual-monitor mode no VMCS memory type WB Advanced INS/OUTS info yes True VMX MSRs yes Exception Injection ignores error code no True Pin-Based VM-Execution Controls (0x0000003f00000016) External-interrupt exiting {0,1} NMI exiting {0,1} Virtual NMIs {0,1} Activate VMX-preemption timer { 0 } Process posted interrupts { 0 } True Primary Processor-Based VM-Execution Controls (0xfff9fffe04006172) Interrupt-window exiting {0,1} Use TSC offsetting {0,1} HLT exiting {0,1} INVLPG exiting {0,1} MWAIT exiting {0,1} RDPMC exiting {0,1} RDTSC exiting {0,1} CR3-load exiting {0,1} CR3-store exiting {0,1} Activate tertiary controls { 0 } CR8-load exiting {0,1} CR8-store exiting {0,1} Use TPR shadow {0,1} NMI-window exiting {0,1} MOV-DR exiting {0,1} Unconditional I/O exiting {0,1} Use I/O bitmaps {0,1} Monitor trap flag {0,1} Use MSR bitmaps {0,1} MONITOR exiting {0,1} PAUSE exiting {0,1} Activate secondary controls {0,1} Secondary Processor-Based VM-Execution Controls (0x00553cfe00000000) Virtualize APIC accesses { 0 } Enable EPT {0,1} Descriptor-table exiting {0,1} Enable RDTSCP {0,1} Virtualize x2APIC mode {0,1} Enable VPID {0,1} WBINVD exiting {0,1} Unrestricted guest {0,1} APIC-register virtualization { 0 } Virtual-interrupt delivery { 0 } PAUSE-loop exiting {0,1} RDRAND exiting {0,1} Enable INVPCID {0,1} Enable VM Functions {0,1} Use VMCS shadowing { 0 } ENCLS exiting { 0 } RDSEED exiting {0,1} Enable PML { 0 } EPT-violation #VE {0,1} Conceal VMX from PT { 0 } Enable XSAVES/XRSTORS {0,1} PASID translation { 0 } Mode-based execute control for EPT {0,1} Sub-page write permissions for EPT { 0 } PT uses guest physical addresses { 0 } Use TSC scaling { 0 } Enable UMWAIT and TPAUSE { 0 } Enable ENCLV in VMX non-root mode { 0 } Enable EPC Virtualization Extensions { 0 } Bus lock exiting { 0 } Notification VM exits { 0 } Tertiary Processor-Based VM-Execution Controls (0x0000000000000000) LOADIWKEY exiting no Enable HLAT no Enable Paging-Write no Enable Guest Paging Verification no Enable IPI Virtualization no True VM-Exit Controls (0x003fefff00036dfb) Save debug controls {0,1} Host address-space size {0,1} Load IA32_PERF_GLOBAL_CTRL { 0 } Acknowledge interrupt on exit {0,1} Save IA32_PAT {0,1} Load IA32_PAT {0,1} Save IA32_EFER {0,1} Load IA32_EFER {0,1} Save VMX-preemption timer { 0 } Clear IA32_BNDCFGS { 0 } Conceal VMX from processor trace { 0 } Clear IA32_RTIT MSR { 0 } Clear IA32_LBR_CTL MSR { 0 } Clear user-interrupt notification vector { 0 } Load CET state { 0 } Load PKRS { 0 } True VM-Entry Controls (0x0000d3ff000011fb) Load debug controls {0,1} IA-32e mode guest {0,1} Entry to SMM { 0 } Deactivate dual-monitor mode { 0 } Load IA32_PERF_GLOBAL_CTRL { 0 } Load IA32_PAT {0,1} Load IA32_EFER {0,1} Load IA32_BNDCFGS { 0 } Conceal VMX from processor trace { 0 } Load IA32_RTIT MSR { 0 } Load user-interrupt notification vector { 0 } Load CET state { 0 } Load IA32_LBR_CTL MSR { 0 } Load PKRS { 0 } VPID and EPT Capabilities (0x00000f0106714141) R=0/W=0/X=1 yes Page-walk length 3 yes EPT memory type WB yes 2MB super-page yes 1GB super-page no INVEPT support yes Access & Dirty Bits yes Advanced VM exit information for EPT violations yes Supervisor shadow-stack control no Type 1 INVEPT yes Type 2 INVEPT yes INVVPID support yes Type 0 INVVPID yes Type 1 INVVPID yes Type 2 INVVPID yes Type 3 INVVPID yes Miscellaneous VMX Data (0x00000000400401e0) TSC to preemption timer ratio 0 VM-Exit saves EFER.LMA yes Activity State HLT yes Activity State shutdown yes Activity State wait-for-SIPI yes Processor trace in VMX no RDMSR SMBASE MSR in SMM no CR3 targets supported 4 Maximum MSR list size 512 VMXOFF holdoff of SMIs no Allow all VMWRITEs no Allow zero instruction length yes MSEG revision ID 0 VMX-Fixed Bits in CR0 (0x0000000080000021/0x00000000ffffffff) Fixed to 0 0xffffffff00000000 Fixed to 1 0x0000000080000021 Variable 0x000000007fffffde VMX-Fixed Bits in CR4 (0x0000000000002000/0x00000000003727ff) Fixed to 0 0xffffffffffc8d800 Fixed to 1 0x0000000000002000 Variable 0x00000000003707ff VMCS Enumeration (0x000000000000005a) Highest index 0x2d VM Functions (0x0000000000000001) Function 0 (EPTP-switching) supported.  
Hi, I am recently experimenting with how VMware handles NMI interrupts in nested virtualization, and I find that the behavior of VMware is incorrect in a corner case. The version of VMware and my h... See more...
Hi, I am recently experimenting with how VMware handles NMI interrupts in nested virtualization, and I find that the behavior of VMware is incorrect in a corner case. The version of VMware and my host OS are: Product: VMware(R) Workstation 17 Pro Version: 17.0.1 build-21139696 Host OS: Debian 11, kernel version 5.10.0-21-amd64 Host CPU: Intel(R) Core(TM) i5-7600 CPU @ 3.50GHz To reproduce this bug: Create a new virtual machine, choose "Other" as guest OS Open "Edit Virtual Machine Settings" Remove the default disk, add attached VMDK file (d.vmdk) instead Change number of processors from 1 to 2 Enable "Virtualize Intel VT-x/EPT or AMD-V/RVI" Add a serial port to the virtual machine, be able to read it (e.g. use output file) Keep other configurations as default (for me, it is 256 MB memory, ...) Start the virtual machine Observe the serial port output The source code of the VMDK file can be found at: https://github.com/lxylxy123456/uberxmhf/blob/9eb50d71910b2586c11f92462f37096a7066502b/xmhf/src/xmhf-core/xmhf-runtime/xmhf-startup/lhv-guest.c#L1030 Actual output (on VMware):     ... Detecting environment End detecting environment Experiment: 17 Enter host, exp=17, state=0 hlt_wait() begin, source = EXIT_NMI_H (5) Inject NMI Interrupt recorded: EXIT_NMI_H (5) At instruction: 0x00000000 VM-exit reason: 0x00000012 hlt_wait() end hlt_wait() begin, source = EXIT_TIMER_H (6) Inject interrupt Interrupt recorded: EXIT_TIMER_H (6) hlt_wait() end hlt_wait() begin, source = EXIT_TIMER_H (6) Inject NMI Inject interrupt Interrupt recorded: EXIT_TIMER_H (6) hlt_wait() end Leave host Interrupt recorded: EXIT_NMI_H (5) At instruction: 0x00000000 VM-exit reason: 0x0000000a CPU(0x02): key press: 65, guest=1 source: EXIT_VMEXIT (7) exit_source: EXIT_NMI_H (5) rip: 0x082013df exit_rip: 0x08208488 TEST_ASSERT '0 && (exit_source == source)' failed, line 372, file lhv-guest.c     Expected output (reproducible on real Intel hardware):     ... Detecting environment End detecting environment Experiment: 17 Enter host, exp=17, state=0 hlt_wait() begin, source = EXIT_NMI_H (5) Inject NMI Interrupt recorded: EXIT_NMI_H (5) At instruction: 0x00000000 VM-exit reason: 0x00000012 hlt_wait() end hlt_wait() begin, source = EXIT_TIMER_H (6) Inject interrupt Interrupt recorded: EXIT_TIMER_H (6) hlt_wait() end hlt_wait() begin, source = EXIT_TIMER_H (6) Inject NMI Inject interrupt Interrupt recorded: EXIT_TIMER_H (6) hlt_wait() end Leave host Interrupt recorded: EXIT_VMEXIT (7) CPU(0x01): key press: 250, guest=1 Enter host, exp=17, state=1 iret_wait() begin, source = EXIT_MEASURE (1) iret_wait() end Leave host Experiment: 1 ... (endless)     Explanation: The VMDK file (d.vmdk) contains a micro-hypervisor called LHV. Assume VMware runs in L0, LHV runs in L1, the guest of LHV runs in L2. The code in LHV performs an experiment (called "Experiment 17" in serial output) on CPU 0 to test the behavior of NMI blocking. The experiment steps are: Prepare state such that the CPU is currently in L1 (LHV), and NMI is blocked An NMI interrupt arrives at the CPU. However, since NMI is blocked, the NMI interrupt handler of L1 is not invoked Modify VMCS to make sure that L2 has virtual NMIs enabled (NMI exiting = 1, Virtual NMIs = 1), and L2 blocks NMI (Blocking by NMI = 1) Modify VMCS to inject a normal interrupt (vector 0x21) to L2 at VM entry VM entry to L2 The expected behavior is: 6. Immediately after VM entry L2's interrupt 0x21 handler is invoked 7. VM exit happens immediately due to the NMI interrupt at step 2 However, on VMware, the behavior appears to be: 6. Immediately after VM entry L2's interrupt 0x21 handler is invoked 7. After executing some instructions, L2 executes the CPUID instruction, which causes VM exit to L1 8. Immediately after VM exit, L1's NMI interrupt handler is executed It appears that VMware's implementation is incorrect. NMI is blocked in L2, but Intel's SDM says that NMI is always unblocked in L2. Quote from Intel SDM: The following items describe the use of bit 3 (blocking by NMI) in the interruptibility-state field if the “virtual NMIs” VM-execution control is 1: The bit’s value does not affect the blocking of NMIs after VM entry. NMIs are not blocked in VMX non-root operation (except for ordinary blocking for other reasons, such as by the MOV SS instruction, the wait-for-SIPI state, etc.) ... Could you please fix this implementation problem in VMware? Thank you.
Hi RDPetruska, thank you for the prompt reply. VMware workstation player 16.2.5 build-20904516 have the same bug. VMware workstation player 17.0.0 build-20800274 have the same bug.  
Hi, I recently realized that on x86 virtual machines, memory operations on VGA memory mapped IO with the LOCK prefix are not atomic. An example instruction is the XCHG instruction operating on memor... See more...
Hi, I recently realized that on x86 virtual machines, memory operations on VGA memory mapped IO with the LOCK prefix are not atomic. An example instruction is the XCHG instruction operating on memory address 0xb8000. Here is the information about my environment: Product: VMware(R) Workstation 16 Player Version: 16.2.4 build-20089737 Host OS: Debian 11, kernel version 5.10.0-19-amd64 (Note: this bug is also reproducible on Windows host) I have written a system-level software to demonstrate this bug. The software is attached to this post as "c.vmdk" in "c.zip". To run my code: Open VMware workstation player Create a new virtual machine Select "I will install the operating system later" Guest Operating System: Other Complete creating the virtual machine following default settings Edit virtual machine settings Keep Memory 256 MB default Set Number of processor cores to 4 Remove the default hard disk Create a new harddisk, type is IDE, select "Use existing disk file", use "c.vmdk" attached with this post, do not convert to new vmdk format Create a serial port, use any output file (e.g. "/tmp/serial.txt") Start the virtual machine Observe the content of serial port output file The expected behavior is to see the following from serial port: ... counts: count0 count1 count2 count0-count1+count2 counts: 1 0 1 0 counts: 2 1 1 0 counts: 3 1 2 0 counts: 4 2 2 0 counts: 5 2 3 0 counts: 6 3 3 0 counts: 7 3 4 0 counts: 8 4 4 0 counts: 9 5 4 0 counts: 10 5 5 0 counts: 11 5 6 0 counts: 12 6 6 0 counts: 13 7 6 0 ... (does not stop, right most column is always 0) However, VMware's output stops at some point, and the last column may become negative. For example:     ... counts: count0 count1 count2 count0-count1+count2 counts: 1 1 0 0 counts: 2 2 0 0 counts: 3 3 0 0 counts: 4 4 0 0 counts: 5 5 1 -1 ... (no more outputs)   My code is available in https://github.com/lxylxy123456/uberxmhf/blob/b5935eaf8aab38ce1933da1c1be22dcf1b992eaf/xmhf/src/xmhf-core/xmhf-runtime/xmhf-startup/lhv.c#L5 line 5 - 89 My code performs the following experiment repeatedly on 3 CPUs: Initially, "ptr" at address 0xb8000 (VGA memory mapped I/O) is set to 0 CPU 0 writes 0x12345678 to ptr, then increases counter "count0". In an infinite loop, CPU 1 tries exchanges ptr with register EAX (contains 0) using the XCHG instruction. If CPU 1 sees 0x12345678, it increases counter "count1". CPU 2's behavior is similar to CPU 1, except it increases counter "count2" when it sees 0x12345678. Ideally, after each experiment there should always be count1 + count2 = count0. However, in VMware, there may be count1 + count2 > count0. This because CPU 0 writes 0x12345678 to ptr once, but CPU 1 and CPU 2 both get 0x12345678 in XCHG. Note that according to Intel's specification, XCHG instruction always implements the locking protocol.   The correct behavior can be reproduced on: Convert c.vmdk to raw disk image, then run on real hardware Virtual Box (version 6.1.40), the way to create the virtual machine is similar to VMware QEMU TCG (version 5.2.0), use "qemu-system-i386 -serial stdio -drive file=c.vmdk -m 256M -smp 4" Hyper-V The incorrect behavior can be reproduced on: VMware (this bug report) QEMU KVM (Linux version 5.10.0-19-amd64), use "qemu-system-i386 -serial stdio -drive file=c.vmdk -m 256M -smp 4 --enable-kvm". The KVM bug report is in https://bugzilla.kernel.org/show_bug.cgi?id=216867 . Thank you!