7 Replies Latest reply on Sep 25, 2017 5:22 AM by continuum

    ESXi 6 VM unusable slow in WS 13 beta

    continuum Guru
    User ModeratorsCommunity WarriorsvExpert

      Is anybody else using ESXi 6 VMs in Workstation ?
      I just started an existing VM created with 12.01 in the current Beta and after an hour ESXi is still here:esxi6boot.png
      In 12.01 the boot time for the same VM was about 30 seconds.
      Only difference: in the beta I had to add the parameter:
      monitor.allowLegacyCPU = "true"
      to the vmx-file.

        • 1. Re: ESXi 6 VM unusable slow in WS 13 beta
          continuum Guru
          User ModeratorsvExpertCommunity Warriors

          Making progress : 5 hours later
          esxi6boot-2.png

          • 2. Re: ESXi 6 VM unusable slow in WS 13 beta
            continuum Guru
            User ModeratorsCommunity WarriorsvExpert

            Completed boot about 12 hours later.
            Well - with a performance like that it makes no sense to continue using this Beta.

            • 3. Re: ESXi 6 VM unusable slow in WS 13 beta
              continuum Guru
              Community WarriorsvExpertUser Moderators

              Is anybody else using ESXi VMs in this beta ?
              Please let me know if you are also seeing a radical slower performance as you were used to with WS 12.

              • 4. Re: ESXi 6 VM unusable slow in WS 13 beta
                bluefirestorm Master

                I just copied out the ESXi 6.5 VM from a Windows 10 x64 (i7-6700HQ, 16GB RAM) Workstation Pro 12.5.7 to a USB 3.0 HDD.

                 

                Booted from the same machine a OpenSUSE 42.3 host via external USB 3.0 SSD, using Workstation Pro 2017 Preview, no radical difference in boot time for the ESXi 6.5 VM.

                 

                Also tried it on a Surface Pro 3 (i5-4300U) with Windows 10 x64 Creators Update, able to boot up the same ESXi 6.5 VM although I had to reduce the vCPU from 6 to 4 and memory from 10240MB to 4096MB. First time I had tried booting up an ESXi VM on a Surface Pro 3. Noticeably slower, but not the long hours you experienced, most likely because it was running on battery and the Surface Pro 3 throttles down the CPU a lot when running on battery.

                 

                On a slightly related issue, on a Penryn Core 2 Duo 9700 Ubuntu host (one generation before Nehalem Bloomfield/Lynnfield processors), which does not have EPT and does not have VMX Unrestricted Guest feature, the Tech Preview cannot boot up any VM (even with the monitor.allowLegacyCPU). I also tried with monitor.virtual_mmu and monitor.virtual_exec set to "software" (equivalent to Binary Translation mode) and it still complained about the absence of EPT. Workstation Player 12.5.x could boot up VMs without EPT with preferred mode to auto which probably would have meant monitor.virtual_mmu = "software" and monitor.virtual_exec = "hardware"

                 

                I don't have any system with Nehalem processor or any that has EPT but no Unrestricted Guest feature. The oldest with EPT would be an i7-620M and i5-580M but they are Westmere Arrandale (one generation after Nehalem) and both has EPT and Unrestricted Guest features.

                 

                I would guess it has to do with the the monitor.allowLegacyCPU and the absence of the VMX Unrestricted Guest feature that somehow slows everything to a crawl. I have no idea what the allowLegacyCPU does but from the log that you attached, it looks like approximately every 3 minutes there is some memory ballooning and a number 1048576 along with it in the log. Don't know if the 1048576 means 1MiB in real mode and swapping in and out as a result of not having the Unrestricted Guest, the boot process is unable to run bare metal inside the Intel processor?

                • 5. Re: ESXi 6 VM unusable slow in WS 13 beta
                  continuum Guru
                  Community WarriorsUser ModeratorsvExpert

                  The allowLegacyCPU parameter seems to have almost the same bad effect as running a VM without VT-support enabled in BIOS a few years ago,
                  It really is a pity that nobody from VMware answered so far.
                  Maybe they know a tweak that has less side effects

                  • 6. Re: ESXi 6 VM unusable slow in WS 13 beta
                    bluefirestorm Master

                    I pasted below the first and last non-zero swapped balloon entries, and the first with more than one VM below (because of the 2 vCPUs???). It looks like it tries to achieve the minmempct 50 (presumably meaning minimum memory percentage). So perhaps another possible tweak is the prefvmx.minVmMemPct. But it is unclear whether to change number up or down (e.g. 75 or 25) to minimise the swapping as the default is the midway point of 50.

                     

                    It does look suspiciously 1048576 is 1MiB but without a label of bytes or pages or some other term to confirm it. If it is indeed so, it is amazing that some design decision made by Intel more than 40 years ago for the 8085/8086 to maintain backward compatibility with the 8080 still is around with this real mode stuff. If there were no such backward compatibility, we wouldn't have this real mode, protected mode (in 80286), 386 mode and virtual 8086 mode (in 80386) and now IA-32e guest mode (aka unrestricted guest, which is like a virtual Core processor mode on steroids inside Westmere and later chips); and perhaps Intel would not have been that successful either. If I am not mistaken, all Intel processors would power up in real mode first (for backward compatibility reasons) and thus limited to addressable 1MiB memory; it is a matter of the OS to switch the processor to some other mode.

                     

                    2017-08-21T16:44:59.343+02:00| vmx| I125: MemSched_PowerOn: balloon minGuestSize 838860 (80% of min required size 1048576)

                     

                    2017-08-21T16:48:02.083+02:00| vmx| I125: MemSched: caller 0 numvm 1 locked pages: num 17075 max 2546688

                    2017-08-21T16:48:02.083+02:00| vmx| I125: MemSched: locked Page Limit: host 3011215 config 2554880 dynam 2656468

                    2017-08-21T16:48:02.083+02:00| vmx| I125: MemSched: minmempct 50  timestamp 793787

                    2017-08-21T16:48:02.083+02:00| vmx| I125: MemSched: VM 0 min 547136 max 1071424 shares 1048576 paged 107223 nonpaged 11125 anonymous 11723 locked 17075 touchedPct 3 dirtiedPct 3 timestamp 793787 vmResponsive is 1

                    2017-08-21T16:48:02.083+02:00| vmx| I125: MemSched: locked 17075 target 1071424 balloon 0 0 0 swapped 1 0 allocd 24 512 state 0 100

                    2017-08-21T16:48:02.083+02:00| vmx| I125: MemSched: states: 0 181 : 1 0 : 2 0 : 3 0

                    2017-08-21T16:48:02.083+02:00| vmx| I125: MemSched: Balloon enabled 1 guestType 0 maxSize 0

                     

                    2017-08-21T17:57:54.928+02:00| vmx| I125: MemSched: caller 0 numvm 2 locked pages: num 157788 max 2538496

                    2017-08-21T17:57:54.928+02:00| vmx| I125: MemSched: locked Page Limit: host 3010091 config 2554880 dynam 2784031

                    2017-08-21T17:57:54.928+02:00| vmx| I125: MemSched: minmempct 50  timestamp 797979

                    2017-08-21T17:57:54.928+02:00| vmx| I125: MemSched: VM 0 min 547136 max 1071424 shares 1048576 paged 107223 nonpaged 11125 anonymous 11723 locked 42715 touchedPct 0 dirtiedPct 0 timestamp 797979 vmResponsive is 1

                    2017-08-21T17:57:54.928+02:00| vmx| I125: MemSched: VM 1 min 169052 max 300124 shares 262144 paged 572802 nonpaged 34612 anonymous 3368 locked 115073 touchedPct 75 dirtiedPct 75 timestamp 797979 vmResponsive is 1

                    2017-08-21T17:57:54.928+02:00| vmx| I125: MemSched: locked 42715 target 1071424 balloon 0 0 0 swapped 14 0 allocd 8 512 state 0 100

                    2017-08-21T17:57:54.928+02:00| vmx| I125: MemSched: states: 0 4321 : 1 0 : 2 0 : 3 0

                    2017-08-21T17:57:54.928+02:00| vmx| I125: MemSched: Balloon enabled 1 guestType 0 maxSize 0

                     

                    2017-08-22T05:01:18.850+02:00| vmx| I125: MemSched: caller 0 numvm 2 locked pages: num 587998 max 2538496

                    2017-08-22T05:01:18.850+02:00| vmx| I125: MemSched: locked Page Limit: host 2984988 config 2554880 dynam 3145101

                    2017-08-22T05:01:18.850+02:00| vmx| I125: MemSched: minmempct 50  timestamp 837783

                    2017-08-22T05:01:18.850+02:00| vmx| I125: MemSched: VM 0 min 547136 max 1071424 shares 1048576 paged 107223 nonpaged 11125 anonymous 11723 locked 451909 touchedPct 6 dirtiedPct 2 timestamp 837783 vmResponsive is 1

                    2017-08-22T05:01:18.850+02:00| vmx| I125: MemSched: VM 1 min 169052 max 300124 shares 262144 paged 572802 nonpaged 34612 anonymous 3368 locked 136089 touchedPct 2 dirtiedPct 0 timestamp 837783 vmResponsive is 1

                    2017-08-22T05:01:18.850+02:00| vmx| I125: MemSched: locked 451909 target 1071424 balloon 0 0 0 swapped 816 0 allocd 0 512 state 0 100

                    2017-08-22T05:01:18.850+02:00| vmx| I125: MemSched: states: 0 44101 : 1 0 : 2 0 : 3 0

                    2017-08-22T05:01:18.850+02:00| vmx| I125: MemSched: Balloon enabled 1 guestType 0 maxSize 0

                    • 7. Re: ESXi 6 VM unusable slow in WS 13 beta
                      continuum Guru
                      vExpertCommunity WarriorsUser Moderators

                      @ bluefirestorm
                      Hi - sorry I was not able to test any of your suggestions.
                      I need a functional Workstation to get my work done and so I reverted to last known good version 12.01
                      Regards Ulli