1 2 3 Previous Next 64 Replies Latest reply on Apr 16, 2009 3:38 AM by Halo2

    ESXi with VMI enabled on the VM and enabled in the Linux kernel = 10x slower

    Risner Novice

       

      I have a performance problem with VMI enabled on the ESXi and Linux systems, does anyone have any advice for me?

       

       

      I have four ESXi boxes.  One on a single processor 1.26 Ghz, one with dual 2.26 Ghz hyperthread (4 logical), one with dual 1.4 Ghz hyperthread (4 logical), and one on a single 3.0 Ghz hyperthread.

       

       

      I don't believe any of them have Intel's VI hardware virtualization instructions.  I use ESXi 3.5 U3 as the host and Gentoo 2.6.26 as the client OS.

       

       

      I've tried enabling VMI on each machine and every time I do, the performance of the VM is decreased by a factor of 4 to 10 times.  For example:

       

       

      portage: Thu Dec 18 08:13:29 2008: 43 seconds

      portage: Fri Dec 19 02:58:21 2008: 167 seconds

      portage: Fri Dec 19 03:01:57 2008: 193 seconds

      portage: Sat Dec 20 23:40:14 2008: 54 seconds

       

       

      Here are 4 emerges on Gentoo.  One with tickless and VMI disabled (43 seconds), one with Tickless enabled and VMI in ESXi and in Gentoo paravirtual VMI enabled, one with HZ=100 and VMI enabled, and the last one again with tickless and VMI disabled (54 seconds.)

       

       

      Everything I find online suggests  VMI (when enabled in the kernel and in the .vmx file) should increase speed or at the very least leave it at the same.  I have been working on this for a week and googled to death on it.  I've consulted with others running ESXi and they can reproduce the same problem when using Gentoo.  I haven't tried Ubuntu Fiesty Fawn or any other OS, but I wouldn't think that would matter much.

       

       

      If anyone is curious on the gentoo configs, here is a world file:

       

       

      app-admin/newsyslog

      app-admin/sysklogd

      app-editors/nvi

      app-emulation/open-vm-tools

      app-portage/gentoolkit

      app-portage/portage-utils

      app-shells/tcsh

      dev-util/strace

      net-analyzer/tcpdump

      net-dns/bind

      net-misc/bsdwhois

      net-misc/ntpclient

      net-misc/telnet-bsd

      sys-apps/pciutils

      sys-apps/slocate

      sys-boot/grub

      sys-fs/jfsutils

      sys-kernel/gentoo-sources

      sys-process/lsof

      sys-process/vixie-cron

      www-client/w3m

       

       

      Here is the kernel config:

       

       

      CONFIG_X86_32=y

      CONFIG_X86=y

      CONFIG_ARCH_DEFCONFIG="arch/x86/configs/i386_defconfig"

      CONFIG_GENERIC_TIME=y

      CONFIG_GENERIC_CMOS_UPDATE=y

      CONFIG_CLOCKSOURCE_WATCHDOG=y

      CONFIG_GENERIC_CLOCKEVENTS=y

      CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y

      CONFIG_LOCKDEP_SUPPORT=y

      CONFIG_STACKTRACE_SUPPORT=y

      CONFIG_HAVE_LATENCYTOP_SUPPORT=y

      CONFIG_FAST_CMPXCHG_LOCAL=y

      CONFIG_MMU=y

      CONFIG_ZONE_DMA=y

      CONFIG_GENERIC_ISA_DMA=y

      CONFIG_GENERIC_IOMAP=y

      CONFIG_GENERIC_BUG=y

      CONFIG_GENERIC_HWEIGHT=y

      CONFIG_ARCH_MAY_HAVE_PC_FDC=y

      CONFIG_RWSEM_XCHGADD_ALGORITHM=y

      CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y

      CONFIG_GENERIC_CALIBRATE_DELAY=y

      CONFIG_ARCH_HAS_CPU_RELAX=y

      CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y

      CONFIG_HAVE_SETUP_PER_CPU_AREA=y

      CONFIG_ARCH_HIBERNATION_POSSIBLE=y

      CONFIG_ARCH_SUSPEND_POSSIBLE=y

      CONFIG_ARCH_POPULATES_NODE_MAP=y

      CONFIG_ARCH_SUPPORTS_AOUT=y

      CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y

      CONFIG_GENERIC_HARDIRQS=y

      CONFIG_GENERIC_IRQ_PROBE=y

      CONFIG_GENERIC_PENDING_IRQ=y

      CONFIG_X86_SMP=y

      CONFIG_X86_32_SMP=y

      CONFIG_X86_HT=y

      CONFIG_X86_BIOS_REBOOT=y

      CONFIG_X86_TRAMPOLINE=y

      CONFIG_KTIME_SCALAR=y

      CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"

      CONFIG_EXPERIMENTAL=y

      CONFIG_LOCK_KERNEL=y

      CONFIG_INIT_ENV_ARG_LIMIT=32

      CONFIG_LOCALVERSION=""

      CONFIG_LOCALVERSION_AUTO=y

      CONFIG_SWAP=y

      CONFIG_SYSVIPC=y

      CONFIG_SYSVIPC_SYSCTL=y

      CONFIG_POSIX_MQUEUE=y

      CONFIG_IKCONFIG=y

      CONFIG_IKCONFIG_PROC=y

      CONFIG_LOG_BUF_SHIFT=18

      CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y

      CONFIG_SYSFS_DEPRECATED=y

      CONFIG_SYSFS_DEPRECATED_V2=y

      CONFIG_RELAY=y

      CONFIG_NAMESPACES=y

      CONFIG_BLK_DEV_INITRD=y

      CONFIG_INITRAMFS_SOURCE=""

      CONFIG_CC_OPTIMIZE_FOR_SIZE=y

      CONFIG_SYSCTL=y

      CONFIG_UID16=y

      CONFIG_SYSCTL_SYSCALL=y

      CONFIG_SYSCTL_SYSCALL_CHECK=y

      CONFIG_KALLSYMS=y

      CONFIG_KALLSYMS_ALL=y

      CONFIG_HOTPLUG=y

      CONFIG_PRINTK=y

      CONFIG_BUG=y

      CONFIG_ELF_CORE=y

      CONFIG_PCSPKR_PLATFORM=y

      CONFIG_COMPAT_BRK=y

      CONFIG_BASE_FULL=y

      CONFIG_FUTEX=y

      CONFIG_ANON_INODES=y

      CONFIG_EPOLL=y

      CONFIG_SIGNALFD=y

      CONFIG_TIMERFD=y

      CONFIG_EVENTFD=y

      CONFIG_SHMEM=y

      CONFIG_VM_EVENT_COUNTERS=y

      CONFIG_SLUB_DEBUG=y

      CONFIG_SLUB=y

      CONFIG_HAVE_OPROFILE=y

      CONFIG_KPROBES=y

      CONFIG_KRETPROBES=y

      CONFIG_HAVE_KPROBES=y

      CONFIG_HAVE_KRETPROBES=y

      CONFIG_PROC_PAGE_MONITOR=y

      CONFIG_SLABINFO=y

      CONFIG_RT_MUTEXES=y

      CONFIG_BASE_SMALL=0

      CONFIG_MODULES=y

      CONFIG_MODULE_UNLOAD=y

      CONFIG_MODULE_FORCE_UNLOAD=y

      CONFIG_KMOD=y

      CONFIG_STOP_MACHINE=y

      CONFIG_BLOCK=y

      CONFIG_LBD=y

      CONFIG_IOSCHED_NOOP=y

      CONFIG_IOSCHED_AS=y

      CONFIG_IOSCHED_DEADLINE=y

      CONFIG_IOSCHED_CFQ=y

      CONFIG_DEFAULT_CFQ=y

      CONFIG_DEFAULT_IOSCHED="cfq"

      CONFIG_CLASSIC_RCU=y

      CONFIG_TICK_ONESHOT=y

      CONFIG_NO_HZ=y

      CONFIG_HIGH_RES_TIMERS=y

      CONFIG_GENERIC_CLOCKEVENTS_BUILD=y

      CONFIG_SMP=y

      CONFIG_X86_GENERICARCH=y

      CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y

      CONFIG_PARAVIRT_GUEST=y

      CONFIG_VMI=y

      CONFIG_PARAVIRT=y

      CONFIG_X86_CYCLONE_TIMER=y

      CONFIG_MPENTIUMIII=y

      CONFIG_X86_GENERIC=y

      CONFIG_X86_CPU=y

      CONFIG_X86_CMPXCHG=y

      CONFIG_X86_L1_CACHE_SHIFT=7

      CONFIG_X86_XADD=y

      CONFIG_X86_WP_WORKS_OK=y

      CONFIG_X86_INVLPG=y

      CONFIG_X86_BSWAP=y

      CONFIG_X86_POPAD_OK=y

      CONFIG_X86_GOOD_APIC=y

      CONFIG_X86_INTEL_USERCOPY=y

      CONFIG_X86_USE_PPRO_CHECKSUM=y

      CONFIG_X86_TSC=y

      CONFIG_X86_CMOV=y

      CONFIG_X86_MINIMUM_CPU_FAMILY=4

      CONFIG_X86_DEBUGCTLMSR=y

      CONFIG_DMI=y

      CONFIG_NR_CPUS=8

      CONFIG_SCHED_SMT=y

      CONFIG_SCHED_MC=y

      CONFIG_PREEMPT_VOLUNTARY=y

      CONFIG_X86_LOCAL_APIC=y

      CONFIG_X86_IO_APIC=y

      CONFIG_X86_MCE=y

      CONFIG_X86_MCE_NONFATAL=y

      CONFIG_VM86=y

      CONFIG_MICROCODE=y

      CONFIG_MICROCODE_OLD_INTERFACE=y

      CONFIG_X86_MSR=y

      CONFIG_X86_CPUID=y

      CONFIG_HIGHMEM4G=y

      CONFIG_PAGE_OFFSET=0xC0000000

      CONFIG_HIGHMEM=y

      CONFIG_SELECT_MEMORY_MODEL=y

      CONFIG_FLATMEM_MANUAL=y

      CONFIG_FLATMEM=y

      CONFIG_FLAT_NODE_MEM_MAP=y

      CONFIG_PAGEFLAGS_EXTENDED=y

      CONFIG_SPLIT_PTLOCK_CPUS=4

      CONFIG_RESOURCES_64BIT=y

      CONFIG_ZONE_DMA_FLAG=1

      CONFIG_BOUNCE=y

      CONFIG_VIRT_TO_BUS=y

      CONFIG_MTRR=y

      CONFIG_IRQBALANCE=y

      CONFIG_SECCOMP=y

      CONFIG_HZ_100=y

      CONFIG_HZ=100

      CONFIG_SCHED_HRTICK=y

      CONFIG_PHYSICAL_START=0x100000

      CONFIG_PHYSICAL_ALIGN=0x100000

      CONFIG_HOTPLUG_CPU=y

      CONFIG_COMPAT_VDSO=y

      CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

      CONFIG_PM=y

      CONFIG_PM_SLEEP_SMP=y

      CONFIG_PM_SLEEP=y

      CONFIG_SUSPEND=y

      CONFIG_SUSPEND_FREEZER=y

      CONFIG_ACPI=y

      CONFIG_ACPI_SLEEP=y

      CONFIG_ACPI_PROCFS=y

      CONFIG_ACPI_PROCFS_POWER=y

      CONFIG_ACPI_SYSFS_POWER=y

      CONFIG_ACPI_PROC_EVENT=y

      CONFIG_ACPI_AC=y

      CONFIG_ACPI_BATTERY=y

      CONFIG_ACPI_BUTTON=y

      CONFIG_ACPI_FAN=y

      CONFIG_ACPI_DOCK=y

      CONFIG_ACPI_PROCESSOR=y

      CONFIG_ACPI_HOTPLUG_CPU=y

      CONFIG_ACPI_THERMAL=y

      CONFIG_ACPI_BLACKLIST_YEAR=2001

      CONFIG_ACPI_DEBUG=y

      CONFIG_ACPI_EC=y

      CONFIG_ACPI_POWER=y

      CONFIG_ACPI_SYSTEM=y

      CONFIG_X86_PM_TIMER=y

      CONFIG_ACPI_CONTAINER=y

      CONFIG_CPU_FREQ=y

      CONFIG_CPU_FREQ_TABLE=y

      CONFIG_CPU_FREQ_DEBUG=y

      CONFIG_CPU_FREQ_STAT=y

      CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y

      CONFIG_CPU_FREQ_GOV_PERFORMANCE=y

      CONFIG_CPU_FREQ_GOV_USERSPACE=y

      CONFIG_CPU_FREQ_GOV_ONDEMAND=y

      CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y

      CONFIG_X86_ACPI_CPUFREQ=y

      CONFIG_X86_POWERNOW_K8=y

      CONFIG_X86_POWERNOW_K8_ACPI=y

      CONFIG_X86_ACPI_CPUFREQ_PROC_INTF=y

      CONFIG_CPU_IDLE=y

      CONFIG_CPU_IDLE_GOV_LADDER=y

      CONFIG_CPU_IDLE_GOV_MENU=y

      CONFIG_PCI=y

      CONFIG_PCI_GOANY=y

      CONFIG_PCI_BIOS=y

      CONFIG_PCI_DIRECT=y

      CONFIG_PCI_MMCONFIG=y

      CONFIG_PCI_DOMAINS=y

      CONFIG_ARCH_SUPPORTS_MSI=y

      CONFIG_PCI_MSI=y

      CONFIG_PCI_LEGACY=y

      CONFIG_ISA_DMA_API=y

      CONFIG_K8_NB=y

      CONFIG_BINFMT_ELF=y

      CONFIG_NET=y

      CONFIG_PACKET=y

      CONFIG_UNIX=y

      CONFIG_INET=y

      CONFIG_IP_MULTICAST=y

      CONFIG_IP_FIB_HASH=y

      CONFIG_IP_PNP=y

      CONFIG_IP_PNP_DHCP=y

      CONFIG_INET_DIAG=y

      CONFIG_INET_TCP_DIAG=y

      CONFIG_TCP_CONG_CUBIC=y

      CONFIG_DEFAULT_TCP_CONG="cubic"

      CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"

      CONFIG_STANDALONE=y

      CONFIG_PREVENT_FIRMWARE_BUILD=y

      CONFIG_FW_LOADER=y

      CONFIG_PNP=y

      CONFIG_PNPACPI=y

      CONFIG_BLK_DEV=y

      CONFIG_BLK_DEV_FD=y

      CONFIG_BLK_DEV_LOOP=y

      CONFIG_BLK_DEV_RAM=y

      CONFIG_BLK_DEV_RAM_COUNT=16

      CONFIG_BLK_DEV_RAM_SIZE=4096

      CONFIG_HAVE_IDE=y

      CONFIG_IDE=y

      CONFIG_BLK_DEV_IDE=y

      CONFIG_BLK_DEV_IDEDISK=y

      CONFIG_IDEDISK_MULTI_MODE=y

      CONFIG_BLK_DEV_IDECD=y

      CONFIG_BLK_DEV_IDECD_VERBOSE_ERRORS=y

      CONFIG_BLK_DEV_IDEACPI=y

      CONFIG_IDE_PROC_FS=y

      CONFIG_IDE_GENERIC=y

      CONFIG_BLK_DEV_IDEDMA_SFF=y

      CONFIG_BLK_DEV_IDEPCI=y

      CONFIG_BLK_DEV_IDEDMA_PCI=y

      CONFIG_BLK_DEV_PIIX=y

      CONFIG_BLK_DEV_IDEDMA=y

      CONFIG_SCSI=y

      CONFIG_SCSI_DMA=y

      CONFIG_SCSI_NETLINK=y

      CONFIG_BLK_DEV_SD=y

      CONFIG_CHR_DEV_SG=y

      CONFIG_SCSI_WAIT_SCAN=m

      CONFIG_SCSI_SPI_ATTRS=y

      CONFIG_SCSI_FC_ATTRS=y

      CONFIG_SCSI_LOWLEVEL=y

      CONFIG_MD=y

      CONFIG_FUSION=y

      CONFIG_FUSION_SPI=y

      CONFIG_FUSION_MAX_SGE=128

      CONFIG_NETDEVICES=y

      CONFIG_NETDEVICES_MULTIQUEUE=y

      CONFIG_NET_ETHERNET=y

      CONFIG_MII=y

      CONFIG_NETDEV_1000=y

      CONFIG_E1000=y

      CONFIG_NETCONSOLE=y

      CONFIG_NETPOLL=y

      CONFIG_NET_POLL_CONTROLLER=y

      CONFIG_INPUT=y

      CONFIG_INPUT_MOUSEDEV=y

      CONFIG_INPUT_MOUSEDEV_PSAUX=y

      CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024

      CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768

      CONFIG_INPUT_EVDEV=y

      CONFIG_INPUT_KEYBOARD=y

      CONFIG_KEYBOARD_ATKBD=y

      CONFIG_INPUT_MOUSE=y

      CONFIG_MOUSE_PS2=y

      CONFIG_MOUSE_PS2_ALPS=y

      CONFIG_MOUSE_PS2_LOGIPS2PP=y

      CONFIG_MOUSE_PS2_SYNAPTICS=y

      CONFIG_MOUSE_PS2_LIFEBOOK=y

      CONFIG_MOUSE_PS2_TRACKPOINT=y

      CONFIG_SERIO=y

      CONFIG_SERIO_I8042=y

      CONFIG_SERIO_LIBPS2=y

      CONFIG_VT=y

      CONFIG_VT_CONSOLE=y

      CONFIG_HW_CONSOLE=y

      CONFIG_DEVKMEM=y

      CONFIG_SERIAL_8250=y

      CONFIG_SERIAL_8250_CONSOLE=y

      CONFIG_FIX_EARLYCON_MEM=y

      CONFIG_SERIAL_8250_PCI=y

      CONFIG_SERIAL_8250_PNP=y

      CONFIG_SERIAL_8250_NR_UARTS=4

      CONFIG_SERIAL_8250_RUNTIME_UARTS=4

      CONFIG_SERIAL_CORE=y

      CONFIG_SERIAL_CORE_CONSOLE=y

      CONFIG_UNIX98_PTYS=y

      CONFIG_LEGACY_PTYS=y

      CONFIG_LEGACY_PTY_COUNT=256

      CONFIG_HW_RANDOM=y

      CONFIG_RTC=y

      CONFIG_RAW_DRIVER=y

      CONFIG_MAX_RAW_DEVS=256

      CONFIG_DEVPORT=y

      CONFIG_POWER_SUPPLY=y

      CONFIG_THERMAL=y

      CONFIG_SSB_POSSIBLE=y

      CONFIG_SSB=y

      CONFIG_SSB_SPROM=y

      CONFIG_SSB_PCIHOST_POSSIBLE=y

      CONFIG_SSB_PCIHOST=y

      CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y

      CONFIG_SSB_DRIVER_PCICORE=y

      CONFIG_DAB=y

      CONFIG_AGP=y

      CONFIG_AGP_AMD64=y

      CONFIG_AGP_INTEL=y

      CONFIG_VGA_CONSOLE=y

      CONFIG_VGACON_SOFT_SCROLLBACK=y

      CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=128

      CONFIG_VIDEO_SELECT=y

      CONFIG_DUMMY_CONSOLE=y

      CONFIG_SOUND=y

      CONFIG_SOUND_PRIME=y

      CONFIG_HID_SUPPORT=y

      CONFIG_HID=y

      CONFIG_USB_HID=y

      CONFIG_USB_SUPPORT=y

      CONFIG_USB_ARCH_HAS_HCD=y

      CONFIG_USB_ARCH_HAS_OHCI=y

      CONFIG_USB_ARCH_HAS_EHCI=y

      CONFIG_USB=y

      CONFIG_USB_DEVICEFS=y

      CONFIG_USB_EHCI_HCD=y

      CONFIG_USB_OHCI_HCD=y

      CONFIG_USB_OHCI_LITTLE_ENDIAN=y

      CONFIG_USB_UHCI_HCD=y

      CONFIG_USB_STORAGE=y

      CONFIG_DMIID=y

      CONFIG_EXT2_FS=y

      CONFIG_EXT2_FS_XATTR=y

      CONFIG_EXT2_FS_POSIX_ACL=y

      CONFIG_EXT3_FS=y

      CONFIG_EXT3_FS_XATTR=y

      CONFIG_EXT3_FS_POSIX_ACL=y

      CONFIG_JBD=y

      CONFIG_FS_MBCACHE=y

      CONFIG_JFS_FS=y

      CONFIG_FS_POSIX_ACL=y

      CONFIG_DNOTIFY=y

      CONFIG_INOTIFY=y

      CONFIG_INOTIFY_USER=y

      CONFIG_AUTOFS4_FS=y

      CONFIG_GENERIC_ACL=y

      CONFIG_ISO9660_FS=y

      CONFIG_FAT_FS=y

      CONFIG_MSDOS_FS=y

      CONFIG_VFAT_FS=y

      CONFIG_FAT_DEFAULT_CODEPAGE=437

      CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"

      CONFIG_PROC_FS=y

      CONFIG_PROC_KCORE=y

      CONFIG_PROC_SYSCTL=y

      CONFIG_SYSFS=y

      CONFIG_TMPFS=y

      CONFIG_TMPFS_POSIX_ACL=y

      CONFIG_HUGETLBFS=y

      CONFIG_HUGETLB_PAGE=y

      CONFIG_MSDOS_PARTITION=y

      CONFIG_NLS=y

      CONFIG_NLS_DEFAULT="iso8859-1"

      CONFIG_NLS_CODEPAGE_437=y

      CONFIG_NLS_ASCII=y

      CONFIG_NLS_ISO8859_1=y

      CONFIG_NLS_ISO8859_15=y

      CONFIG_NLS_UTF8=y

      CONFIG_TRACE_IRQFLAGS_SUPPORT=y

      CONFIG_ENABLE_WARN_DEPRECATED=y

      CONFIG_FRAME_WARN=1024

      CONFIG_MAGIC_SYSRQ=y

      CONFIG_UNUSED_SYMBOLS=y

      CONFIG_DEBUG_KERNEL=y

      CONFIG_DETECT_SOFTLOCKUP=y

      CONFIG_TIMER_STATS=y

      CONFIG_DEBUG_BUGVERBOSE=y

      CONFIG_HAVE_ARCH_KGDB=y

      CONFIG_EARLY_PRINTK=y

      CONFIG_DEBUG_STACKOVERFLOW=y

      CONFIG_X86_FIND_SMP_CONFIG=y

      CONFIG_X86_MPPARSE=y

      CONFIG_DOUBLEFAULT=y

      CONFIG_IO_DELAY_TYPE_0X80=0

      CONFIG_IO_DELAY_TYPE_0XED=1

      CONFIG_IO_DELAY_TYPE_UDELAY=2

      CONFIG_IO_DELAY_TYPE_NONE=3

      CONFIG_IO_DELAY_0X80=y

      CONFIG_DEFAULT_IO_DELAY_TYPE=0

      CONFIG_CRYPTO=y

      CONFIG_CRYPTO_HW=y

      CONFIG_HAVE_KVM=y

      CONFIG_BITREVERSE=y

      CONFIG_GENERIC_FIND_FIRST_BIT=y

      CONFIG_GENERIC_FIND_NEXT_BIT=y

      CONFIG_CRC32=y

      CONFIG_PLIST=y

      CONFIG_HAS_IOMEM=y

      CONFIG_HAS_IOPORT=y

      CONFIG_HAS_DMA=y

       

       

       

        • 1. Re: ESXi with VMI enabled on the VM and enabled in the Linux kernel = 10x slower
          nick.couchman Champion

          I've not experienced any slower performance with VMI enabled - in fact, my performance has increased.  However, as mentioned in this thread, http://communities.vmware.com/thread/185915?tstart=0, I've experienced some incredible memory overhead in VMs with VMI enabled.  I'm also using Gentoo in one of those VMs, with kernel 2.6.27 and VMI enabled.  You may check your memory usage on those boxes - if you're running up against a physical memory limit and ESXi is having to swap out memory, you may experience some performance issues.  Other than that, I'm out of ideas...seems like there may be some work in order for VMI...

          • 2. Re: ESXi with VMI enabled on the VM and enabled in the Linux kernel = 10x slower
            Risner Novice

             

            Sadly, it isn't memory.

             

             

            One of the test machines has only 1 (one) VM  (the slow one) and has 6 gigabytes of ram with only 512 mb activated on the VM with 96 mb of memory overhead.

             

             

             

             

             

            • 3. Re: ESXi with VMI enabled on the VM and enabled in the Linux kernel = 10x slower
              jasonlitka Hot Shot

              VMI only works if your CPU has Virtualization instructions (such as Intel's VT).

               

              Jason Litka

              http://www.jasonlitka.com

              • 4. Re: ESXi with VMI enabled on the VM and enabled in the Linux kernel = 10x slower
                Risner Novice

                I did some more research.

                 

                Emerge on the machine running as Non VMI:

                bind: Sun Dec 21 08:10:52 2008: 988 seconds

                 

                 

                 

                I cloned the VM (ssh into ESXi, cp -r the directory and fixed up the files to refer to the new name), then emerged bind on the NON VMI machine with the VMI machine idle:

                bind: Mon Dec 22 18:21:23 2008: 993 seconds

                 

                 

                 

                I than ran the emerge on the VMI machine with the non VMI machine idle:

                bind: Mon Dec 22 18:39:46 2008: 3975 seconds

                 

                 

                This thread says VMI doesn't require VT instructions (which non of the systems have):

                http://communities.vmware.com/thread/163521

                • 5. Re: ESXi with VMI enabled on the VM and enabled in the Linux kernel = 10x slower
                  nick.couchman Champion

                   

                  I don't think so - especially given the thread Risner referenced (with a response by a VMware employee) that says that VMI does not require VT.  VMI is independent of the hardware capabilities - all VMI does is provide hooks for the guest O/S through the hardware virtualization layer down to the hypervisor to speed up access to things like disk and network devices without requiring those devices to be fully emulated by virtual machines process.  This should speed up network and disk I/O, but should definitely not drive CPU usage through the roof - the CPU usage should be the same and just as fast if not lower and faster.

                   

                   

                  • 6. Re: ESXi with VMI enabled on the VM and enabled in the Linux kernel = 10x slower
                    nick.couchman Champion

                    I'll try to do some tests on my machines - I have a few ESXi machines that don't support VT technology - and see if I can come up with similar results.

                    1 person found this helpful
                    • 7. Re: ESXi with VMI enabled on the VM and enabled in the Linux kernel = 10x slower
                      Risner Novice

                       

                      Just to verify, when VMI is enabled, the guest sees it:

                       

                       

                      test# lspci

                      ...

                      00:12.0 Memory controller: VMware Inc Device 0801 (rev 01)

                      test# dmesg | grep -i vmi

                      VMI: Found VMware, Inc. Hypervisor OPROM, API version 3.0, ROM version 1.0

                      vmi: registering clock event vmi-timer. mult=5299255 shift=22

                      Booting paravirtualized kernel on vmi

                      vmi: registering clock source khz=1263441

                       

                       

                      • 8. Re: ESXi with VMI enabled on the VM and enabled in the Linux kernel = 10x slower
                        nick.couchman Champion

                        What, exactly, are you emerging to test out the performance?  I did some disk I/O tests on my setup and saw better (in some cases much better) performance with VMI enabled.  Once I run a few more tests I'll post the results.  In my guest, I also have the VMware Memory controller in lspci plus the VMI PROM output from dmesg.

                        • 9. Re: ESXi with VMI enabled on the VM and enabled in the Linux kernel = 10x slower
                          Dave.Mishchenko Guru

                          Your post has been moved to the Performance forum.

                           

                          Dave Mishchenko

                          VMware Communities User Moderator

                          • 10. Re: ESXi with VMI enabled on the VM and enabled in the Linux kernel = 10x slower
                            Risner Novice

                             

                            emerge net-dns/bind

                             

                             

                            That has been my test for a while, since it takes a long time and has a lot of different actions.

                             

                             

                            Everything I timed was slower on all the boxes.

                             

                             

                            Is there something about how I set my ESXi boxes up that is causing the slowness?

                             

                             

                            • 11. Re: ESXi with VMI enabled on the VM and enabled in the Linux kernel = 10x slower
                              nick.couchman Champion

                               

                              Okay, here's my current configuration on ESXi that I'm using for this test.  I have a P4 2.4 machine with 2GB of RAM running ESXi 3.5 build 130755.  It has a single Intel PRO 100 VE network card that I use for management and for connecting to an NFS-based datastore.  The NFS datastore is located on an Openfiler system backed by 2Gb FC-based storage (not that that matters much, since I'm accessing it over 100Mb Ethernet).

                               

                               

                              I created a single VM and installed Gentoo - used the 2008.0 Stage 3 image, then brough it up with a world update to the latest, installed necessary packages, kernel, etc.  I'm using the 2.6.26-gentoo-r4 kernel on the system.  I accepted a lot of the defaults, but made sure to enable VMI in the kernel and tweaked network and storage controllers a little bit for the VMware environment.   My root filesystem is ext3.  The VM has 1024MB of RAM allocated for it and a 1GB swap partition on the VMDK hard drive.

                               

                               

                              For my initial tests I used the bonnie++ program for testing disk throughput.  My rationale for this was that, even though it doesn't necessarily test the performance of the entire system, it should give a pretty good idea of how well VMI actually accelerates hard drive access and how CPU usage is affected by disk access - that is, the CPU overhead required for doing certain disk-intensive tasks.  I tested it twice without VMI, enabled VMI and tested four times, then disabled VMI and tested twice more.  The results are very, very mixed - latency for any of the tasks with VMI enabled was generally higher - in some cases significantly so.  The areas where VMI tended to do better were in sequential I/O character and block-based operations per second, with the exception of rewriting.  In the Random operations, VMI tended to be better, as well.  The exceptions to this increased performance seem to be that latency was higher. 

                               

                               

                              I then went and performed your test on emerging net-dns/bind.  I saw exactly the same results as you did - ~13 minutes, give or take 30 seconds, to build bind with VMI disabled, and 30 to 60 minutes with VMI enabled.

                               

                               

                              One thing I have noticed is that VMI performance seems to deteriorate rapidly over time.  When I first boot a VM, the VMI performance is very good.  As I use it, it tends to get worse and worse.  My initial build of bind with VMI enabled took 30 minutes (more than twice what it took without VMI).  My next build of bind took 45 minutes.  I'm on my third build now, and it's looking like it'll make an hour easily.  Also, my memory usage for the VM is very, very odd.  The Host memory usage is listed at 671 MB, the Guest memory usage at 102 MB, and the memory overhead at 542 MB.  If I look at the memory stats inside the guest O/S (during my bind compile), I have 815 MB free - it's hardly using anything at all! The guest is also not using any swap - it's almost as though the VMI memory scheduler is only allowing a very small portion of the RAM to be used, maybe due to the high memory overhead of the VM??  Very weird...

                               

                               

                              I'm definitely going to do some more tests on this - I'll probably enable debugging on my VM and see if it dumps any useful information to the log file.  I also have NOT installed VMware Tools, yet, which will probably be my next step.   

                               

                               

                              • 12. Re: ESXi with VMI enabled on the VM and enabled in the Linux kernel = 10x slower
                                nick.couchman Champion

                                Oh, one other odd thing...System CPU usage seems to be sitting at 50% constantly.  I'm not sure why...could be because I'm running an SMP kernel on a single-CPU VM, but that doesn't usually happen.  Anyway, I'm recompiling the kernel w/o SMP support (plus a couple other tweaks), so we'll see if that helps at all.

                                • 13. Re: ESXi with VMI enabled on the VM and enabled in the Linux kernel = 10x slower
                                  Risner Novice

                                   

                                   

                                   

                                   

                                  I had 2.6.26-gentoo-r4 kernel also.  All my ESX boxes have quite a bit of ram (usually 4 to 8 gb) and I monitor the ram usage closely.  I also noticed the several hundred mb of additional ram when running VMI, but the extra ram usage of VMI is well documented.

                                   

                                   

                                  I never noticed the decrease in performance over time like you found.  I also tried it with and without vmware tools, but frankly I only ever used the open-vm-tools.  I don't tend to use the cdrom based vmware tools shipped with the ESX system in /var/lib.

                                   

                                   

                                  CPU usage was often odd in VMI mode.  When the machines was virtually idle, the CPU usage on the host would be very high.  I also assumed it could be SMP, so I tried it with and without SMP in the kernel.

                                   

                                  Since you have the same problem and others, I am wondering if something in Linux 2.6.26 is broken?  Should we try 2.6.28 (released soon) or 2.6.21/2.6.22 or so?

                                  • 14. Re: ESXi with VMI enabled on the VM and enabled in the Linux kernel = 10x slower
                                    nick.couchman Champion

                                    I'm going to try out Gentoo kernels 2.6.25 and 2.6.27.  If I still see issues with those, I'll try out one of the Vanilla kernels - maybe 2.6.25 or something like that.

                                     

                                     

                                    Also, for what it's worth, I have a Gentoo box running 2.6.25-gentoo-r7 and doesn't seem to have any issues.  However, it is running on a VT-enabled system, so I can't eliminate that as a possibility, yet, despite what the VMware employee said in the other post.  It doesn't make sense to me that it would require VT technology, but who knows how they built VMI??

                                    1 2 3 Previous Next