I have a performance problem with VMI enabled on the ESXi and Linux systems, does anyone have any advice for me?
I have four ESXi boxes. One on a single processor 1.26 Ghz, one with dual 2.26 Ghz hyperthread (4 logical), one with dual 1.4 Ghz hyperthread (4 logical), and one on a single 3.0 Ghz hyperthread.
I don't believe any of them have Intel's VI hardware virtualization instructions. I use ESXi 3.5 U3 as the host and Gentoo 2.6.26 as the client OS.
I've tried enabling VMI on each machine and every time I do, the performance of the VM is decreased by a factor of 4 to 10 times. For example:
portage: Thu Dec 18 08:13:29 2008: 43 seconds
portage: Fri Dec 19 02:58:21 2008: 167 seconds
portage: Fri Dec 19 03:01:57 2008: 193 seconds
portage: Sat Dec 20 23:40:14 2008: 54 seconds
Here are 4 emerges on Gentoo. One with tickless and VMI disabled (43 seconds), one with Tickless enabled and VMI in ESXi and in Gentoo paravirtual VMI enabled, one with HZ=100 and VMI enabled, and the last one again with tickless and VMI disabled (54 seconds.)
Everything I find online suggests VMI (when enabled in the kernel and in the .vmx file) should increase speed or at the very least leave it at the same. I have been working on this for a week and googled to death on it. I've consulted with others running ESXi and they can reproduce the same problem when using Gentoo. I haven't tried Ubuntu Fiesty Fawn or any other OS, but I wouldn't think that would matter much.
If anyone is curious on the gentoo configs, here is a world file:
app-admin/newsyslog
app-admin/sysklogd
app-editors/nvi
app-emulation/open-vm-tools
app-portage/gentoolkit
app-portage/portage-utils
app-shells/tcsh
dev-util/strace
net-analyzer/tcpdump
net-dns/bind
net-misc/bsdwhois
net-misc/ntpclient
net-misc/telnet-bsd
sys-apps/pciutils
sys-apps/slocate
sys-boot/grub
sys-fs/jfsutils
sys-kernel/gentoo-sources
sys-process/lsof
sys-process/vixie-cron
www-client/w3m
Here is the kernel config:
CONFIG_X86_32=y
CONFIG_X86=y
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/i386_defconfig"
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_FAST_CMPXCHG_LOCAL=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_ARCH_SUPPORTS_AOUT=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_X86_SMP=y
CONFIG_X86_32_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_KTIME_SCALAR=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=18
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y
CONFIG_RELAY=y
CONFIG_NAMESPACES=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_SYSCTL_SYSCALL_CHECK=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_COMPAT_BRK=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_ANON_INODES=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLUB_DEBUG=y
CONFIG_SLUB=y
CONFIG_HAVE_OPROFILE=y
CONFIG_KPROBES=y
CONFIG_KRETPROBES=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_KMOD=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_LBD=y
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_CFQ=y
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_CLASSIC_RCU=y
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
CONFIG_X86_GENERICARCH=y
CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y
CONFIG_PARAVIRT_GUEST=y
CONFIG_VMI=y
CONFIG_PARAVIRT=y
CONFIG_X86_CYCLONE_TIMER=y
CONFIG_MPENTIUMIII=y
CONFIG_X86_GENERIC=y
CONFIG_X86_CPU=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=7
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=4
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_DMI=y
CONFIG_NR_CPUS=8
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
CONFIG_PREEMPT_VOLUNTARY=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
CONFIG_VM86=y
CONFIG_MICROCODE=y
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_HIGHMEM4G=y
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_HIGHMEM=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_RESOURCES_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MTRR=y
CONFIG_IRQBALANCE=y
CONFIG_SECCOMP=y
CONFIG_HZ_100=y
CONFIG_HZ=100
CONFIG_SCHED_HRTICK=y
CONFIG_PHYSICAL_START=0x100000
CONFIG_PHYSICAL_ALIGN=0x100000
CONFIG_HOTPLUG_CPU=y
CONFIG_COMPAT_VDSO=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_PM=y
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_SLEEP=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_PROCFS=y
CONFIG_ACPI_PROCFS_POWER=y
CONFIG_ACPI_SYSFS_POWER=y
CONFIG_ACPI_PROC_EVENT=y
CONFIG_ACPI_AC=y
CONFIG_ACPI_BATTERY=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_BLACKLIST_YEAR=2001
CONFIG_ACPI_DEBUG=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
CONFIG_CPU_FREQ_DEBUG=y
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
CONFIG_X86_ACPI_CPUFREQ=y
CONFIG_X86_POWERNOW_K8=y
CONFIG_X86_POWERNOW_K8_ACPI=y
CONFIG_X86_ACPI_CPUFREQ_PROC_INTF=y
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y
CONFIG_PCI=y
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_PCI_LEGACY=y
CONFIG_ISA_DMA_API=y
CONFIG_K8_NB=y
CONFIG_BINFMT_ELF=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_FIB_HASH=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_PNP=y
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=4096
CONFIG_HAVE_IDE=y
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDECD=y
CONFIG_BLK_DEV_IDECD_VERBOSE_ERRORS=y
CONFIG_BLK_DEV_IDEACPI=y
CONFIG_IDE_PROC_FS=y
CONFIG_IDE_GENERIC=y
CONFIG_BLK_DEV_IDEDMA_SFF=y
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
CONFIG_BLK_DEV_PIIX=y
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_NETLINK=y
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_SG=y
CONFIG_SCSI_WAIT_SCAN=m
CONFIG_SCSI_SPI_ATTRS=y
CONFIG_SCSI_FC_ATTRS=y
CONFIG_SCSI_LOWLEVEL=y
CONFIG_MD=y
CONFIG_FUSION=y
CONFIG_FUSION_SPI=y
CONFIG_FUSION_MAX_SGE=128
CONFIG_NETDEVICES=y
CONFIG_NETDEVICES_MULTIQUEUE=y
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
CONFIG_NETDEV_1000=y
CONFIG_E1000=y
CONFIG_NETCONSOLE=y
CONFIG_NETPOLL=y
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_INPUT=y
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_EVDEV=y
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_LIBPS2=y
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_DEVKMEM=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
CONFIG_HW_RANDOM=y
CONFIG_RTC=y
CONFIG_RAW_DRIVER=y
CONFIG_MAX_RAW_DEVS=256
CONFIG_DEVPORT=y
CONFIG_POWER_SUPPLY=y
CONFIG_THERMAL=y
CONFIG_SSB_POSSIBLE=y
CONFIG_SSB=y
CONFIG_SSB_SPROM=y
CONFIG_SSB_PCIHOST_POSSIBLE=y
CONFIG_SSB_PCIHOST=y
CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
CONFIG_SSB_DRIVER_PCICORE=y
CONFIG_DAB=y
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=128
CONFIG_VIDEO_SELECT=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_SOUND=y
CONFIG_SOUND_PRIME=y
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
CONFIG_USB_HID=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
CONFIG_USB_DEVICEFS=y
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_OHCI_HCD=y
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=y
CONFIG_USB_STORAGE=y
CONFIG_DMIID=y
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_JBD=y
CONFIG_FS_MBCACHE=y
CONFIG_JFS_FS=y
CONFIG_FS_POSIX_ACL=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_AUTOFS4_FS=y
CONFIG_GENERIC_ACL=y
CONFIG_ISO9660_FS=y
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_MSDOS_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=y
CONFIG_NLS_ISO8859_15=y
CONFIG_NLS_UTF8=y
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_FRAME_WARN=1024
CONFIG_MAGIC_SYSRQ=y
CONFIG_UNUSED_SYMBOLS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DETECT_SOFTLOCKUP=y
CONFIG_TIMER_STATS=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_EARLY_PRINTK=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_DOUBLEFAULT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
CONFIG_DEFAULT_IO_DELAY_TYPE=0
CONFIG_CRYPTO=y
CONFIG_CRYPTO_HW=y
CONFIG_HAVE_KVM=y
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_CRC32=y
CONFIG_PLIST=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
Hi Nick,
This is related to a kernel issue that we identified a few day back. I have posted a patch to fix the issue on this thread.
http://lkml.org/lkml/2009/2/6/339
Can you and/or anyone else seeing this problem please apply the patch from that link and see if it fixes the issue for you.
This bug was introduced in the kernel in around 2.6.25 time frame so every kernel starting from 2.6.25 will be bad.
Thanks,
Alok
I've not experienced any slower performance with VMI enabled - in fact, my performance has increased. However, as mentioned in this thread, http://communities.vmware.com/thread/185915?tstart=0, I've experienced some incredible memory overhead in VMs with VMI enabled. I'm also using Gentoo in one of those VMs, with kernel 2.6.27 and VMI enabled. You may check your memory usage on those boxes - if you're running up against a physical memory limit and ESXi is having to swap out memory, you may experience some performance issues. Other than that, I'm out of ideas...seems like there may be some work in order for VMI...
Sadly, it isn't memory.
One of the test machines has only 1 (one) VM (the slow one) and has 6 gigabytes of ram with only 512 mb activated on the VM with 96 mb of memory overhead.
I did some more research.
Emerge on the machine running as Non VMI:
bind: Sun Dec 21 08:10:52 2008: 988 seconds
I cloned the VM (ssh into ESXi, cp -r the directory and fixed up the files to refer to the new name), then emerged bind on the NON VMI machine with the VMI machine idle:
bind: Mon Dec 22 18:21:23 2008: 993 seconds
I than ran the emerge on the VMI machine with the non VMI machine idle:
bind: Mon Dec 22 18:39:46 2008: 3975 seconds
This thread says VMI doesn't require VT instructions (which non of the systems have):
I don't think so - especially given the thread Risner referenced (with a response by a VMware employee) that says that VMI does not require VT. VMI is independent of the hardware capabilities - all VMI does is provide hooks for the guest O/S through the hardware virtualization layer down to the hypervisor to speed up access to things like disk and network devices without requiring those devices to be fully emulated by virtual machines process. This should speed up network and disk I/O, but should definitely not drive CPU usage through the roof - the CPU usage should be the same and just as fast if not lower and faster.
I'll try to do some tests on my machines - I have a few ESXi machines that don't support VT technology - and see if I can come up with similar results.
Just to verify, when VMI is enabled, the guest sees it:
test# lspci
...
00:12.0 Memory controller: VMware Inc Device 0801 (rev 01)
test# dmesg | grep -i vmi
VMI: Found VMware, Inc. Hypervisor OPROM, API version 3.0, ROM version 1.0
vmi: registering clock event vmi-timer. mult=5299255 shift=22
Booting paravirtualized kernel on vmi
vmi: registering clock source khz=1263441
What, exactly, are you emerging to test out the performance? I did some disk I/O tests on my setup and saw better (in some cases much better) performance with VMI enabled. Once I run a few more tests I'll post the results. In my guest, I also have the VMware Memory controller in lspci plus the VMI PROM output from dmesg.
Your post has been moved to the Performance forum.
Dave Mishchenko
VMware Communities User Moderator
emerge net-dns/bind
That has been my test for a while, since it takes a long time and has a lot of different actions.
Everything I timed was slower on all the boxes.
Is there something about how I set my ESXi boxes up that is causing the slowness?
Okay, here's my current configuration on ESXi that I'm using for this test. I have a P4 2.4 machine with 2GB of RAM running ESXi 3.5 build 130755. It has a single Intel PRO 100 VE network card that I use for management and for connecting to an NFS-based datastore. The NFS datastore is located on an Openfiler system backed by 2Gb FC-based storage (not that that matters much, since I'm accessing it over 100Mb Ethernet).
I created a single VM and installed Gentoo - used the 2008.0 Stage 3 image, then brough it up with a world update to the latest, installed necessary packages, kernel, etc. I'm using the 2.6.26-gentoo-r4 kernel on the system. I accepted a lot of the defaults, but made sure to enable VMI in the kernel and tweaked network and storage controllers a little bit for the VMware environment. My root filesystem is ext3. The VM has 1024MB of RAM allocated for it and a 1GB swap partition on the VMDK hard drive.
For my initial tests I used the bonnie++ program for testing disk throughput. My rationale for this was that, even though it doesn't necessarily test the performance of the entire system, it should give a pretty good idea of how well VMI actually accelerates hard drive access and how CPU usage is affected by disk access - that is, the CPU overhead required for doing certain disk-intensive tasks. I tested it twice without VMI, enabled VMI and tested four times, then disabled VMI and tested twice more. The results are very, very mixed - latency for any of the tasks with VMI enabled was generally higher - in some cases significantly so. The areas where VMI tended to do better were in sequential I/O character and block-based operations per second, with the exception of rewriting. In the Random operations, VMI tended to be better, as well. The exceptions to this increased performance seem to be that latency was higher.
I then went and performed your test on emerging net-dns/bind. I saw exactly the same results as you did - ~13 minutes, give or take 30 seconds, to build bind with VMI disabled, and 30 to 60 minutes with VMI enabled.
One thing I have noticed is that VMI performance seems to deteriorate rapidly over time. When I first boot a VM, the VMI performance is very good. As I use it, it tends to get worse and worse. My initial build of bind with VMI enabled took 30 minutes (more than twice what it took without VMI). My next build of bind took 45 minutes. I'm on my third build now, and it's looking like it'll make an hour easily. Also, my memory usage for the VM is very, very odd. The Host memory usage is listed at 671 MB, the Guest memory usage at 102 MB, and the memory overhead at 542 MB. If I look at the memory stats inside the guest O/S (during my bind compile), I have 815 MB free - it's hardly using anything at all! The guest is also not using any swap - it's almost as though the VMI memory scheduler is only allowing a very small portion of the RAM to be used, maybe due to the high memory overhead of the VM?? Very weird...
I'm definitely going to do some more tests on this - I'll probably enable debugging on my VM and see if it dumps any useful information to the log file. I also have NOT installed VMware Tools, yet, which will probably be my next step.
Oh, one other odd thing...System CPU usage seems to be sitting at 50% constantly. I'm not sure why...could be because I'm running an SMP kernel on a single-CPU VM, but that doesn't usually happen. Anyway, I'm recompiling the kernel w/o SMP support (plus a couple other tweaks), so we'll see if that helps at all.
I had 2.6.26-gentoo-r4 kernel also. All my ESX boxes have quite a bit of ram (usually 4 to 8 gb) and I monitor the ram usage closely. I also noticed the several hundred mb of additional ram when running VMI, but the extra ram usage of VMI is well documented.
I never noticed the decrease in performance over time like you found. I also tried it with and without vmware tools, but frankly I only ever used the open-vm-tools. I don't tend to use the cdrom based vmware tools shipped with the ESX system in /var/lib.
CPU usage was often odd in VMI mode. When the machines was virtually idle, the CPU usage on the host would be very high. I also assumed it could be SMP, so I tried it with and without SMP in the kernel.
Since you have the same problem and others, I am wondering if something in Linux 2.6.26 is broken? Should we try 2.6.28 (released soon) or 2.6.21/2.6.22 or so?
I'm going to try out Gentoo kernels 2.6.25 and 2.6.27. If I still see issues with those, I'll try out one of the Vanilla kernels - maybe 2.6.25 or something like that.
Also, for what it's worth, I have a Gentoo box running 2.6.25-gentoo-r7 and doesn't seem to have any issues. However, it is running on a VT-enabled system, so I can't eliminate that as a possibility, yet, despite what the VMware employee said in the other post. It doesn't make sense to me that it would require VT technology, but who knows how they built VMI??
bind: Sat Dec 27 03:45:25 2008: 2726 seconds
emerge bind with VMI on a 2.6.27-gentoo-r7 kernel. Slightly faster, not sure why. But slower compared to 900 seconds with VMI disabled.
Well, my testing over the past couple of days has led me to formulate the following two theories:
New Theory #1: VMI is not made to handle VMs with high CPU usage, it's made for VMs with high network and/or disk I/O usage. Tools like bonnie++ indicate that enabling VMI does improve disk I/O throughput and CPU usage associated with that significantly. Tests that stress CPU resources tend to cause highly latent effects on both the host and the guest. Both seem to cause very high memory overhead for the guest VM.
New Theory #2: VMware still has not pinned down VMI exactly how they want it. There are still some things on both the guest kernel side and the Hypervisor end to get straightened out, and it's still in its early stages.
I hope #2 is the correct one - performance on anything CPU-intensive is abysmal! I did get moderately good performance on the emerge net-dns/bind when I set the Timer Frequency to 1000 (vs. 100 or 250). Still not better than with VMI disabled, but better than with other clock speeds. I'll probably continue to play around with some of the settings and see what I can figure out.
Interesting.
I found this http://communities.vmware.com/message/599359#599359 article. Damoiselle says that VMI was broken in 2.6.21 and that 2.6.21 was the first kernel to have VMI in vanilla. Evidently Ubuntu uses 2.6.20 versions of kernel. Something linux changed during the same time VMI was committed broke VMI.
[13.|m-599359]
Re: VMI enabled kernel Mar 15, 2007 8:57 AM
[13.|m-599317]
in response to: Noel
</div>
Indeed... interesting. I have been following this thread with some interest as I was having a VMI enabled VM that showed me some odd behavior.
At times I was getting an Out of memory error while it should not do that. There's similar VMs running on the same host without the VMI part enabled that just work and never error out.
In my case the problem VM is a ubuntu 8.04 jeos based VM with kernel 2.6.24-22-virtual
I've disabled VMI a few days ago and there hasn't been a problem since then, if it stays up like this for another week.. then this might indeed have been the problem area.
The physical CPU in the host is also a P4 based Xeon without VT support.
Right now, I'm not 100% convinced yet that the problem is indeed VMI, but it sure does smell like it...
--
Wil
_____________________________________________________
Visit the new VMware developers wiki at http://www.vi-toolkit.com
edit: my host is ESX3.5 (not embedded)