VMware Cloud Community
Risner
Contributor
Contributor
Jump to solution

ESXi with VMI enabled on the VM and enabled in the Linux kernel = 10x slower

I have a performance problem with VMI enabled on the ESXi and Linux systems, does anyone have any advice for me?

I have four ESXi boxes. One on a single processor 1.26 Ghz, one with dual 2.26 Ghz hyperthread (4 logical), one with dual 1.4 Ghz hyperthread (4 logical), and one on a single 3.0 Ghz hyperthread.

I don't believe any of them have Intel's VI hardware virtualization instructions. I use ESXi 3.5 U3 as the host and Gentoo 2.6.26 as the client OS.

I've tried enabling VMI on each machine and every time I do, the performance of the VM is decreased by a factor of 4 to 10 times. For example:

portage: Thu Dec 18 08:13:29 2008: 43 seconds

portage: Fri Dec 19 02:58:21 2008: 167 seconds

portage: Fri Dec 19 03:01:57 2008: 193 seconds

portage: Sat Dec 20 23:40:14 2008: 54 seconds

Here are 4 emerges on Gentoo. One with tickless and VMI disabled (43 seconds), one with Tickless enabled and VMI in ESXi and in Gentoo paravirtual VMI enabled, one with HZ=100 and VMI enabled, and the last one again with tickless and VMI disabled (54 seconds.)

Everything I find online suggests VMI (when enabled in the kernel and in the .vmx file) should increase speed or at the very least leave it at the same. I have been working on this for a week and googled to death on it. I've consulted with others running ESXi and they can reproduce the same problem when using Gentoo. I haven't tried Ubuntu Fiesty Fawn or any other OS, but I wouldn't think that would matter much.

If anyone is curious on the gentoo configs, here is a world file:

app-admin/newsyslog

app-admin/sysklogd

app-editors/nvi

app-emulation/open-vm-tools

app-portage/gentoolkit

app-portage/portage-utils

app-shells/tcsh

dev-util/strace

net-analyzer/tcpdump

net-dns/bind

net-misc/bsdwhois

net-misc/ntpclient

net-misc/telnet-bsd

sys-apps/pciutils

sys-apps/slocate

sys-boot/grub

sys-fs/jfsutils

sys-kernel/gentoo-sources

sys-process/lsof

sys-process/vixie-cron

www-client/w3m

Here is the kernel config:

CONFIG_X86_32=y

CONFIG_X86=y

CONFIG_ARCH_DEFCONFIG="arch/x86/configs/i386_defconfig"

CONFIG_GENERIC_TIME=y

CONFIG_GENERIC_CMOS_UPDATE=y

CONFIG_CLOCKSOURCE_WATCHDOG=y

CONFIG_GENERIC_CLOCKEVENTS=y

CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y

CONFIG_LOCKDEP_SUPPORT=y

CONFIG_STACKTRACE_SUPPORT=y

CONFIG_HAVE_LATENCYTOP_SUPPORT=y

CONFIG_FAST_CMPXCHG_LOCAL=y

CONFIG_MMU=y

CONFIG_ZONE_DMA=y

CONFIG_GENERIC_ISA_DMA=y

CONFIG_GENERIC_IOMAP=y

CONFIG_GENERIC_BUG=y

CONFIG_GENERIC_HWEIGHT=y

CONFIG_ARCH_MAY_HAVE_PC_FDC=y

CONFIG_RWSEM_XCHGADD_ALGORITHM=y

CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y

CONFIG_GENERIC_CALIBRATE_DELAY=y

CONFIG_ARCH_HAS_CPU_RELAX=y

CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y

CONFIG_HAVE_SETUP_PER_CPU_AREA=y

CONFIG_ARCH_HIBERNATION_POSSIBLE=y

CONFIG_ARCH_SUSPEND_POSSIBLE=y

CONFIG_ARCH_POPULATES_NODE_MAP=y

CONFIG_ARCH_SUPPORTS_AOUT=y

CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y

CONFIG_GENERIC_HARDIRQS=y

CONFIG_GENERIC_IRQ_PROBE=y

CONFIG_GENERIC_PENDING_IRQ=y

CONFIG_X86_SMP=y

CONFIG_X86_32_SMP=y

CONFIG_X86_HT=y

CONFIG_X86_BIOS_REBOOT=y

CONFIG_X86_TRAMPOLINE=y

CONFIG_KTIME_SCALAR=y

CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"

CONFIG_EXPERIMENTAL=y

CONFIG_LOCK_KERNEL=y

CONFIG_INIT_ENV_ARG_LIMIT=32

CONFIG_LOCALVERSION=""

CONFIG_LOCALVERSION_AUTO=y

CONFIG_SWAP=y

CONFIG_SYSVIPC=y

CONFIG_SYSVIPC_SYSCTL=y

CONFIG_POSIX_MQUEUE=y

CONFIG_IKCONFIG=y

CONFIG_IKCONFIG_PROC=y

CONFIG_LOG_BUF_SHIFT=18

CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y

CONFIG_SYSFS_DEPRECATED=y

CONFIG_SYSFS_DEPRECATED_V2=y

CONFIG_RELAY=y

CONFIG_NAMESPACES=y

CONFIG_BLK_DEV_INITRD=y

CONFIG_INITRAMFS_SOURCE=""

CONFIG_CC_OPTIMIZE_FOR_SIZE=y

CONFIG_SYSCTL=y

CONFIG_UID16=y

CONFIG_SYSCTL_SYSCALL=y

CONFIG_SYSCTL_SYSCALL_CHECK=y

CONFIG_KALLSYMS=y

CONFIG_KALLSYMS_ALL=y

CONFIG_HOTPLUG=y

CONFIG_PRINTK=y

CONFIG_BUG=y

CONFIG_ELF_CORE=y

CONFIG_PCSPKR_PLATFORM=y

CONFIG_COMPAT_BRK=y

CONFIG_BASE_FULL=y

CONFIG_FUTEX=y

CONFIG_ANON_INODES=y

CONFIG_EPOLL=y

CONFIG_SIGNALFD=y

CONFIG_TIMERFD=y

CONFIG_EVENTFD=y

CONFIG_SHMEM=y

CONFIG_VM_EVENT_COUNTERS=y

CONFIG_SLUB_DEBUG=y

CONFIG_SLUB=y

CONFIG_HAVE_OPROFILE=y

CONFIG_KPROBES=y

CONFIG_KRETPROBES=y

CONFIG_HAVE_KPROBES=y

CONFIG_HAVE_KRETPROBES=y

CONFIG_PROC_PAGE_MONITOR=y

CONFIG_SLABINFO=y

CONFIG_RT_MUTEXES=y

CONFIG_BASE_SMALL=0

CONFIG_MODULES=y

CONFIG_MODULE_UNLOAD=y

CONFIG_MODULE_FORCE_UNLOAD=y

CONFIG_KMOD=y

CONFIG_STOP_MACHINE=y

CONFIG_BLOCK=y

CONFIG_LBD=y

CONFIG_IOSCHED_NOOP=y

CONFIG_IOSCHED_AS=y

CONFIG_IOSCHED_DEADLINE=y

CONFIG_IOSCHED_CFQ=y

CONFIG_DEFAULT_CFQ=y

CONFIG_DEFAULT_IOSCHED="cfq"

CONFIG_CLASSIC_RCU=y

CONFIG_TICK_ONESHOT=y

CONFIG_NO_HZ=y

CONFIG_HIGH_RES_TIMERS=y

CONFIG_GENERIC_CLOCKEVENTS_BUILD=y

CONFIG_SMP=y

CONFIG_X86_GENERICARCH=y

CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y

CONFIG_PARAVIRT_GUEST=y

CONFIG_VMI=y

CONFIG_PARAVIRT=y

CONFIG_X86_CYCLONE_TIMER=y

CONFIG_MPENTIUMIII=y

CONFIG_X86_GENERIC=y

CONFIG_X86_CPU=y

CONFIG_X86_CMPXCHG=y

CONFIG_X86_L1_CACHE_SHIFT=7

CONFIG_X86_XADD=y

CONFIG_X86_WP_WORKS_OK=y

CONFIG_X86_INVLPG=y

CONFIG_X86_BSWAP=y

CONFIG_X86_POPAD_OK=y

CONFIG_X86_GOOD_APIC=y

CONFIG_X86_INTEL_USERCOPY=y

CONFIG_X86_USE_PPRO_CHECKSUM=y

CONFIG_X86_TSC=y

CONFIG_X86_CMOV=y

CONFIG_X86_MINIMUM_CPU_FAMILY=4

CONFIG_X86_DEBUGCTLMSR=y

CONFIG_DMI=y

CONFIG_NR_CPUS=8

CONFIG_SCHED_SMT=y

CONFIG_SCHED_MC=y

CONFIG_PREEMPT_VOLUNTARY=y

CONFIG_X86_LOCAL_APIC=y

CONFIG_X86_IO_APIC=y

CONFIG_X86_MCE=y

CONFIG_X86_MCE_NONFATAL=y

CONFIG_VM86=y

CONFIG_MICROCODE=y

CONFIG_MICROCODE_OLD_INTERFACE=y

CONFIG_X86_MSR=y

CONFIG_X86_CPUID=y

CONFIG_HIGHMEM4G=y

CONFIG_PAGE_OFFSET=0xC0000000

CONFIG_HIGHMEM=y

CONFIG_SELECT_MEMORY_MODEL=y

CONFIG_FLATMEM_MANUAL=y

CONFIG_FLATMEM=y

CONFIG_FLAT_NODE_MEM_MAP=y

CONFIG_PAGEFLAGS_EXTENDED=y

CONFIG_SPLIT_PTLOCK_CPUS=4

CONFIG_RESOURCES_64BIT=y

CONFIG_ZONE_DMA_FLAG=1

CONFIG_BOUNCE=y

CONFIG_VIRT_TO_BUS=y

CONFIG_MTRR=y

CONFIG_IRQBALANCE=y

CONFIG_SECCOMP=y

CONFIG_HZ_100=y

CONFIG_HZ=100

CONFIG_SCHED_HRTICK=y

CONFIG_PHYSICAL_START=0x100000

CONFIG_PHYSICAL_ALIGN=0x100000

CONFIG_HOTPLUG_CPU=y

CONFIG_COMPAT_VDSO=y

CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

CONFIG_PM=y

CONFIG_PM_SLEEP_SMP=y

CONFIG_PM_SLEEP=y

CONFIG_SUSPEND=y

CONFIG_SUSPEND_FREEZER=y

CONFIG_ACPI=y

CONFIG_ACPI_SLEEP=y

CONFIG_ACPI_PROCFS=y

CONFIG_ACPI_PROCFS_POWER=y

CONFIG_ACPI_SYSFS_POWER=y

CONFIG_ACPI_PROC_EVENT=y

CONFIG_ACPI_AC=y

CONFIG_ACPI_BATTERY=y

CONFIG_ACPI_BUTTON=y

CONFIG_ACPI_FAN=y

CONFIG_ACPI_DOCK=y

CONFIG_ACPI_PROCESSOR=y

CONFIG_ACPI_HOTPLUG_CPU=y

CONFIG_ACPI_THERMAL=y

CONFIG_ACPI_BLACKLIST_YEAR=2001

CONFIG_ACPI_DEBUG=y

CONFIG_ACPI_EC=y

CONFIG_ACPI_POWER=y

CONFIG_ACPI_SYSTEM=y

CONFIG_X86_PM_TIMER=y

CONFIG_ACPI_CONTAINER=y

CONFIG_CPU_FREQ=y

CONFIG_CPU_FREQ_TABLE=y

CONFIG_CPU_FREQ_DEBUG=y

CONFIG_CPU_FREQ_STAT=y

CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y

CONFIG_CPU_FREQ_GOV_PERFORMANCE=y

CONFIG_CPU_FREQ_GOV_USERSPACE=y

CONFIG_CPU_FREQ_GOV_ONDEMAND=y

CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y

CONFIG_X86_ACPI_CPUFREQ=y

CONFIG_X86_POWERNOW_K8=y

CONFIG_X86_POWERNOW_K8_ACPI=y

CONFIG_X86_ACPI_CPUFREQ_PROC_INTF=y

CONFIG_CPU_IDLE=y

CONFIG_CPU_IDLE_GOV_LADDER=y

CONFIG_CPU_IDLE_GOV_MENU=y

CONFIG_PCI=y

CONFIG_PCI_GOANY=y

CONFIG_PCI_BIOS=y

CONFIG_PCI_DIRECT=y

CONFIG_PCI_MMCONFIG=y

CONFIG_PCI_DOMAINS=y

CONFIG_ARCH_SUPPORTS_MSI=y

CONFIG_PCI_MSI=y

CONFIG_PCI_LEGACY=y

CONFIG_ISA_DMA_API=y

CONFIG_K8_NB=y

CONFIG_BINFMT_ELF=y

CONFIG_NET=y

CONFIG_PACKET=y

CONFIG_UNIX=y

CONFIG_INET=y

CONFIG_IP_MULTICAST=y

CONFIG_IP_FIB_HASH=y

CONFIG_IP_PNP=y

CONFIG_IP_PNP_DHCP=y

CONFIG_INET_DIAG=y

CONFIG_INET_TCP_DIAG=y

CONFIG_TCP_CONG_CUBIC=y

CONFIG_DEFAULT_TCP_CONG="cubic"

CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"

CONFIG_STANDALONE=y

CONFIG_PREVENT_FIRMWARE_BUILD=y

CONFIG_FW_LOADER=y

CONFIG_PNP=y

CONFIG_PNPACPI=y

CONFIG_BLK_DEV=y

CONFIG_BLK_DEV_FD=y

CONFIG_BLK_DEV_LOOP=y

CONFIG_BLK_DEV_RAM=y

CONFIG_BLK_DEV_RAM_COUNT=16

CONFIG_BLK_DEV_RAM_SIZE=4096

CONFIG_HAVE_IDE=y

CONFIG_IDE=y

CONFIG_BLK_DEV_IDE=y

CONFIG_BLK_DEV_IDEDISK=y

CONFIG_IDEDISK_MULTI_MODE=y

CONFIG_BLK_DEV_IDECD=y

CONFIG_BLK_DEV_IDECD_VERBOSE_ERRORS=y

CONFIG_BLK_DEV_IDEACPI=y

CONFIG_IDE_PROC_FS=y

CONFIG_IDE_GENERIC=y

CONFIG_BLK_DEV_IDEDMA_SFF=y

CONFIG_BLK_DEV_IDEPCI=y

CONFIG_BLK_DEV_IDEDMA_PCI=y

CONFIG_BLK_DEV_PIIX=y

CONFIG_BLK_DEV_IDEDMA=y

CONFIG_SCSI=y

CONFIG_SCSI_DMA=y

CONFIG_SCSI_NETLINK=y

CONFIG_BLK_DEV_SD=y

CONFIG_CHR_DEV_SG=y

CONFIG_SCSI_WAIT_SCAN=m

CONFIG_SCSI_SPI_ATTRS=y

CONFIG_SCSI_FC_ATTRS=y

CONFIG_SCSI_LOWLEVEL=y

CONFIG_MD=y

CONFIG_FUSION=y

CONFIG_FUSION_SPI=y

CONFIG_FUSION_MAX_SGE=128

CONFIG_NETDEVICES=y

CONFIG_NETDEVICES_MULTIQUEUE=y

CONFIG_NET_ETHERNET=y

CONFIG_MII=y

CONFIG_NETDEV_1000=y

CONFIG_E1000=y

CONFIG_NETCONSOLE=y

CONFIG_NETPOLL=y

CONFIG_NET_POLL_CONTROLLER=y

CONFIG_INPUT=y

CONFIG_INPUT_MOUSEDEV=y

CONFIG_INPUT_MOUSEDEV_PSAUX=y

CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024

CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768

CONFIG_INPUT_EVDEV=y

CONFIG_INPUT_KEYBOARD=y

CONFIG_KEYBOARD_ATKBD=y

CONFIG_INPUT_MOUSE=y

CONFIG_MOUSE_PS2=y

CONFIG_MOUSE_PS2_ALPS=y

CONFIG_MOUSE_PS2_LOGIPS2PP=y

CONFIG_MOUSE_PS2_SYNAPTICS=y

CONFIG_MOUSE_PS2_LIFEBOOK=y

CONFIG_MOUSE_PS2_TRACKPOINT=y

CONFIG_SERIO=y

CONFIG_SERIO_I8042=y

CONFIG_SERIO_LIBPS2=y

CONFIG_VT=y

CONFIG_VT_CONSOLE=y

CONFIG_HW_CONSOLE=y

CONFIG_DEVKMEM=y

CONFIG_SERIAL_8250=y

CONFIG_SERIAL_8250_CONSOLE=y

CONFIG_FIX_EARLYCON_MEM=y

CONFIG_SERIAL_8250_PCI=y

CONFIG_SERIAL_8250_PNP=y

CONFIG_SERIAL_8250_NR_UARTS=4

CONFIG_SERIAL_8250_RUNTIME_UARTS=4

CONFIG_SERIAL_CORE=y

CONFIG_SERIAL_CORE_CONSOLE=y

CONFIG_UNIX98_PTYS=y

CONFIG_LEGACY_PTYS=y

CONFIG_LEGACY_PTY_COUNT=256

CONFIG_HW_RANDOM=y

CONFIG_RTC=y

CONFIG_RAW_DRIVER=y

CONFIG_MAX_RAW_DEVS=256

CONFIG_DEVPORT=y

CONFIG_POWER_SUPPLY=y

CONFIG_THERMAL=y

CONFIG_SSB_POSSIBLE=y

CONFIG_SSB=y

CONFIG_SSB_SPROM=y

CONFIG_SSB_PCIHOST_POSSIBLE=y

CONFIG_SSB_PCIHOST=y

CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y

CONFIG_SSB_DRIVER_PCICORE=y

CONFIG_DAB=y

CONFIG_AGP=y

CONFIG_AGP_AMD64=y

CONFIG_AGP_INTEL=y

CONFIG_VGA_CONSOLE=y

CONFIG_VGACON_SOFT_SCROLLBACK=y

CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=128

CONFIG_VIDEO_SELECT=y

CONFIG_DUMMY_CONSOLE=y

CONFIG_SOUND=y

CONFIG_SOUND_PRIME=y

CONFIG_HID_SUPPORT=y

CONFIG_HID=y

CONFIG_USB_HID=y

CONFIG_USB_SUPPORT=y

CONFIG_USB_ARCH_HAS_HCD=y

CONFIG_USB_ARCH_HAS_OHCI=y

CONFIG_USB_ARCH_HAS_EHCI=y

CONFIG_USB=y

CONFIG_USB_DEVICEFS=y

CONFIG_USB_EHCI_HCD=y

CONFIG_USB_OHCI_HCD=y

CONFIG_USB_OHCI_LITTLE_ENDIAN=y

CONFIG_USB_UHCI_HCD=y

CONFIG_USB_STORAGE=y

CONFIG_DMIID=y

CONFIG_EXT2_FS=y

CONFIG_EXT2_FS_XATTR=y

CONFIG_EXT2_FS_POSIX_ACL=y

CONFIG_EXT3_FS=y

CONFIG_EXT3_FS_XATTR=y

CONFIG_EXT3_FS_POSIX_ACL=y

CONFIG_JBD=y

CONFIG_FS_MBCACHE=y

CONFIG_JFS_FS=y

CONFIG_FS_POSIX_ACL=y

CONFIG_DNOTIFY=y

CONFIG_INOTIFY=y

CONFIG_INOTIFY_USER=y

CONFIG_AUTOFS4_FS=y

CONFIG_GENERIC_ACL=y

CONFIG_ISO9660_FS=y

CONFIG_FAT_FS=y

CONFIG_MSDOS_FS=y

CONFIG_VFAT_FS=y

CONFIG_FAT_DEFAULT_CODEPAGE=437

CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"

CONFIG_PROC_FS=y

CONFIG_PROC_KCORE=y

CONFIG_PROC_SYSCTL=y

CONFIG_SYSFS=y

CONFIG_TMPFS=y

CONFIG_TMPFS_POSIX_ACL=y

CONFIG_HUGETLBFS=y

CONFIG_HUGETLB_PAGE=y

CONFIG_MSDOS_PARTITION=y

CONFIG_NLS=y

CONFIG_NLS_DEFAULT="iso8859-1"

CONFIG_NLS_CODEPAGE_437=y

CONFIG_NLS_ASCII=y

CONFIG_NLS_ISO8859_1=y

CONFIG_NLS_ISO8859_15=y

CONFIG_NLS_UTF8=y

CONFIG_TRACE_IRQFLAGS_SUPPORT=y

CONFIG_ENABLE_WARN_DEPRECATED=y

CONFIG_FRAME_WARN=1024

CONFIG_MAGIC_SYSRQ=y

CONFIG_UNUSED_SYMBOLS=y

CONFIG_DEBUG_KERNEL=y

CONFIG_DETECT_SOFTLOCKUP=y

CONFIG_TIMER_STATS=y

CONFIG_DEBUG_BUGVERBOSE=y

CONFIG_HAVE_ARCH_KGDB=y

CONFIG_EARLY_PRINTK=y

CONFIG_DEBUG_STACKOVERFLOW=y

CONFIG_X86_FIND_SMP_CONFIG=y

CONFIG_X86_MPPARSE=y

CONFIG_DOUBLEFAULT=y

CONFIG_IO_DELAY_TYPE_0X80=0

CONFIG_IO_DELAY_TYPE_0XED=1

CONFIG_IO_DELAY_TYPE_UDELAY=2

CONFIG_IO_DELAY_TYPE_NONE=3

CONFIG_IO_DELAY_0X80=y

CONFIG_DEFAULT_IO_DELAY_TYPE=0

CONFIG_CRYPTO=y

CONFIG_CRYPTO_HW=y

CONFIG_HAVE_KVM=y

CONFIG_BITREVERSE=y

CONFIG_GENERIC_FIND_FIRST_BIT=y

CONFIG_GENERIC_FIND_NEXT_BIT=y

CONFIG_CRC32=y

CONFIG_PLIST=y

CONFIG_HAS_IOMEM=y

CONFIG_HAS_IOPORT=y

CONFIG_HAS_DMA=y

Reply
0 Kudos
1 Solution

Accepted Solutions
akataria
Enthusiast
Enthusiast
Jump to solution

Hi Nick,

This is related to a kernel issue that we identified a few day back. I have posted a patch to fix the issue on this thread.

http://lkml.org/lkml/2009/2/6/339

Can you and/or anyone else seeing this problem please apply the patch from that link and see if it fixes the issue for you.

This bug was introduced in the kernel in around 2.6.25 time frame so every kernel starting from 2.6.25 will be bad.

Thanks,

Alok

View solution in original post

Reply
0 Kudos
64 Replies
nick_couchman
Immortal
Immortal
Jump to solution

I've not experienced any slower performance with VMI enabled - in fact, my performance has increased. However, as mentioned in this thread, http://communities.vmware.com/thread/185915?tstart=0, I've experienced some incredible memory overhead in VMs with VMI enabled. I'm also using Gentoo in one of those VMs, with kernel 2.6.27 and VMI enabled. You may check your memory usage on those boxes - if you're running up against a physical memory limit and ESXi is having to swap out memory, you may experience some performance issues. Other than that, I'm out of ideas...seems like there may be some work in order for VMI...

Reply
0 Kudos
Risner
Contributor
Contributor
Jump to solution

Sadly, it isn't memory.

One of the test machines has only 1 (one) VM (the slow one) and has 6 gigabytes of ram with only 512 mb activated on the VM with 96 mb of memory overhead.

Reply
0 Kudos
jasonlitka
Enthusiast
Enthusiast
Jump to solution

VMI only works if your CPU has Virtualization instructions (such as Intel's VT).

Jason Litka

Jason Litka http://www.jasonlitka.com
Reply
0 Kudos
Risner
Contributor
Contributor
Jump to solution

I did some more research.

Emerge on the machine running as Non VMI:

bind: Sun Dec 21 08:10:52 2008: 988 seconds

I cloned the VM (ssh into ESXi, cp -r the directory and fixed up the files to refer to the new name), then emerged bind on the NON VMI machine with the VMI machine idle:

bind: Mon Dec 22 18:21:23 2008: 993 seconds

I than ran the emerge on the VMI machine with the non VMI machine idle:

bind: Mon Dec 22 18:39:46 2008: 3975 seconds

This thread says VMI doesn't require VT instructions (which non of the systems have):

http://communities.vmware.com/thread/163521

Reply
0 Kudos
nick_couchman
Immortal
Immortal
Jump to solution

I don't think so - especially given the thread Risner referenced (with a response by a VMware employee) that says that VMI does not require VT. VMI is independent of the hardware capabilities - all VMI does is provide hooks for the guest O/S through the hardware virtualization layer down to the hypervisor to speed up access to things like disk and network devices without requiring those devices to be fully emulated by virtual machines process. This should speed up network and disk I/O, but should definitely not drive CPU usage through the roof - the CPU usage should be the same and just as fast if not lower and faster.

Reply
0 Kudos
nick_couchman
Immortal
Immortal
Jump to solution

I'll try to do some tests on my machines - I have a few ESXi machines that don't support VT technology - and see if I can come up with similar results.

Risner
Contributor
Contributor
Jump to solution

Just to verify, when VMI is enabled, the guest sees it:

test# lspci

...

00:12.0 Memory controller: VMware Inc Device 0801 (rev 01)

test# dmesg | grep -i vmi

VMI: Found VMware, Inc. Hypervisor OPROM, API version 3.0, ROM version 1.0

vmi: registering clock event vmi-timer. mult=5299255 shift=22

Booting paravirtualized kernel on vmi

vmi: registering clock source khz=1263441

Reply
0 Kudos
nick_couchman
Immortal
Immortal
Jump to solution

What, exactly, are you emerging to test out the performance? I did some disk I/O tests on my setup and saw better (in some cases much better) performance with VMI enabled. Once I run a few more tests I'll post the results. In my guest, I also have the VMware Memory controller in lspci plus the VMI PROM output from dmesg.

Reply
0 Kudos
Dave_Mishchenko
Immortal
Immortal
Jump to solution

Your post has been moved to the Performance forum.

Dave Mishchenko

VMware Communities User Moderator

Reply
0 Kudos
Risner
Contributor
Contributor
Jump to solution

emerge net-dns/bind

That has been my test for a while, since it takes a long time and has a lot of different actions.

Everything I timed was slower on all the boxes.

Is there something about how I set my ESXi boxes up that is causing the slowness?

Reply
0 Kudos
nick_couchman
Immortal
Immortal
Jump to solution

Okay, here's my current configuration on ESXi that I'm using for this test. I have a P4 2.4 machine with 2GB of RAM running ESXi 3.5 build 130755. It has a single Intel PRO 100 VE network card that I use for management and for connecting to an NFS-based datastore. The NFS datastore is located on an Openfiler system backed by 2Gb FC-based storage (not that that matters much, since I'm accessing it over 100Mb Ethernet).

I created a single VM and installed Gentoo - used the 2008.0 Stage 3 image, then brough it up with a world update to the latest, installed necessary packages, kernel, etc. I'm using the 2.6.26-gentoo-r4 kernel on the system. I accepted a lot of the defaults, but made sure to enable VMI in the kernel and tweaked network and storage controllers a little bit for the VMware environment. My root filesystem is ext3. The VM has 1024MB of RAM allocated for it and a 1GB swap partition on the VMDK hard drive.

For my initial tests I used the bonnie++ program for testing disk throughput. My rationale for this was that, even though it doesn't necessarily test the performance of the entire system, it should give a pretty good idea of how well VMI actually accelerates hard drive access and how CPU usage is affected by disk access - that is, the CPU overhead required for doing certain disk-intensive tasks. I tested it twice without VMI, enabled VMI and tested four times, then disabled VMI and tested twice more. The results are very, very mixed - latency for any of the tasks with VMI enabled was generally higher - in some cases significantly so. The areas where VMI tended to do better were in sequential I/O character and block-based operations per second, with the exception of rewriting. In the Random operations, VMI tended to be better, as well. The exceptions to this increased performance seem to be that latency was higher.

I then went and performed your test on emerging net-dns/bind. I saw exactly the same results as you did - ~13 minutes, give or take 30 seconds, to build bind with VMI disabled, and 30 to 60 minutes with VMI enabled.

One thing I have noticed is that VMI performance seems to deteriorate rapidly over time. When I first boot a VM, the VMI performance is very good. As I use it, it tends to get worse and worse. My initial build of bind with VMI enabled took 30 minutes (more than twice what it took without VMI). My next build of bind took 45 minutes. I'm on my third build now, and it's looking like it'll make an hour easily. Also, my memory usage for the VM is very, very odd. The Host memory usage is listed at 671 MB, the Guest memory usage at 102 MB, and the memory overhead at 542 MB. If I look at the memory stats inside the guest O/S (during my bind compile), I have 815 MB free - it's hardly using anything at all! The guest is also not using any swap - it's almost as though the VMI memory scheduler is only allowing a very small portion of the RAM to be used, maybe due to the high memory overhead of the VM?? Very weird...

I'm definitely going to do some more tests on this - I'll probably enable debugging on my VM and see if it dumps any useful information to the log file. I also have NOT installed VMware Tools, yet, which will probably be my next step.

Reply
0 Kudos
nick_couchman
Immortal
Immortal
Jump to solution

Oh, one other odd thing...System CPU usage seems to be sitting at 50% constantly. I'm not sure why...could be because I'm running an SMP kernel on a single-CPU VM, but that doesn't usually happen. Anyway, I'm recompiling the kernel w/o SMP support (plus a couple other tweaks), so we'll see if that helps at all.

Reply
0 Kudos
Risner
Contributor
Contributor
Jump to solution

I had 2.6.26-gentoo-r4 kernel also. All my ESX boxes have quite a bit of ram (usually 4 to 8 gb) and I monitor the ram usage closely. I also noticed the several hundred mb of additional ram when running VMI, but the extra ram usage of VMI is well documented.

I never noticed the decrease in performance over time like you found. I also tried it with and without vmware tools, but frankly I only ever used the open-vm-tools. I don't tend to use the cdrom based vmware tools shipped with the ESX system in /var/lib.

CPU usage was often odd in VMI mode. When the machines was virtually idle, the CPU usage on the host would be very high. I also assumed it could be SMP, so I tried it with and without SMP in the kernel.

Since you have the same problem and others, I am wondering if something in Linux 2.6.26 is broken? Should we try 2.6.28 (released soon) or 2.6.21/2.6.22 or so?

Reply
0 Kudos
nick_couchman
Immortal
Immortal
Jump to solution

I'm going to try out Gentoo kernels 2.6.25 and 2.6.27. If I still see issues with those, I'll try out one of the Vanilla kernels - maybe 2.6.25 or something like that.

Also, for what it's worth, I have a Gentoo box running 2.6.25-gentoo-r7 and doesn't seem to have any issues. However, it is running on a VT-enabled system, so I can't eliminate that as a possibility, yet, despite what the VMware employee said in the other post. It doesn't make sense to me that it would require VT technology, but who knows how they built VMI??

Reply
0 Kudos
Risner
Contributor
Contributor
Jump to solution

bind: Sat Dec 27 03:45:25 2008: 2726 seconds

emerge bind with VMI on a 2.6.27-gentoo-r7 kernel. Slightly faster, not sure why. But slower compared to 900 seconds with VMI disabled.

Reply
0 Kudos
nick_couchman
Immortal
Immortal
Jump to solution

Well, my testing over the past couple of days has led me to formulate the following two theories:

New Theory #1: VMI is not made to handle VMs with high CPU usage, it's made for VMs with high network and/or disk I/O usage. Tools like bonnie++ indicate that enabling VMI does improve disk I/O throughput and CPU usage associated with that significantly. Tests that stress CPU resources tend to cause highly latent effects on both the host and the guest. Both seem to cause very high memory overhead for the guest VM.

New Theory #2: VMware still has not pinned down VMI exactly how they want it. There are still some things on both the guest kernel side and the Hypervisor end to get straightened out, and it's still in its early stages.

I hope #2 is the correct one - performance on anything CPU-intensive is abysmal! I did get moderately good performance on the emerge net-dns/bind when I set the Timer Frequency to 1000 (vs. 100 or 250). Still not better than with VMI disabled, but better than with other clock speeds. I'll probably continue to play around with some of the settings and see what I can figure out.

Reply
0 Kudos
Risner
Contributor
Contributor
Jump to solution

Interesting.

I found this http://communities.vmware.com/message/599359#599359 article. Damoiselle says that VMI was broken in 2.6.21 and that 2.6.21 was the first kernel to have VMI in vanilla. Evidently Ubuntu uses 2.6.20 versions of kernel. Something linux changed during the same time VMI was committed broke VMI.

[13.|m-599359]

Re: VMI enabled kernel Mar 15, 2007 8:57 AM

[13.|m-599317]

in response to: Noel

</div>

Reply
0 Kudos
wila
Immortal
Immortal
Jump to solution

Indeed... interesting. I have been following this thread with some interest as I was having a VMI enabled VM that showed me some odd behavior.

At times I was getting an Out of memory error while it should not do that. There's similar VMs running on the same host without the VMI part enabled that just work and never error out.

In my case the problem VM is a ubuntu 8.04 jeos based VM with kernel 2.6.24-22-virtual

I've disabled VMI a few days ago and there hasn't been a problem since then, if it stays up like this for another week.. then this might indeed have been the problem area.

The physical CPU in the host is also a P4 based Xeon without VT support.

Right now, I'm not 100% convinced yet that the problem is indeed VMI, but it sure does smell like it...



--

Wil

_____________________________________________________

Visit the new VMware developers wiki at http://www.vi-toolkit.com

edit: my host is ESX3.5 (not embedded)

| Author of Vimalin. The virtual machine Backup app for VMware Fusion, VMware Workstation and Player |
| More info at vimalin.com | Twitter @wilva
Reply
0 Kudos
admin
Immortal
Immortal
Jump to solution

I believe you may be hitting a known bug that is not yet fixed in ESX 3.5 U3. Can you try as a workaround copying the attached vmi.rom file into the VM directory and adding the following config options to the .vmx file?

monitor_control.disable_shared_mmu = "true"

vmibios.filename = "vmi.rom"

Reply
0 Kudos