ESXi with VMI enabled on the VM and enabled in the Linux kernel = 10x slower

[Risner]

I have a performance problem with VMI enabled on the ESXi and Linux systems, does anyone have any advice for me?

I have four ESXi boxes. One on a single processor 1.26 Ghz, one with dual 2.26 Ghz hyperthread (4 logical), one with dual 1.4 Ghz hyperthread (4 logical), and one on a single 3.0 Ghz hyperthread.

I don't believe any of them have Intel's VI hardware virtualization instructions. I use ESXi 3.5 U3 as the host and Gentoo 2.6.26 as the client OS.

I've tried enabling VMI on each machine and every time I do, the performance of the VM is decreased by a factor of 4 to 10 times. For example:

portage: Thu Dec 18 08:13:29 2008: 43 seconds

portage: Fri Dec 19 02:58:21 2008: 167 seconds

portage: Fri Dec 19 03:01:57 2008: 193 seconds

portage: Sat Dec 20 23:40:14 2008: 54 seconds

Here are 4 emerges on Gentoo. One with tickless and VMI disabled (43 seconds), one with Tickless enabled and VMI in ESXi and in Gentoo paravirtual VMI enabled, one with HZ=100 and VMI enabled, and the last one again with tickless and VMI disabled (54 seconds.)

Everything I find online suggests VMI (when enabled in the kernel and in the .vmx file) should increase speed or at the very least leave it at the same. I have been working on this for a week and googled to death on it. I've consulted with others running ESXi and they can reproduce the same problem when using Gentoo. I haven't tried Ubuntu Fiesty Fawn or any other OS, but I wouldn't think that would matter much.

If anyone is curious on the gentoo configs, here is a world file:

app-admin/newsyslog

app-admin/sysklogd

app-editors/nvi

app-emulation/open-vm-tools

app-portage/gentoolkit

app-portage/portage-utils

app-shells/tcsh

dev-util/strace

net-analyzer/tcpdump

net-dns/bind

net-misc/bsdwhois

net-misc/ntpclient

net-misc/telnet-bsd

sys-apps/pciutils

sys-apps/slocate

sys-boot/grub

sys-fs/jfsutils

sys-kernel/gentoo-sources

sys-process/lsof

sys-process/vixie-cron

www-client/w3m

Here is the kernel config:

CONFIG_X86_32=y

CONFIG_X86=y

CONFIG_ARCH_DEFCONFIG="arch/x86/configs/i386_defconfig"

CONFIG_GENERIC_TIME=y

CONFIG_GENERIC_CMOS_UPDATE=y

CONFIG_CLOCKSOURCE_WATCHDOG=y

CONFIG_GENERIC_CLOCKEVENTS=y

CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y

CONFIG_LOCKDEP_SUPPORT=y

CONFIG_STACKTRACE_SUPPORT=y

CONFIG_HAVE_LATENCYTOP_SUPPORT=y

CONFIG_FAST_CMPXCHG_LOCAL=y

CONFIG_MMU=y

CONFIG_ZONE_DMA=y

CONFIG_GENERIC_ISA_DMA=y

CONFIG_GENERIC_IOMAP=y

CONFIG_GENERIC_BUG=y

CONFIG_GENERIC_HWEIGHT=y

CONFIG_ARCH_MAY_HAVE_PC_FDC=y

CONFIG_RWSEM_XCHGADD_ALGORITHM=y

CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y

CONFIG_GENERIC_CALIBRATE_DELAY=y

CONFIG_ARCH_HAS_CPU_RELAX=y

CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y

CONFIG_HAVE_SETUP_PER_CPU_AREA=y

CONFIG_ARCH_HIBERNATION_POSSIBLE=y

CONFIG_ARCH_SUSPEND_POSSIBLE=y

CONFIG_ARCH_POPULATES_NODE_MAP=y

CONFIG_ARCH_SUPPORTS_AOUT=y

CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y

CONFIG_GENERIC_HARDIRQS=y

CONFIG_GENERIC_IRQ_PROBE=y

CONFIG_GENERIC_PENDING_IRQ=y

CONFIG_X86_SMP=y

CONFIG_X86_32_SMP=y

CONFIG_X86_HT=y

CONFIG_X86_BIOS_REBOOT=y

CONFIG_X86_TRAMPOLINE=y

CONFIG_KTIME_SCALAR=y

CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"

CONFIG_EXPERIMENTAL=y

CONFIG_LOCK_KERNEL=y

CONFIG_INIT_ENV_ARG_LIMIT=32

CONFIG_LOCALVERSION=""

CONFIG_LOCALVERSION_AUTO=y

CONFIG_SWAP=y

CONFIG_SYSVIPC=y

CONFIG_SYSVIPC_SYSCTL=y

CONFIG_POSIX_MQUEUE=y

CONFIG_IKCONFIG=y

CONFIG_IKCONFIG_PROC=y

CONFIG_LOG_BUF_SHIFT=18

CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y

CONFIG_SYSFS_DEPRECATED=y

CONFIG_SYSFS_DEPRECATED_V2=y

CONFIG_RELAY=y

CONFIG_NAMESPACES=y

CONFIG_BLK_DEV_INITRD=y

CONFIG_INITRAMFS_SOURCE=""

CONFIG_CC_OPTIMIZE_FOR_SIZE=y

CONFIG_SYSCTL=y

CONFIG_UID16=y

CONFIG_SYSCTL_SYSCALL=y

CONFIG_SYSCTL_SYSCALL_CHECK=y

CONFIG_KALLSYMS=y

CONFIG_KALLSYMS_ALL=y

CONFIG_HOTPLUG=y

CONFIG_PRINTK=y

CONFIG_BUG=y

CONFIG_ELF_CORE=y

CONFIG_PCSPKR_PLATFORM=y

CONFIG_COMPAT_BRK=y

CONFIG_BASE_FULL=y

CONFIG_FUTEX=y

CONFIG_ANON_INODES=y

CONFIG_EPOLL=y

CONFIG_SIGNALFD=y

CONFIG_TIMERFD=y

CONFIG_EVENTFD=y

CONFIG_SHMEM=y

CONFIG_VM_EVENT_COUNTERS=y

CONFIG_SLUB_DEBUG=y

CONFIG_SLUB=y

CONFIG_HAVE_OPROFILE=y

CONFIG_KPROBES=y

CONFIG_KRETPROBES=y

CONFIG_HAVE_KPROBES=y

CONFIG_HAVE_KRETPROBES=y

CONFIG_PROC_PAGE_MONITOR=y

CONFIG_SLABINFO=y

CONFIG_RT_MUTEXES=y

CONFIG_BASE_SMALL=0

CONFIG_MODULES=y

CONFIG_MODULE_UNLOAD=y

CONFIG_MODULE_FORCE_UNLOAD=y

CONFIG_KMOD=y

CONFIG_STOP_MACHINE=y

CONFIG_BLOCK=y

CONFIG_LBD=y

CONFIG_IOSCHED_NOOP=y

CONFIG_IOSCHED_AS=y

CONFIG_IOSCHED_DEADLINE=y

CONFIG_IOSCHED_CFQ=y

CONFIG_DEFAULT_CFQ=y

CONFIG_DEFAULT_IOSCHED="cfq"

CONFIG_CLASSIC_RCU=y

CONFIG_TICK_ONESHOT=y

CONFIG_NO_HZ=y

CONFIG_HIGH_RES_TIMERS=y

CONFIG_GENERIC_CLOCKEVENTS_BUILD=y

CONFIG_SMP=y

CONFIG_X86_GENERICARCH=y

CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y

CONFIG_PARAVIRT_GUEST=y

CONFIG_VMI=y

CONFIG_PARAVIRT=y

CONFIG_X86_CYCLONE_TIMER=y

CONFIG_MPENTIUMIII=y

CONFIG_X86_GENERIC=y

CONFIG_X86_CPU=y

CONFIG_X86_CMPXCHG=y

CONFIG_X86_L1_CACHE_SHIFT=7

CONFIG_X86_XADD=y

CONFIG_X86_WP_WORKS_OK=y

CONFIG_X86_INVLPG=y

CONFIG_X86_BSWAP=y

CONFIG_X86_POPAD_OK=y

CONFIG_X86_GOOD_APIC=y

CONFIG_X86_INTEL_USERCOPY=y

CONFIG_X86_USE_PPRO_CHECKSUM=y

CONFIG_X86_TSC=y

CONFIG_X86_CMOV=y

CONFIG_X86_MINIMUM_CPU_FAMILY=4

CONFIG_X86_DEBUGCTLMSR=y

CONFIG_DMI=y

CONFIG_NR_CPUS=8

CONFIG_SCHED_SMT=y

CONFIG_SCHED_MC=y

CONFIG_PREEMPT_VOLUNTARY=y

CONFIG_X86_LOCAL_APIC=y

CONFIG_X86_IO_APIC=y

CONFIG_X86_MCE=y

CONFIG_X86_MCE_NONFATAL=y

CONFIG_VM86=y

CONFIG_MICROCODE=y

CONFIG_MICROCODE_OLD_INTERFACE=y

CONFIG_X86_MSR=y

CONFIG_X86_CPUID=y

CONFIG_HIGHMEM4G=y

CONFIG_PAGE_OFFSET=0xC0000000

CONFIG_HIGHMEM=y

CONFIG_SELECT_MEMORY_MODEL=y

CONFIG_FLATMEM_MANUAL=y

CONFIG_FLATMEM=y

CONFIG_FLAT_NODE_MEM_MAP=y

CONFIG_PAGEFLAGS_EXTENDED=y

CONFIG_SPLIT_PTLOCK_CPUS=4

CONFIG_RESOURCES_64BIT=y

CONFIG_ZONE_DMA_FLAG=1

CONFIG_BOUNCE=y

CONFIG_VIRT_TO_BUS=y

CONFIG_MTRR=y

CONFIG_IRQBALANCE=y

CONFIG_SECCOMP=y

CONFIG_HZ_100=y

CONFIG_HZ=100

CONFIG_SCHED_HRTICK=y

CONFIG_PHYSICAL_START=0x100000

CONFIG_PHYSICAL_ALIGN=0x100000

CONFIG_HOTPLUG_CPU=y

CONFIG_COMPAT_VDSO=y

CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

CONFIG_PM=y

CONFIG_PM_SLEEP_SMP=y

CONFIG_PM_SLEEP=y

CONFIG_SUSPEND=y

CONFIG_SUSPEND_FREEZER=y

CONFIG_ACPI=y

CONFIG_ACPI_SLEEP=y

CONFIG_ACPI_PROCFS=y

CONFIG_ACPI_PROCFS_POWER=y

CONFIG_ACPI_SYSFS_POWER=y

CONFIG_ACPI_PROC_EVENT=y

CONFIG_ACPI_AC=y

CONFIG_ACPI_BATTERY=y

CONFIG_ACPI_BUTTON=y

CONFIG_ACPI_FAN=y

CONFIG_ACPI_DOCK=y

CONFIG_ACPI_PROCESSOR=y

CONFIG_ACPI_HOTPLUG_CPU=y

CONFIG_ACPI_THERMAL=y

CONFIG_ACPI_BLACKLIST_YEAR=2001

CONFIG_ACPI_DEBUG=y

CONFIG_ACPI_EC=y

CONFIG_ACPI_POWER=y

CONFIG_ACPI_SYSTEM=y

CONFIG_X86_PM_TIMER=y

CONFIG_ACPI_CONTAINER=y

CONFIG_CPU_FREQ=y

CONFIG_CPU_FREQ_TABLE=y

CONFIG_CPU_FREQ_DEBUG=y

CONFIG_CPU_FREQ_STAT=y

CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y

CONFIG_CPU_FREQ_GOV_PERFORMANCE=y

CONFIG_CPU_FREQ_GOV_USERSPACE=y

CONFIG_CPU_FREQ_GOV_ONDEMAND=y

CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y

CONFIG_X86_ACPI_CPUFREQ=y

CONFIG_X86_POWERNOW_K8=y

CONFIG_X86_POWERNOW_K8_ACPI=y

CONFIG_X86_ACPI_CPUFREQ_PROC_INTF=y

CONFIG_CPU_IDLE=y

CONFIG_CPU_IDLE_GOV_LADDER=y

CONFIG_CPU_IDLE_GOV_MENU=y

CONFIG_PCI=y

CONFIG_PCI_GOANY=y

CONFIG_PCI_BIOS=y

CONFIG_PCI_DIRECT=y

CONFIG_PCI_MMCONFIG=y

CONFIG_PCI_DOMAINS=y

CONFIG_ARCH_SUPPORTS_MSI=y

CONFIG_PCI_MSI=y

CONFIG_PCI_LEGACY=y

CONFIG_ISA_DMA_API=y

CONFIG_K8_NB=y

CONFIG_BINFMT_ELF=y

CONFIG_NET=y

CONFIG_PACKET=y

CONFIG_UNIX=y

CONFIG_INET=y

CONFIG_IP_MULTICAST=y

CONFIG_IP_FIB_HASH=y

CONFIG_IP_PNP=y

CONFIG_IP_PNP_DHCP=y

CONFIG_INET_DIAG=y

CONFIG_INET_TCP_DIAG=y

CONFIG_TCP_CONG_CUBIC=y

CONFIG_DEFAULT_TCP_CONG="cubic"

CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"

CONFIG_STANDALONE=y

CONFIG_PREVENT_FIRMWARE_BUILD=y

CONFIG_FW_LOADER=y

CONFIG_PNP=y

CONFIG_PNPACPI=y

CONFIG_BLK_DEV=y

CONFIG_BLK_DEV_FD=y

CONFIG_BLK_DEV_LOOP=y

CONFIG_BLK_DEV_RAM=y

CONFIG_BLK_DEV_RAM_COUNT=16

CONFIG_BLK_DEV_RAM_SIZE=4096

CONFIG_HAVE_IDE=y

CONFIG_IDE=y

CONFIG_BLK_DEV_IDE=y

CONFIG_BLK_DEV_IDEDISK=y

CONFIG_IDEDISK_MULTI_MODE=y

CONFIG_BLK_DEV_IDECD=y

CONFIG_BLK_DEV_IDECD_VERBOSE_ERRORS=y

CONFIG_BLK_DEV_IDEACPI=y

CONFIG_IDE_PROC_FS=y

CONFIG_IDE_GENERIC=y

CONFIG_BLK_DEV_IDEDMA_SFF=y

CONFIG_BLK_DEV_IDEPCI=y

CONFIG_BLK_DEV_IDEDMA_PCI=y

CONFIG_BLK_DEV_PIIX=y

CONFIG_BLK_DEV_IDEDMA=y

CONFIG_SCSI=y

CONFIG_SCSI_DMA=y

CONFIG_SCSI_NETLINK=y

CONFIG_BLK_DEV_SD=y

CONFIG_CHR_DEV_SG=y

CONFIG_SCSI_WAIT_SCAN=m

CONFIG_SCSI_SPI_ATTRS=y

CONFIG_SCSI_FC_ATTRS=y

CONFIG_SCSI_LOWLEVEL=y

CONFIG_MD=y

CONFIG_FUSION=y

CONFIG_FUSION_SPI=y

CONFIG_FUSION_MAX_SGE=128

CONFIG_NETDEVICES=y

CONFIG_NETDEVICES_MULTIQUEUE=y

CONFIG_NET_ETHERNET=y

CONFIG_MII=y

CONFIG_NETDEV_1000=y

CONFIG_E1000=y

CONFIG_NETCONSOLE=y

CONFIG_NETPOLL=y

CONFIG_NET_POLL_CONTROLLER=y

CONFIG_INPUT=y

CONFIG_INPUT_MOUSEDEV=y

CONFIG_INPUT_MOUSEDEV_PSAUX=y

CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024

CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768

CONFIG_INPUT_EVDEV=y

CONFIG_INPUT_KEYBOARD=y

CONFIG_KEYBOARD_ATKBD=y

CONFIG_INPUT_MOUSE=y

CONFIG_MOUSE_PS2=y

CONFIG_MOUSE_PS2_ALPS=y

CONFIG_MOUSE_PS2_LOGIPS2PP=y

CONFIG_MOUSE_PS2_SYNAPTICS=y

CONFIG_MOUSE_PS2_LIFEBOOK=y

CONFIG_MOUSE_PS2_TRACKPOINT=y

CONFIG_SERIO=y

CONFIG_SERIO_I8042=y

CONFIG_SERIO_LIBPS2=y

CONFIG_VT=y

CONFIG_VT_CONSOLE=y

CONFIG_HW_CONSOLE=y

CONFIG_DEVKMEM=y

CONFIG_SERIAL_8250=y

CONFIG_SERIAL_8250_CONSOLE=y

CONFIG_FIX_EARLYCON_MEM=y

CONFIG_SERIAL_8250_PCI=y

CONFIG_SERIAL_8250_PNP=y

CONFIG_SERIAL_8250_NR_UARTS=4

CONFIG_SERIAL_8250_RUNTIME_UARTS=4

CONFIG_SERIAL_CORE=y

CONFIG_SERIAL_CORE_CONSOLE=y

CONFIG_UNIX98_PTYS=y

CONFIG_LEGACY_PTYS=y

CONFIG_LEGACY_PTY_COUNT=256

CONFIG_HW_RANDOM=y

CONFIG_RTC=y

CONFIG_RAW_DRIVER=y

CONFIG_MAX_RAW_DEVS=256

CONFIG_DEVPORT=y

CONFIG_POWER_SUPPLY=y

CONFIG_THERMAL=y

CONFIG_SSB_POSSIBLE=y

CONFIG_SSB=y

CONFIG_SSB_SPROM=y

CONFIG_SSB_PCIHOST_POSSIBLE=y

CONFIG_SSB_PCIHOST=y

CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y

CONFIG_SSB_DRIVER_PCICORE=y

CONFIG_DAB=y

CONFIG_AGP=y

CONFIG_AGP_AMD64=y

CONFIG_AGP_INTEL=y

CONFIG_VGA_CONSOLE=y

CONFIG_VGACON_SOFT_SCROLLBACK=y

CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=128

CONFIG_VIDEO_SELECT=y

CONFIG_DUMMY_CONSOLE=y

CONFIG_SOUND=y

CONFIG_SOUND_PRIME=y

CONFIG_HID_SUPPORT=y

CONFIG_HID=y

CONFIG_USB_HID=y

CONFIG_USB_SUPPORT=y

CONFIG_USB_ARCH_HAS_HCD=y

CONFIG_USB_ARCH_HAS_OHCI=y

CONFIG_USB_ARCH_HAS_EHCI=y

CONFIG_USB=y

CONFIG_USB_DEVICEFS=y

CONFIG_USB_EHCI_HCD=y

CONFIG_USB_OHCI_HCD=y

CONFIG_USB_OHCI_LITTLE_ENDIAN=y

CONFIG_USB_UHCI_HCD=y

CONFIG_USB_STORAGE=y

CONFIG_DMIID=y

CONFIG_EXT2_FS=y

CONFIG_EXT2_FS_XATTR=y

CONFIG_EXT2_FS_POSIX_ACL=y

CONFIG_EXT3_FS=y

CONFIG_EXT3_FS_XATTR=y

CONFIG_EXT3_FS_POSIX_ACL=y

CONFIG_JBD=y

CONFIG_FS_MBCACHE=y

CONFIG_JFS_FS=y

CONFIG_FS_POSIX_ACL=y

CONFIG_DNOTIFY=y

CONFIG_INOTIFY=y

CONFIG_INOTIFY_USER=y

CONFIG_AUTOFS4_FS=y

CONFIG_GENERIC_ACL=y

CONFIG_ISO9660_FS=y

CONFIG_FAT_FS=y

CONFIG_MSDOS_FS=y

CONFIG_VFAT_FS=y

CONFIG_FAT_DEFAULT_CODEPAGE=437

CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"

CONFIG_PROC_FS=y

CONFIG_PROC_KCORE=y

CONFIG_PROC_SYSCTL=y

CONFIG_SYSFS=y

CONFIG_TMPFS=y

CONFIG_TMPFS_POSIX_ACL=y

CONFIG_HUGETLBFS=y

CONFIG_HUGETLB_PAGE=y

CONFIG_MSDOS_PARTITION=y

CONFIG_NLS=y

CONFIG_NLS_DEFAULT="iso8859-1"

CONFIG_NLS_CODEPAGE_437=y

CONFIG_NLS_ASCII=y

CONFIG_NLS_ISO8859_1=y

CONFIG_NLS_ISO8859_15=y

CONFIG_NLS_UTF8=y

CONFIG_TRACE_IRQFLAGS_SUPPORT=y

CONFIG_ENABLE_WARN_DEPRECATED=y

CONFIG_FRAME_WARN=1024

CONFIG_MAGIC_SYSRQ=y

CONFIG_UNUSED_SYMBOLS=y

CONFIG_DEBUG_KERNEL=y

CONFIG_DETECT_SOFTLOCKUP=y

CONFIG_TIMER_STATS=y

CONFIG_DEBUG_BUGVERBOSE=y

CONFIG_HAVE_ARCH_KGDB=y

CONFIG_EARLY_PRINTK=y

CONFIG_DEBUG_STACKOVERFLOW=y

CONFIG_X86_FIND_SMP_CONFIG=y

CONFIG_X86_MPPARSE=y

CONFIG_DOUBLEFAULT=y

CONFIG_IO_DELAY_TYPE_0X80=0

CONFIG_IO_DELAY_TYPE_0XED=1

CONFIG_IO_DELAY_TYPE_UDELAY=2

CONFIG_IO_DELAY_TYPE_NONE=3

CONFIG_IO_DELAY_0X80=y

CONFIG_DEFAULT_IO_DELAY_TYPE=0

CONFIG_CRYPTO=y

CONFIG_CRYPTO_HW=y

CONFIG_HAVE_KVM=y

CONFIG_BITREVERSE=y

CONFIG_GENERIC_FIND_FIRST_BIT=y

CONFIG_GENERIC_FIND_NEXT_BIT=y

CONFIG_CRC32=y

CONFIG_PLIST=y

CONFIG_HAS_IOMEM=y

CONFIG_HAS_IOPORT=y

CONFIG_HAS_DMA=y

[wila]

Hi,

are you working for VMware? If so can you get the three box logo setup next to your name, it's not a habit of mine to just add a random binary from the internet to my setup....

FWIW, the OOM error just happened again and I noticed a configuration error where my VM didn't have swap setup properly. The UUID of the swap partition was changed and it was identified in fstab by the UUID and not device ID. So guess I have to test again now.. :smileyshocked:

--

Wil

_____________________________________________________

Visit the new VMware developers wiki at

[nick_couchman]

I can vouch for him being a VMware employee - I've been corresponding with him directly via e-mail, and he mentioned that he would post this solution in the forum. I contacted him via e-mail after finding his name listed in the header file for the VMI section of the kernel sources.

I'll be trying out this work-around to see if it helps me out.

[admin]

Nevertheless, I'll try to get that logo thing in place... or you can contact me directly at zach@vmware.com

[Dave_Mishchenko]

I've put in a request to the forum admins for this.

Dave Mishchenko

VMware Communities User Moderator

[nick_couchman]

It's certainly better than before, but still quite a bit slower than with VMI disabled. I tried it out on my PE2800 with 2 x 2.8 GHz processors, and the "emerge net-dns/bind" on Gentoo took 25 minutes (wall clock time) with VMI enabled, vs. about 10 minutes with VMI disabled. I'll run it a couple more times to make sure the results are consistent. Also, System CPU usage still seems to spike and stay high. After the emerge is complete, it takes it a while for the System CPU usage to drop back down - it almost acts like a load average where it comes down over time.

[admin]

I'm just coming up to speed on this thread, but if you're seeing system spikes, that isn't normal. There's a couple experimental variables it would be good to eliminate or highlight as factors:

1) SMP vs. UP

2) PAE vs. non-PAE kernel

3) CONFIG_HIGHPTE

I think you already did the VMI timer vs. standard timer experiment. In the meantime, I have enough info to try reproducing it here.

[nick_couchman]

Before the fix that you sent, I had already tried SMP vs. UP and PAE vs. non-PAE, and neither of those seemed to make a difference. I will go back and try them again with the supplied fix. The HIGHPTE also did not seem to make a difference. Currently, I'm running a UP kernel withOUT PAE support, with CONFIG_HPET=y and CONFIG_HPET_TIMER. CONFIG_HPET_MMAP is also undefined.

[wila]

Hi,

I've added the suggested settings and the rom file and so far things look better. Memory usage appears to be normal. No out of memory errors.

In regards to your questions:

VM is UP, kernel has CONFIG_SMP=y

non-PAE

CONFIG_HIGHPTE=y

--

Wil

_____________________________________________________

Visit the new VMware developers wiki at http://www.vi-toolkit.com

[nick_couchman]

How much better? If you run with VMI enabled, is the performance better than running with VMI disabled?

I have a machine running ESX 3.5.0 build 82663, and VMI seems to work perfectly find on this one. My ESX 3.5 build 130756 and ESXi 3.5 build 130755 machines seem to not work very well, though, even with the update ROM and parameters.

[nick_couchman]

Okay, tried out the SMP kernel, again, and that doesn't seem to make a difference. Also tried it with HPET enabled and disabled - again, no real change. I'm still seeing the behavior where System CPU usage gradually goes up as the VM is used, and then takes a very, very long time to come back down. I ran a compile (emerge) on one of the VMI-enabled machines with the new ROM file, and it's now been two hours since it finished and System CPU usage is at 20% and falling about 0.10% per minute. This machine is running 2.6.27 - as stated previously, I have a system running 2.6.25 with VMI enabled on VMware 3.5.0 build 82663 and it works fine. This is a UP kernel, non-PAE, with absolutely no HPET support.

[admin]

Well, the fact that it happens on UP rules out my first theory. Still, it is useful to know, did you also try disabling the shared MMU, or just copy the ROM file over? Can you measure

cat /proc/vmstat | grep fault; sleep 10; cat /proc/vmstat | grep fault

during the high system CPU activity period? Perhaps some output from the kernel or CPUID is confusing something during the emerge, and something is crashing in a loop? Any unusual processes hanging around eating lots of CPU? Any unusual lingering procs in a ps axf?

[wila]

Nick,

How much better?

Umm.. it doesn't crash anymore.. the problem here is that I'm running production on the VM, it simply runs as a LAMP server and not crashing is kinda important.

If you run with VMI enabled, is the performance better than running with VMI disabled?

The main difference I note is that memory usage is normal now, in my case better performance isn't the most important issue (but it would be nice)

I can't experiment with the VM itself now, as I need it to run and keep on doing so, but I've taken a clone yesterday and I can experiment with that one. i'll set up a test version of it tomorrow when I get the chance and see if I can get something more tangible.

--

Wil

_____________________________________________________

Visit the new VMware developers wiki at http://www.vi-toolkit.com

[admin]

Wila, how are your VM's crashing? This is the first I see of crashing..

[admin]

Nick, you seem to have two variables.. ESX 3.5.0 82663 + 2.6.25 works fine, but later ESX 3.5.0 + 2.6.26 has problems? Have you been able to try 2.6.25 on later Esx? It should be useful to know if it is Esx or kernel upgrade which brought the bug.

[wila]

Hi Zach,

Sorry for not being clear, the VM isn't crashing. On the vm mysql was crashing due to out of memory errors.

So far I'm not completely convinced that was VMI that was the culprit, which is why I must retest and do a better job there. The problem appears to have gone on the production VM, but it wasn't always easy reproducable as there was sometime a week between the crash.. Unfortunately my time to test, troubleshoot and isolate was extremeley limited last week.

I'll try and setup a decent test environment for this over the next week and see if I can properly isolate the problem.

--

Wil

_____________________________________________________

Visit the new VMware developers wiki at http://www.vi-toolkit.com

[nick_couchman]

I've eliminated the kernel version as the variable - I took the config from the working VM on ESX 3.5.0 82663 and copied it over to the VM on ESX 3.5.0 130756 and ESXi 3.5.0 130755 and built kernel 2.6.25 with the same config. I still get very, very poor performance on the newer ESX/ESXi instances even with an identical kernel version and config.

[nick_couchman]

Yes, I do have shared MMU disabled with the new ROM file. I'm currently running the following command while doing the emerge:

while true; do cat /proc/vmstat | grep fault; sleep 10; done

I started this command just before starting the emerge and terminated it just after the emerge finished. I've compressed the output and attached it here. For what it's worth, the System CPU usage seems to stay low at first, and then at some point during the emerge it rockets up to 95% and then takes it time coming down (it's around 91% now, 5 minutes after shooting up to 95%). Is there anything in the kernel config I should turn on that would provide any more debugging information? I have almost all of the kernel debugging turned off...

Added file attachment...

[nick_couchman]

BTW, no, there are no unusual processes hanging around. The unusual part about the system CPU usage spiking so high is that there's no process in top that correlates to the high system CPU usage - none of the processes are using anything close to that much CPU, yet it still shows very high usage.

[admin]

Well, those page fault rates look pretty normal for a compile - 5,000 - 20,000 faults a second sounds okay for a compile workload. Since you appear to have narrowed it down to an ESX 3.5 difference, I'll try upgrading my ESX box to see if I can reproduce the same slowdown.

[wila]

Sorry for the long delay in replying I got a bit sidetracked by various other things. Darn those customers

I've setup a test VM here without the additional VMI ROM, It has been run for 2 days without a problem, but without much load either.. Right now i've setup a cron job with a wget pull for some pages and it does make the memory usage go up bit by bit over time, but I've got to hammer the site in order to get some quick results.

I'll add some more load tomorrow and see if I can make it a reproducable case and make sure we find the guilty party (it could be something else...)

Btw, please would it be possible to roll the VMI ROM version number so that you can see in /var/log/messages which version of the ROM we are using?

PS: Will there be an update to the vmi rom sources on sourceforge ?

--

Wil

_____________________________________________________

Visit the new VMware developers wiki at http://www.vi-toolkit.com