nmvdaemon
Contributor
Contributor

64bit VM Performance issues

Hey all,

I have this weird and unfortunately, quite urgent, issue with one of my VM's

First, let me give you a full overview of our setup:

2x HP DL380 G5 (2x 2,83Ghz Quad-core and 32GB RAM each) connected to an HP MSA2012i iSCSI SAN via Software iSCSI.

On these machines, I have about 10 VM's running without any problems. They are all running RHEL5.2 32bit.

Yesterday, I installed my first RHEL5.2 64bit machine. Allocated it 2 logical CPU's, 8GB of memory. Turned on VT in the BIOS of the hosts, started install and I had no issues during installation.

Now, however, I notice that performance in this VM is extremely bad, even when doing practically nothing. For example: untarring the VMware tools (35MB) took 2 full minutes! (No other VM's were present on the same host at the time of testing. This should have only taken seconds. I tried it again, this time with a top open and CPU skyrocketed to 100%.

Anybody, please. Help. Smiley Wink

Best regards,

Koen

Tags (3)
0 Kudos
2 Replies
Dave_Mishchenko
Immortal
Immortal

Hello Koen, welcome to the VMware Community forums. Your post has been moved to the Performance forum.

Dave Mishchenko

VMware Communities User Moderator

0 Kudos
nmvdaemon
Contributor
Contributor

# vmstat

procs -


memory----


-swap -


io---- system -


cpu------

r b swpd free buff cache si so bi bo in cs us sy id wa st

0 0 0 7710432 15784 136772 0 0 32 195 504 21 1 2 97 0 0

# dmesg

Linux version 2.6.18-92.1.1.el5 (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-41)) #1 SMP Thu May 22 09:01:47 EDT 2008

Command line: ro root=/dev/local/root rhgb quiet

BIOS-provided physical RAM map:

BIOS-e820: 0000000000000000 - 000000000009f800 (usable)

BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)

BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved)

BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)

BIOS-e820: 0000000000100000 - 00000000efef0000 (usable)

BIOS-e820: 00000000efef0000 - 00000000efeff000 (ACPI data)

BIOS-e820: 00000000efeff000 - 00000000eff00000 (ACPI NVS)

BIOS-e820: 00000000eff00000 - 00000000f0000000 (usable)

BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)

BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)

BIOS-e820: 00000000fffe0000 - 0000000100000000 (reserved)

BIOS-e820: 0000000100000000 - 0000000200000000 (usable)

DMI present.

ACPI: RSDP (v000 PTLTD ) @ 0x00000000000f6c60

ACPI: RSDT (v001 PTLTD RSDT 0x06040000 LTP 0x00000000) @ 0x00000000efefab5a

ACPI: FADT (v001 INTEL 440BX 0x06040000 PTL 0x000f4240) @ 0x00000000efefef06

ACPI: MADT (v001 PTLTD APIC 0x06040000 LTP 0x00000000) @ 0x00000000efefef7a

ACPI: BOOT (v001 PTLTD $SBFTBL$ 0x06040000 LTP 0x00000001) @ 0x00000000efefefd8

ACPI: DSDT (v001 PTLTD Custom 0x06040000 MSFT 0x0100000d) @ 0x0000000000000000

No NUMA configuration found

Faking a node at 0000000000000000-0000000200000000

Bootmem setup node 0 0000000000000000-0000000200000000

Memory for crash kernel (0x0 to 0x0) notwithin permissible range

disabling kdump

On node 0 totalpages: 2001576

DMA zone: 2688 pages, LIFO batch:0

DMA32 zone: 964648 pages, LIFO batch:31

Normal zone: 1034240 pages, LIFO batch:31

ACPI: PM-Timer IO Port: 0x1008

ACPI: Local APIC address 0xfee00000

ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)

Processor #0 7:7 APIC version 17

ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)

Processor #1 7:7 APIC version 17

ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])

ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])

ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])

IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, GSI 0-23

ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)

ACPI: IRQ0 used by override.

ACPI: IRQ2 used by override.

ACPI: IRQ9 used by override.

Setting APIC routing to physical flat

Using ACPI (MADT) for SMP configuration information

Nosave address range: 000000000009f000 - 00000000000a0000

Nosave address range: 00000000000a0000 - 00000000000ca000

Nosave address range: 00000000000ca000 - 00000000000cc000

Nosave address range: 00000000000cc000 - 00000000000dc000

Nosave address range: 00000000000dc000 - 0000000000100000

Nosave address range: 00000000efef0000 - 00000000efeff000

Nosave address range: 00000000efeff000 - 00000000eff00000

Nosave address range: 00000000f0000000 - 00000000fec00000

Nosave address range: 00000000fec00000 - 00000000fec10000

Nosave address range: 00000000fec10000 - 00000000fee00000

Nosave address range: 00000000fee00000 - 00000000fee01000

Nosave address range: 00000000fee01000 - 00000000fffe0000

Nosave address range: 00000000fffe0000 - 0000000100000000

Allocating PCI resources starting at f1000000 (gap: f0000000:ec00000)

SMP: Allowing 2 CPUs, 0 hotplug CPUs

Built 1 zonelists. Total pages: 2001576

Kernel command line: ro root=/dev/local/root rhgb quiet

Initializing CPU#0

PID hash table entries: 4096 (order: 12, 32768 bytes)

Console: colour VGA+ 80x25

Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)

Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)

Checking aperture...

PCI-DMA: Using software bounce buffering for IO (SWIOTLB)

Placing software IO TLB between 0x1c2b000 - 0x5c2b000

Memory: 7924360k/8388608k available (2459k kernel code, 201652k reserved, 1247k data, 196k init)

Calibrating delay using timer specific routine.. 5677.46 BogoMIPS (lpj=2838730)

Security Framework v1.0.0 initialized

SELinux: Initializing.

SELinux: Starting in permissive mode

selinux_register_security: Registering secondary module capability

Capability LSM initialized as secondary

Mount-cache hash table entries: 256

CPU: L1 I cache: 32K, L1 D cache: 32K

CPU: L2 cache: 6144K

SMP alternatives: switching to UP code

ACPI: Core revision 20060707

Using local APIC timer interrupts.

result 4124540

Detected 4.124 MHz APIC timer.

SMP alternatives: switching to SMP code

Booting processor 1/2 APIC 0x1

Initializing CPU#1

Calibrating delay using timer specific routine.. 5667.53 BogoMIPS (lpj=2833767)

CPU: L1 I cache: 32K, L1 D cache: 32K

CPU: L2 cache: 6144K

Intel(R) Xeon(R) CPU E5440 @ 2.83GHz stepping 08

Brought up 2 CPUs

testing NMI watchdog ... <4>WARNING: CPU#0: NMI appears to be stuck (0->0)!

time.c: Using 3.579545 MHz WALL PM GTOD PIT/TSC timer.

time.c: Detected 2833.008 MHz processor.

sizeof(vma)=176 bytes

sizeof(page)=56 bytes

sizeof(inode)=560 bytes

sizeof(dentry)=216 bytes

sizeof(ext3inode)=760 bytes

sizeof(buffer_head)=96 bytes

sizeof(skbuff)=240 bytes

migration_cost=221

checking if image is initramfs... it is

Freeing initrd memory: 3205k freed

NET: Registered protocol family 16

No dock devices found.

ACPI: bus type pci registered

PCI: Using configuration type 1

ACPI: Interpreter enabled

ACPI: Using IOAPIC for interrupt routing

ACPI: PCI Root Bridge (0000:00)

PCI: Probing PCI hardware (bus 00)

PCI quirk: region 1000-103f claimed by PIIX4 ACPI

PCI quirk: region 1040-104f claimed by PIIX4 SMB

Boot video device is 0000:00:0f.0

ACPI: PCI Interrupt Routing Table http://\_SB_.PCI0._PRT

ACPI: PCI Interrupt Link (IRQs 3 4 5 6 7 9 10 11 14 15) *0, disabled.

ACPI: PCI Interrupt Link (IRQs 3 4 5 6 7 *9 10 11 14 15)

ACPI: PCI Interrupt Link (IRQs 3 4 5 6 7 9 10 *11 14 15)

ACPI: PCI Interrupt Link (IRQs 3 4 5 6 7 9 10 11 14 15) *0, disabled.

Linux Plug and Play Support v0.97 (c) Adam Belay

pnp: PnP ACPI init

pnp: PnP ACPI: found 12 devices

usbcore: registered new driver usbfs

usbcore: registered new driver hub

PCI: Using ACPI for IRQ routing

PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report

NetLabel: Initializing

NetLabel: domain hash size = 128

NetLabel: protocols = UNLABELED CIPSOv4

NetLabel: unlabeled traffic allowed by default

PCI-GART: No AMD northbridge found.

PCI: Bridge: 0000:00:01.0

IO window: disabled.

MEM window: disabled.

PREFETCH window: disabled.

PCI: Setting latency timer of device 0000:00:01.0 to 64

NET: Registered protocol family 2

IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)

TCP established hash table entries: 262144 (order: 10, 4194304 bytes)

TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)

TCP: Hash tables configured (established 262144 bind 65536)

TCP reno registered

Simple Boot Flag at 0x36 set to 0x80

audit: initializing netlink socket (disabled)

audit(1214048697.540:1): initialized

Total HugeTLB memory allocated, 0

VFS: Disk quotas dquot_6.5.1

Dquot-cache hash table entries: 512 (order 0, 4096 bytes)

SELinux: Registering netfilter hooks

Initializing Cryptographic API

ksign: Installing public key data

Loading keyring

- Added public key 8D87DC113F30B7

- User ID: Red Hat, Inc. (Kernel Module GPG key)

io scheduler noop registered

io scheduler anticipatory registered

io scheduler deadline registered

io scheduler cfq registered (default)

Limiting direct PCI/PCI transfers.

pci_hotplug: PCI Hot Plug PCI Core version: 0.5

ACPI: Processor (supports 8 throttling states)

ACPI: Processor (supports 8 throttling states)

ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present

ACPI: Getting cpuindex for acpiid 0x2

ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present

ACPI: Getting cpuindex for acpiid 0x3

Real Time Clock Driver v1.12ac

Non-volatile memory driver v1.2

Linux agpgart interface v0.101 (c) Dave Jones

agpgart: Detected an Intel 440BX Chipset.

agpgart: AGP aperture is 256M @ 0x0

Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled

serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A

serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A

00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A

00:0a: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A

RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize

Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2

ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx

PIIX4: IDE controller at PCI slot 0000:00:07.1

PIIX4: chipset revision 1

PIIX4: not 100% native mode: will probe irqs later

ide0: BM-DMA at 0x1050-0x1057, BIOS settings: hda:DMA, hdb:pio

Probing IDE interface ide0...

hda: VMware Virtual IDE CDROM Drive, ATAPI CD/DVD-ROM drive

ide0 at 0x1f0-0x1f7,0x3f6 on irq 14

Probing IDE interface ide1...

ide-floppy driver 0.99.newide

usbcore: registered new driver hiddev

usbcore: registered new driver usbhid

drivers/usb/input/hid-core.c: v2.6:USB HID core driver

PNP: PS/2 Controller at 0x60,0x64 irq 1,12

serio: i8042 KBD port at 0x60,0x64 irq 1

serio: i8042 AUX port at 0x60,0x64 irq 12

mice: PS/2 mouse device common for all mice

md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27

md: bitmap version 4.39

TCP bic registered

Initializing IPsec netlink socket

NET: Registered protocol family 1

NET: Registered protocol family 17

ACPI: (supports S0 S1 S4 S5)

Freeing unused kernel memory: 196k freed

Write protecting the kernel read-only data: 476k

input: AT Translated Set 2 keyboard as /class/input/input0

ohci_hcd: 2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)

USB Universal Host Controller Interface driver v3.0

SCSI subsystem initialized

Fusion MPT base driver 3.04.05

Copyright (c) 1999-2007 LSI Corporation

Fusion MPT SPI Host driver 3.04.05

GSI 16 sharing vector 0xA9 and IRQ 16

ACPI: PCI Interrupt 0000:00:10.0[A] -> GSI 17 (level, low) -> IRQ 169

mptbase: ioc0: Initiating bringup

ioc0: LSI53C1030 B0: Capabilities=

scsi0 : ioc0: LSI53C1030 B0, FwRev=00000000h, Ports=1, MaxQ=128, IRQ=169

Vendor: VMware Model: Virtual disk Rev: 1.0

Type: Direct-Access ANSI SCSI revision: 02

target0:0:0: Beginning Domain Validation

target0:0:0: Domain Validation skipping write tests

target0:0:0: Ending Domain Validation

target0:0:0: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)

SCSI device sda: 15872000 512-byte hdwr sectors (8126 MB)

sda: test WP failed, assume Write Enabled

sda: cache data unavailable

sda: assuming drive cache: write through

SCSI device sda: 15872000 512-byte hdwr sectors (8126 MB)

sda: test WP failed, assume Write Enabled

sda: cache data unavailable

sda: assuming drive cache: write through

sda: sda1 sda2

sd 0:0:0:0: Attached scsi disk sda

Vendor: VMware Model: Virtual disk Rev: 1.0

Type: Direct-Access ANSI SCSI revision: 02

target0:0:1: Beginning Domain Validation

target0:0:1: Domain Validation skipping write tests

target0:0:1: Ending Domain Validation

target0:0:1: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)

SCSI device sdb: 2097152 512-byte hdwr sectors (1074 MB)

sdb: test WP failed, assume Write Enabled

sdb: cache data unavailable

sdb: assuming drive cache: write through

SCSI device sdb: 2097152 512-byte hdwr sectors (1074 MB)

sdb: test WP failed, assume Write Enabled

sdb: cache data unavailable

sdb: assuming drive cache: write through

sdb: sdb1

sd 0:0:1:0: Attached scsi disk sdb

Vendor: HP Model: MSA2012i Rev: J210

Type: Direct-Access ANSI SCSI revision: 05

target0:0:2: Beginning Domain Validation

target0:0:2: Domain Validation skipping write tests

target0:0:2: Ending Domain Validation

target0:0:2: FAST-80 SCSI 80.0 MB/s DT (12.5 ns, offset 127)

SCSI device sdc: 97658880 512-byte hdwr sectors (50001 MB)

sdc: Write Protect is off

sdc: Mode Sense: 93 00 00 08

SCSI device sdc: drive cache: write back

SCSI device sdc: 97658880 512-byte hdwr sectors (50001 MB)

sdc: Write Protect is off

sdc: Mode Sense: 93 00 00 08

SCSI device sdc: drive cache: write back

sdc: sdc1

sd 0:0:2:0: Attached scsi disk sdc

Vendor: VMware Model: Virtual disk Rev: 1.0

Type: Direct-Access ANSI SCSI revision: 02

target0:0:3: Beginning Domain Validation

target0:0:3: Domain Validation skipping write tests

target0:0:3: Ending Domain Validation

target0:0:3: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)

SCSI device sdd: 96468992 512-byte hdwr sectors (49392 MB)

sdd: test WP failed, assume Write Enabled

sdd: cache data unavailable

sdd: assuming drive cache: write through

SCSI device sdd: 96468992 512-byte hdwr sectors (49392 MB)

sdd: test WP failed, assume Write Enabled

sdd: cache data unavailable

sdd: assuming drive cache: write through

sdd: unknown partition table

sd 0:0:3:0: Attached scsi disk sdd

libata version 3.00 loaded.

device-mapper: uevent: version 1.0.3

device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com

input: ImPS/2 Generic Wheel Mouse as /class/input/input1

input: AT Translated Set 2 keyboard as /class/input/input2

kjournald starting. Commit interval 5 seconds

EXT3-fs: mounted filesystem with ordered data mode.

SELinux: Disabled at runtime.

SELinux: Unregistering netfilter hooks

audit(1214048703.838:2): selinux=0 auid=4294967295 ses=4294967295

hda: ATAPI 1X CD-ROM drive, 32kB Cache, UDMA(33)

Uniform CD-ROM driver Revision: 3.20

piix4_smbus 0000:00:07.3: Found 0000:00:07.3 device

piix4_smbus 0000:00:07.3: Host SMBus controller not enabled!

pcnet32.c:v1.32 18.Mar.2006 tsbogend@alpha.franken.de

GSI 17 sharing vector 0xB1 and IRQ 17

ACPI: PCI Interrupt 0000:00:11.0[A] -> GSI 18 (level, low) -> IRQ 177

pcnet32: PCnet/PCI II 79C970A at 0x1400, 00 50 56 b4 4c 34 assigned IRQ 177.

eth0: registered as PCnet/PCI II 79C970A

pcnet32: 1 cards_found.

input: PC Speaker as /class/input/input3

parport: PnPBIOS parport detected.

parport0: PC-style at 0x378, irq 7

shpchp: Standard Hot Plug PCI Controller Driver version: 0.4

Floppy drive(s): fd0 is 1.44M

FDC 0 is a post-1991 82077

sd 0:0:0:0: Attached scsi generic sg0 type 0

sd 0:0:1:0: Attached scsi generic sg1 type 0

sd 0:0:2:0: Attached scsi generic sg2 type 0

sd 0:0:3:0: Attached scsi generic sg3 type 0

lp0: using parport0 (interrupt-driven).

lp0: console ready

ACPI: AC Adapter (on-line)

ACPI: Power Button (FF)

ibm_acpi: ec object not found

md: Autodetecting RAID arrays.

md: autorun ...

md: ... autorun DONE.

device-mapper: multipath: version 1.0.5 loaded

EXT3 FS on dm-0, internal journal

kjournald starting. Commit interval 5 seconds

EXT3 FS on sdb1, internal journal

EXT3-fs: mounted filesystem with ordered data mode.

kjournald starting. Commit interval 5 seconds

EXT3 FS on dm-1, internal journal

EXT3-fs: mounted filesystem with ordered data mode.

kjournald starting. Commit interval 5 seconds

EXT3 FS on sda1, internal journal

EXT3-fs: mounted filesystem with ordered data mode.

Adding 1638392k swap on /dev/local/swap. Priority:-1 extents:1 across:1638392k

IA-32 Microcode Update Driver: v1.14a

eth0: link up

Bluetooth: Core ver 2.10

NET: Registered protocol family 31

Bluetooth: HCI device and connection manager initialized

Bluetooth: HCI socket layer initialized

Bluetooth: L2CAP ver 2.8

Bluetooth: L2CAP socket layer initialized

Bluetooth: RFCOMM socket layer initialized

Bluetooth: RFCOMM TTY layer initialized

Bluetooth: RFCOMM ver 1.8

Bluetooth: HIDP (Human Interface Emulation) ver 1.1

NET: Registered protocol family 10

lo: Disabled Privacy Extensions

IPv6 over IPv4 tunneling driver

eth0: no IPv6 routers present

SCSI device sdd: 96468992 512-byte hdwr sectors (49392 MB)

sdd: test WP failed, assume Write Enabled

sdd: cache data unavailable

sdd: assuming drive cache: write through

sdd: sdd1

SCSI device sdd: 96468992 512-byte hdwr sectors (49392 MB)

sdd: test WP failed, assume Write Enabled

sdd: cache data unavailable

sdd: assuming drive cache: write through

sdd: sdd1

kjournald starting. Commit interval 5 seconds

EXT3 FS on sdd1, internal journal

EXT3-fs: mounted filesystem with ordered data mode.

kjournald starting. Commit interval 5 seconds

EXT3 FS on sdd1, internal journal

EXT3-fs: mounted filesystem with ordered data mode.

I also constantly see logs in /var/log/vmkernel like this:

Jun 21 11:48:04 esx1 vmkernel: 0:21:55:02.428 cpu0:1077)iSCSI: session 0x352180a0 recv_cmd 0x3f64bc00, cdb 0x0, status 0x2, response 0x0, senselen 42, key 06, ASC/ASCQ 29/03, itt 2764 task 0x352019a0 to (0 0 1 5), san01.mydomain.com

Jun 21 11:48:04 esx1 vmkernel: iSCSI: Sense 70000600 0000000a 0000000

same for /var/log/vmkwarning:

Jun 21 11:31:45 esx1 vmkernel: 0:21:38:43.666 cpu1:1040)WARNING: ScsiPath: 3187: Plugin 'legacyMP' had an error (Not support

ed) while claiming path 'vmhba32:C0:T2:L0'.Skipping the path.

Jun 21 11:31:45 esx1 vmkernel: 0:21:38:43.666 cpu1:1040)WARNING: SCSI: 279: SCSI device type 0xd is not supported. Cannot cr

eate target vmhba32:3:0

Jun 21 11:31:45 esx1 vmkernel: 0:21:38:43.666 cpu1:1040)WARNING: SCSI: 1249: LegacyMP Plugin could not claim path: vmhba32:

3:0. Not supported

Jun 21 11:31:45 esx1 vmkernel: 0:21:38:43.666 cpu1:1040)WARNING: ScsiPath: 3187: Plugin 'legacyMP' had an error (Not support

ed) while claiming path 'vmhba32:C0:T3:L0'.Skipping the path.

Now, as far as multipathing is concerned, the SAN is an active/active array but only configured on 1 NIC at the moment because it's a preliminary setup before we move all servers to the datacenter. Could this be the problem? I filled in all 4 IP's for the SAN in the dynamic discovery tab of the software iSCSI initator and it should be able to find them all, as the SAN is fully patched to the switch. (this will of course be 2 switches in the final configuration).

Koen

0 Kudos