ESX4 Broadcom performance issue

Astro · ‎09-11-2009

Hi all

My configuration: 2 x HPDL 380 G5, vSphere Standard, vCenter Foundation, MSA1000

Both servers are equipped with 1 HP NC373i Multifunction Gigabit Server

Adapter (embedded - 2 ports) and 1 HP NC360T (added - 2 ports). The

first is identify by ESX as Intel Corporation 82571EB Gigabit

Ethernet Controller and the second as Broadcom Corporation Broadcom

NetXtreme II BCM5708 1000Base-T. The NIC are connected to the same Gb switch.

When I copy a test file (1GB) from a physical server to a VM (Windos

2003) that is "binded" to the Intel NIC, performance are 40-42 MB/s.

The same test from the same server to other VM (Windows 2003, XP,

Linux) performs 20-22 MB/s.

Maybe a IRQ or a driver problem ?

cat /proc/vmware/interrupts (Broadcom are vmnic0 and vmnic1)

Vector	PCPU 0	PCPU 1	PCPU 2	PCPU 3	PCPU 4	PCPU 5	PCPU 6	PCPU 7
0x21:	0	0	0	0	0	0	0	0	VMK ACPI Interrupt
0x22:	1425133	1310196	980490	919533	742309	764753	976164	955231	VMK cciss0
0x29:	120	0	0	0	0	0	0	0	COS irq 1 (ISA edge)
0x31:	3	0	0	0	0	0	0	0	<COS irq 3 (ISA edge)>
0x39:	4	0	0	0	0	0	0	0	<COS irq 4 (ISA edge)>
0x41:	1	0	0	0	0	0	0	0	<COS irq 6 (ISA edge)>
0x49:	1	0	0	0	0	0	0	0	COS irq 8 (ISA edge)
0x51:	0	0	0	0	0	0	0	0	<COS irq 11 (ISA edge)>
0x59:	157	0	0	0	0	0	0	0	COS irq 12 (ISA edge)
0x61:	0	0	0	0	0	0	0	0	<COS irq 13 (ISA edge)>
0x69:	1275	5883	0	1020	1	0	4080	0	<COS irq 14 (ISA edge)>, VMK libata
0x71:	0	0	0	0	1	0	0	0	<COS irq 15 (ISA edge)>, VMK libata
0x79:	0	0	0	0	0	0	0	0	<COS irq 16 (PCI level)>, VMK uhci_hcd:usb2, VMK ehci_hcd:usb1
0x81:	0	0	0	0	0	0	0	0	<COS irq 17 (PCI level)>, VMK uhci_hcd:usb3
0x89:	6621097	3820460	21339423	22894321	15919549	14392462	14777265	15413282	<COS irq 18 (PCI level)>, VMK qla2xxx, VMK uhci_hcd:usb4
0x91:	257081	123218	249195	227139	159631	332055	172800	166005	<COS irq 19 (PCI level)>, VMK qla2xxx, VMK uhci_hcd:usb5
0x99:	0	0	0	0	0	0	0	0	<COS irq 22 (PCI level)>
0xa1:	0	0	0	0	0	0	0	0	<COS irq 20 (PCI level)>
0xa9:	76	0	0	0	0	0	0	0	<COS irq 21 (PCI level)>, VMK uhci_hcd:usb6
0xb1:	0	0	0	0	434940540	0	0	0	VMK vmnic0
0xb9:	0	0	0	0	3791755	0	0	0	VMK vmnic1
0xc1:	1683702	4355657	916470	1407856	520711	3391248	1266330	3719433	VMK vmnic2
0xc9:	154275	214610	102000	55080	58398	52275	68595	58650	VMK vmnic3
0xdf:	610585043	679585878	689560293	691393738	692551003	696278389	691877800	694125796	VMK timer
0xe1:	70913	310979	1297883	991128	778091	837844	817038	768270	VMK monitor
0xe9:	142000793	110612476	259330606	281321494	136070039	188644905	193187899	197254694	VMK resched
0xf1:	42010	278812	323655	291608	414655	398176	346804	311322	VMK tlb
0xf9:	30107670	0	0	0	0	0	0	0	VMK noop
0xfc:	0	0	0	0	0	0	0	0	VMK thermal
0xfd:	0	0	0	0	0	0	0	0	VMK lint1
0xfe:	0	0	0	0	0	0	0	0	VMK error
0xff:	0	0	0	0	0	0	0	0	VMK spurious

Many thanks and regards

Max

athlon_crazy · ‎09-11-2009

Looks like this 2 vectors (0xb1: & 0xb9:) doesn't distributed interrupt equally among available cpu.

vcbMC-1.0.6 Beta

vcbMC-1.0.7 Lite

http://www.no-x.org

VMmatty · ‎09-12-2009

Did you check the obvious networking related things outside of ESX? I can't tell you how many times I've seen something that I initially think is a complicated hardware, software, or driver issue end up being a simple duplex mismatch on the switch.

Matt | http://www.thelowercasew.com | @mattliebowitz

Astro · ‎09-14-2009

Thanks athlon_crazy

Yes, I think that the problem is this.

Suggestion ?

Regards

Astro · ‎09-14-2009

Thanks VMmatty

Yes, I've checked all the "external" configuration.

Regards

athlon_crazy · ‎09-14-2009

What result you get from below command :

cat /proc/vmware/pci | grep 0xb1 and
cat /proc/vmware/pci | grep 0xb9

vcbMC-1.0.6 Beta

vcbMC-1.0.7 Lite

http://www.no-x.org

Astro · ‎09-14-2009

What result you get from below command :
cat /proc/vmware/pci | grep 0xb1 and
cat /proc/vmware/pci | grep 0xb9
vcbMC-1.0.6 Beta
vcbMC-1.0.7 Lite
http://www.no-x.org

Hi

It seems that in ESX4 there's no /proc/vmware/pci: the command returns

cat: /proc/vmware/pci: No such file or directory

the /proc/vmware contains

-rw-rr 1 root root 0 Sep 14 16:53 chipset

-rw-rr 1 root root 0 Sep 14 16:53 cpuinfo

-rw-rr 1 root root 0 Sep 14 16:53 interrupts

-rw-rr 1 root root 0 Sep 14 16:53 intr-tracker

-rw-rr 1 root root 0 Sep 14 16:53 log

dr-xr-xr-x 2 root root 0 Sep 14 16:53 migration

-rw-rr 1 root root 0 Sep 14 16:53 procstats

dr-xr-xr-x 2 root root 0 Sep 14 16:53 pshare

-rrr-- 1 root root 0 Sep 14 16:53 rootFsVMDKPath

dr-xr-xr-x 3 root root 0 Sep 14 16:53 sched

-rrr-- 1 root root 0 Sep 14 16:53 stats

-rw-rr 1 root root 0 Sep 14 16:53 thermmon

-rw-rr 1 root root 0 Sep 14 16:53 timers

-rw-rr 1 root root 0 Sep 14 16:53 uptime

-rw-rr 1 root root 0 Sep 14 16:53 version

dr-xr-xr-x 26 root root 0 Sep 14 16:53 vm

-rrr-- 1 root root 0 Sep 14 16:53 vmkstor

-rw-rr 1 root root 0 Sep 14 16:53 watchpoints

Thanks

ROMB · ‎04-01-2010

i have the same problem. any updates on this post?

meistermn · ‎04-02-2010

What is now the output of cat /proc/vmware/interrupts after you fixed the duplex mismatch?

ROMB · ‎04-04-2010

cat /proc/vmware/interrupts

Vector PCPU 0 PCPU 1 PCPU 2 PCPU 3

0x21: 0 0 0 0 VMK ACPI Interrupt

0x29: 1 0 0 0 COS irq 1 (ISA edge)

0x31: 5 0 0 0 <COS irq 3 (ISA edge)>

0x39: 6 0 0 0 COS irq 4 (ISA edge)

0x41: 0 0 0 0 <COS irq 7 (ISA edge)>

0x49: 1 0 0 0 COS irq 8 (ISA edge)

0x51: 1 0 0 0 COS irq 12 (ISA edge)

0x59: 0 0 0 0 <COS irq 13 (ISA edge)>

0x61: 40291 53841 12751 20655 <COS irq 14 (ISA edge)>, VMK libata

0x69: 0 0 1 0 <COS irq 15 (ISA edge)>, VMK libata

0x71: 0 0 0 0 <COS irq 17 (PCI level)>

0x79: 25 0 0 0 <COS irq 22 (PCI level)>, VMK uhci_hcd:usb4, VMK uhci_hcd:usb2,

0x81: 661215 403504 105572 117558 <COS irq 23 (PCI level)>, VMK uhci_hcd:usb3

0x89: 2640783 1515480 1843748 1875024 <COS irq 30 (PCI level)>, VMK ioc0

0x91: 0 0 0 0 <COS irq 33 (PCI level)>

0x99: 0 0 0 230172067 VMK vmnic0

0xa1: 0 0 0 245465865 VMK vmnic1

0xdf: 169752608 217902666 258585164 253011804 VMK timer

0xe1: 26991 99843 164193 177964 VMK monitor

0xe9: 178770039 169068390 151438989 85429009 VMK resched

0xf1: 2055 29699 93647 102779 VMK tlb

0xf9: 14220734 0 0 0 VMK noop

0xfc: 0 0 0 0 VMK thermal

0xfd: 0 0 0 0 VMK lint1

0xfe: 0 0 0 0 VMK error

0xff: 0 0 0 0 VMK spurious

ethtool vmnic0

Settings for vmnic0:

Supported ports:

Supported link modes: 10baseT/Half 10baseT/Full

100baseT/Half 100baseT/Full

1000baseT/Full

Supports auto-negotiation: Yes

Advertised link modes: 10baseT/Half 10baseT/Full

100baseT/Half 100baseT/Full

1000baseT/Full

Advertised auto-negotiation: Yes

Speed: 1000Mb/s

Duplex: Full

Port: Twisted Pair

PHYAD: 1

Transceiver: internal

Auto-negotiation: on

Supports Wake-on: g

Wake-on: g

Link detected: yes

ethtool vmnic1

Settings for vmnic1:

Supported ports:

Supported link modes: 10baseT/Half 10baseT/Full

100baseT/Half 100baseT/Full

1000baseT/Full

Supports auto-negotiation: Yes

Advertised link modes: 10baseT/Half 10baseT/Full

100baseT/Half 100baseT/Full

1000baseT/Full

Advertised auto-negotiation: Yes

Speed: 1000Mb/s

Duplex: Full

Port: Twisted Pair

PHYAD: 1

Transceiver: internal

Auto-negotiation: on

Supports Wake-on: g

Wake-on: g

Link detected: yes

iSCSI speed test from VM:

LANG=en_US.UTF-8 dd if=/dev/zero of=/root/tempfile bs=1M count=1024

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 37.0333 s, 29.0 MB/s

LANG=en_US.UTF-8 dd if=/root/tempfile of=/dev/null bs=1M count=1024

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 21.292 s, 50.4 MB/s

iSCSI disk configuration:

4x2TB software SATA raid 5

6TB array is stripped into 4 logical volumes by 1.5 TB each. Datastore is created on first logical volume.

write speed into first volume on the storage server is 100MB/s and upper

All

ESX4 Broadcom performance issue