Astro
Contributor
Contributor

ESX4 Broadcom performance issue

Hi all

My configuration: 2 x HPDL 380 G5, vSphere Standard, vCenter Foundation, MSA1000

Both servers are equipped with 1 HP NC373i Multifunction Gigabit Server

Adapter (embedded - 2 ports) and 1 HP NC360T (added - 2 ports). The

first is identify by ESX as Intel Corporation 82571EB Gigabit

Ethernet Controller and the second as Broadcom Corporation Broadcom

NetXtreme II BCM5708 1000Base-T. The NIC are connected to the same Gb switch.

When I copy a test file (1GB) from a physical server to a VM (Windos

2003) that is "binded" to the Intel NIC, performance are 40-42 MB/s.

The same test from the same server to other VM (Windows 2003, XP,

Linux) performs 20-22 MB/s.

Maybe a IRQ or a driver problem ?

cat /proc/vmware/interrupts (Broadcom are vmnic0 and vmnic1)

Vector

PCPU 0

PCPU 1

PCPU 2

PCPU 3

PCPU 4

PCPU 5

PCPU 6

PCPU 7

0x21:

0

0

0

0

0

0

0

0

VMK ACPI Interrupt

0x22:

1425133

1310196

980490

919533

742309

764753

976164

955231

VMK cciss0

0x29:

120

0

0

0

0

0

0

0

COS irq 1 (ISA edge)

0x31:

3

0

0

0

0

0

0

0

<COS irq 3 (ISA edge)>

0x39:

4

0

0

0

0

0

0

0

<COS irq 4 (ISA edge)>

0x41:

1

0

0

0

0

0

0

0

<COS irq 6 (ISA edge)>

0x49:

1

0

0

0

0

0

0

0

COS irq 8 (ISA edge)

0x51:

0

0

0

0

0

0

0

0

<COS irq 11 (ISA edge)>

0x59:

157

0

0

0

0

0

0

0

COS irq 12 (ISA edge)

0x61:

0

0

0

0

0

0

0

0

<COS irq 13 (ISA edge)>

0x69:

1275

5883

0

1020

1

0

4080

0

<COS irq 14 (ISA edge)>, VMK libata

0x71:

0

0

0

0

1

0

0

0

<COS irq 15 (ISA edge)>, VMK libata

0x79:

0

0

0

0

0

0

0

0

<COS irq 16 (PCI level)>, VMK uhci_hcd:usb2, VMK ehci_hcd:usb1

0x81:

0

0

0

0

0

0

0

0

<COS irq 17 (PCI level)>, VMK uhci_hcd:usb3

0x89:

6621097

3820460

21339423

22894321

15919549

14392462

14777265

15413282

<COS irq 18 (PCI level)>, VMK qla2xxx, VMK uhci_hcd:usb4

0x91:

257081

123218

249195

227139

159631

332055

172800

166005

<COS irq 19 (PCI level)>, VMK qla2xxx, VMK uhci_hcd:usb5

0x99:

0

0

0

0

0

0

0

0

<COS irq 22 (PCI level)>

0xa1:

0

0

0

0

0

0

0

0

<COS irq 20 (PCI level)>

0xa9:

76

0

0

0

0

0

0

0

<COS irq 21 (PCI level)>, VMK uhci_hcd:usb6

0xb1:

0

0

0

0

434940540

0

0

0

VMK vmnic0

0xb9:

0

0

0

0

3791755

0

0

0

VMK vmnic1

0xc1:

1683702

4355657

916470

1407856

520711

3391248

1266330

3719433

VMK vmnic2

0xc9:

154275

214610

102000

55080

58398

52275

68595

58650

VMK vmnic3

0xdf:

610585043

679585878

689560293

691393738

692551003

696278389

691877800

694125796

VMK timer

0xe1:

70913

310979

1297883

991128

778091

837844

817038

768270

VMK monitor

0xe9:

142000793

110612476

259330606

281321494

136070039

188644905

193187899

197254694

VMK resched

0xf1:

42010

278812

323655

291608

414655

398176

346804

311322

VMK tlb

0xf9:

30107670

0

0

0

0

0

0

0

VMK noop

0xfc:

0

0

0

0

0

0

0

0

VMK thermal

0xfd:

0

0

0

0

0

0

0

0

VMK lint1

0xfe:

0

0

0

0

0

0

0

0

VMK error

0xff:

0

0

0

0

0

0

0

0

VMK spurious

Many thanks and regards

Max

Tags (3)
0 Kudos
9 Replies
athlon_crazy
Virtuoso
Virtuoso

Looks like this 2 vectors (0xb1: & 0xb9:) doesn't distributed interrupt equally among available cpu.

vcbMC-1.0.6 Beta

vcbMC-1.0.7 Lite

http://www.no-x.org
0 Kudos
VMmatty
Virtuoso
Virtuoso

Did you check the obvious networking related things outside of ESX? I can't tell you how many times I've seen something that I initially think is a complicated hardware, software, or driver issue end up being a simple duplex mismatch on the switch.

Matt | http://www.thelowercasew.com | @mattliebowitz
Astro
Contributor
Contributor

Thanks athlon_crazy

Yes, I think that the problem is this.

Suggestion ?

Regards

0 Kudos
Astro
Contributor
Contributor

Thanks VMmatty

Yes, I've checked all the "external" configuration.

Regards

0 Kudos
athlon_crazy
Virtuoso
Virtuoso

What result you get from below command :

  1. cat /proc/vmware/pci | grep 0xb1 and

  2. cat /proc/vmware/pci | grep 0xb9

vcbMC-1.0.6 Beta

vcbMC-1.0.7 Lite

http://www.no-x.org

http://www.no-x.org
0 Kudos
Astro
Contributor
Contributor

What result you get from below command :

  1. cat /proc/vmware/pci | grep 0xb1 and

  2. cat /proc/vmware/pci | grep 0xb9

vcbMC-1.0.6 Beta

vcbMC-1.0.7 Lite

http://www.no-x.org

Hi

It seems that in ESX4 there's no /proc/vmware/pci: the command returns

cat: /proc/vmware/pci: No such file or directory

the /proc/vmware contains

-rw-rr 1 root root 0 Sep 14 16:53 chipset

-rw-rr 1 root root 0 Sep 14 16:53 cpuinfo

-rw-rr 1 root root 0 Sep 14 16:53 interrupts

-rw-rr 1 root root 0 Sep 14 16:53 intr-tracker

-rw-rr 1 root root 0 Sep 14 16:53 log

dr-xr-xr-x 2 root root 0 Sep 14 16:53 migration

-rw-rr 1 root root 0 Sep 14 16:53 procstats

dr-xr-xr-x 2 root root 0 Sep 14 16:53 pshare

-rrr-- 1 root root 0 Sep 14 16:53 rootFsVMDKPath

dr-xr-xr-x 3 root root 0 Sep 14 16:53 sched

-rrr-- 1 root root 0 Sep 14 16:53 stats

-rw-rr 1 root root 0 Sep 14 16:53 thermmon

-rw-rr 1 root root 0 Sep 14 16:53 timers

-rw-rr 1 root root 0 Sep 14 16:53 uptime

-rw-rr 1 root root 0 Sep 14 16:53 version

dr-xr-xr-x 26 root root 0 Sep 14 16:53 vm

-rrr-- 1 root root 0 Sep 14 16:53 vmkstor

-rw-rr 1 root root 0 Sep 14 16:53 watchpoints

Thanks

0 Kudos
ROMB
Contributor
Contributor

i have the same problem. any updates on this post?

0 Kudos
meistermn
Expert
Expert

What is now the output of cat /proc/vmware/interrupts after you fixed the duplex mismatch?

0 Kudos
ROMB
Contributor
Contributor

  1. cat /proc/vmware/interrupts

Vector PCPU 0 PCPU 1 PCPU 2 PCPU 3

0x21: 0 0 0 0 VMK ACPI Interrupt

0x29: 1 0 0 0 COS irq 1 (ISA edge)

0x31: 5 0 0 0 <COS irq 3 (ISA edge)>

0x39: 6 0 0 0 COS irq 4 (ISA edge)

0x41: 0 0 0 0 <COS irq 7 (ISA edge)>

0x49: 1 0 0 0 COS irq 8 (ISA edge)

0x51: 1 0 0 0 COS irq 12 (ISA edge)

0x59: 0 0 0 0 <COS irq 13 (ISA edge)>

0x61: 40291 53841 12751 20655 <COS irq 14 (ISA edge)>, VMK libata

0x69: 0 0 1 0 <COS irq 15 (ISA edge)>, VMK libata

0x71: 0 0 0 0 <COS irq 17 (PCI level)>

0x79: 25 0 0 0 <COS irq 22 (PCI level)>, VMK uhci_hcd:usb4, VMK uhci_hcd:usb2,

0x81: 661215 403504 105572 117558 <COS irq 23 (PCI level)>, VMK uhci_hcd:usb3

0x89: 2640783 1515480 1843748 1875024 <COS irq 30 (PCI level)>, VMK ioc0

0x91: 0 0 0 0 <COS irq 33 (PCI level)>

0x99: 0 0 0 230172067 VMK vmnic0

0xa1: 0 0 0 245465865 VMK vmnic1

0xdf: 169752608 217902666 258585164 253011804 VMK timer

0xe1: 26991 99843 164193 177964 VMK monitor

0xe9: 178770039 169068390 151438989 85429009 VMK resched

0xf1: 2055 29699 93647 102779 VMK tlb

0xf9: 14220734 0 0 0 VMK noop

0xfc: 0 0 0 0 VMK thermal

0xfd: 0 0 0 0 VMK lint1

0xfe: 0 0 0 0 VMK error

0xff: 0 0 0 0 VMK spurious

  1. ethtool vmnic0

Settings for vmnic0:

Supported ports:

Supported link modes: 10baseT/Half 10baseT/Full

100baseT/Half 100baseT/Full

1000baseT/Full

Supports auto-negotiation: Yes

Advertised link modes: 10baseT/Half 10baseT/Full

100baseT/Half 100baseT/Full

1000baseT/Full

Advertised auto-negotiation: Yes

Speed: 1000Mb/s

Duplex: Full

Port: Twisted Pair

PHYAD: 1

Transceiver: internal

Auto-negotiation: on

Supports Wake-on: g

Wake-on: g

Link detected: yes

  1. ethtool vmnic1

Settings for vmnic1:

Supported ports:

Supported link modes: 10baseT/Half 10baseT/Full

100baseT/Half 100baseT/Full

1000baseT/Full

Supports auto-negotiation: Yes

Advertised link modes: 10baseT/Half 10baseT/Full

100baseT/Half 100baseT/Full

1000baseT/Full

Advertised auto-negotiation: Yes

Speed: 1000Mb/s

Duplex: Full

Port: Twisted Pair

PHYAD: 1

Transceiver: internal

Auto-negotiation: on

Supports Wake-on: g

Wake-on: g

Link detected: yes

iSCSI speed test from VM:

  1. LANG=en_US.UTF-8 dd if=/dev/zero of=/root/tempfile bs=1M count=1024

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 37.0333 s, 29.0 MB/s

  1. LANG=en_US.UTF-8 dd if=/root/tempfile of=/dev/null bs=1M count=1024

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 21.292 s, 50.4 MB/s

iSCSI disk configuration:

4x2TB software SATA raid 5

6TB array is stripped into 4 logical volumes by 1.5 TB each. Datastore is created on first logical volume.

write speed into first volume on the storage server is 100MB/s and upper

0 Kudos