VMware Cloud Community
Salvagg
Contributor
Contributor
‎02-05-2015 02:33 AM

ESXi 5.5 system with 40 Gbps Mellanox adapters and Switchs

Hello everybody,

   We have implemented a virtualization (ESXi 5.5) system with 40 Gbps Mellanox adapters and Switchs.

  Network Adapters:

NetworkAdapters.png

   Network Configuration:

Networking.png

  

   We are using two CentOS virtual machines and iperf to test bandwidth with the following parameters:

   VM1: iperf -s

   VM2: iperf -c VM1_IP -i 2 -t 60 -P 5

   Our best performance bandwidth it's 25 Gbps between two virtual machines.

   Host Characteristics:

ESXi5.5.png

   What we have tested:

   2 VMs, 4 vCpu, 4096 Gb RAM, vmxnet3, both vm on the same host connected to the same vSwitch, vSwitch without uplinks --> 25 Gbps

   2 VMs, 4 vCpu, 4096 Gb RAM, vmxnet3, both vm on the same host connected to the same vSwitch, vSwitch with uplinks --> 25 Gbps

   2 VMs, 4 vCpu, 4096 Gb RAM, vmxnet3, each vm on a different host, both host connected to the same physical Switch --> 25 Gbps

   We have installed CentOs as Operating System on the physical Host instead of ESXi and we got 37,8 Gbps bandwidth transfers, so this is not a hardware problem.

Thanks you.

Reply
0 Kudos
4 Replies
schepp
Leadership
Leadership
‎02-05-2015 02:36 AM

Hi,

are you using jumbo frames? What MTU did you configure?

Tim

Reply
0 Kudos
Salvagg
Contributor
Contributor
‎02-05-2015 03:02 AM

Yes, we are using Jumbo Frames (mtu=9000) on vSwith and on physical switches.

Selección_157.png

Reply
0 Kudos
prasannag6
Enthusiast
Enthusiast
‎02-05-2015 06:16 AM

What is the NIC speed shown by other tools like ethtool, dmesg etc.,?

----------If you found this or any other answer helpful, please consider to award points (use Correct or Helpful buttons). Regards, Prasanna----------
Reply
0 Kudos
Salvagg
Contributor
Contributor
‎02-06-2015 04:08 AM

This is the ethtool info:

~ # esxcli network nic list

Name          PCI Device     Driver     Link  Speed  Duplex  MAC Address         MTU  Description                                     

------------  -------------  ---------  ----  -----  ------  -----------------  ----  -------------------------------------------------

vmnic0        0000:011:00.0  mlx4_en    Up    40000  Full    f4:52:14:01:03:00  9000  Mellanox Technologies MT27500 Family [ConnectX-3]

vmnic1000002  0000:011:00.0  mlx4_en    Up    40000  Full    f4:52:14:01:03:01  9000  Mellanox Technologies MT27500 Family [ConnectX-3]

vusb0         Pseudo         cdc_ether  Up       10  Half    42:f2:e9:c0:b0:27  1500  Unknown Unknown                                 

~ # ethtool vmnic0

Settings for vmnic0:

    Supported ports: [ TP ]

    Supported link modes:   1000baseT/Full

    Supports auto-negotiation: Yes

    Advertised link modes:  1000baseT/Full

    Advertised auto-negotiation: Yes

    Speed: Unknown! (40000)

    Duplex: Full

    Port: Twisted Pair

    PHYAD: 0

    Transceiver: internal

    Auto-negotiation: on

    Supports Wake-on: d

    Wake-on: d

    Current message level: 0x00000014 (20)

    Link detected: yes

This is dmesg info:

2015-01-28T09:29:18.462Z cpu9:33587)skb_mem_info mempool for module mlx4_en created - max size 23068672

2015-01-28T09:29:18.462Z cpu9:33587)module heap: using memType 2

2015-01-28T09:29:18.462Z cpu9:33587)module heap vmklnx_mlx4_en: creation succeeded. id = 0x410ae14c4000

<6>mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.9.9.0 (Dec-05-2013)

2015-01-28T09:29:18.661Z cpu9:33587)<6>mlx4_en 0000:11:00.0: Activating port:1

2015-01-28T09:29:18.661Z cpu9:33587)<4>mlx4_en: 0000:11:00.0: Port 1 : Initializing port

2015-01-28T09:29:18.661Z cpu9:33587)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_netq_register: registered net queue done

2015-01-28T09:29:18.661Z cpu9:33587)<4>mlx4_en: 0000:11:00.0: Port 1 : Using 16 TX rings

2015-01-28T09:29:18.661Z cpu9:33587)<4>mlx4_en: 0000:11:00.0: Port 1 : Using 8 RX rings

2015-01-28T09:29:18.661Z cpu9:33587)VMK_PCI: 395: Device 0000:11:00.0 name: vmnic0

2015-01-28T09:29:18.661Z cpu9:33587)Uplink: 6510: Device vmnic0 not yet opened

ESC[7m2015-01-28T09:29:18.661Z cpu9:33587)WARNING: LinNet: GetNICDeviceProperties:8479: PCI device registered as pseudo-device 0:17:0.0ESC[0m

2015-01-28T09:29:18.661Z cpu9:33587)DMA: 612: DMA Engine 'vmnic0' created using mapper 'DMANull'.

2015-01-28T09:29:18.661Z cpu9:33587)Uplink: 8226: Opening device vmnic0

ESC[7m2015-01-28T09:29:18.661Z cpu23:32884)WARNING: LinNet: GetNICDeviceProperties:8479: PCI device registered as pseudo-device 0:17:0.0ESC[0m

2015-01-28T09:29:18.666Z cpu23:32884)IRQ: 540: 0x47 <vmnic0-0> exclusive, flags 0x10

2015-01-28T09:29:18.666Z cpu23:32884)VMK_VECTOR: 218: Registered handler for  interrupt 0xff47, flags 0x10

2015-01-28T09:29:18.666Z cpu23:32884)IRQ: 540: 0x48 <vmnic0-1> exclusive, flags 0x10

2015-01-28T09:29:18.666Z cpu23:32884)VMK_VECTOR: 218: Registered handler for  interrupt 0xff48, flags 0x10

2015-01-28T09:29:18.667Z cpu23:32884)IRQ: 540: 0x49 <vmnic0-2> exclusive, flags 0x10

2015-01-28T09:29:18.667Z cpu23:32884)VMK_VECTOR: 218: Registered handler for  interrupt 0xff49, flags 0x10

2015-01-28T09:29:18.667Z cpu23:32884)IRQ: 540: 0x4a <vmnic0-3> exclusive, flags 0x10

2015-01-28T09:29:18.667Z cpu23:32884)VMK_VECTOR: 218: Registered handler for  interrupt 0xff4a, flags 0x10

2015-01-28T09:29:18.667Z cpu23:32884)IRQ: 540: 0x4b <vmnic0-4> exclusive, flags 0x10

2015-01-28T09:29:18.667Z cpu23:32884)VMK_VECTOR: 218: Registered handler for  interrupt 0xff4b, flags 0x10

2015-01-28T09:29:18.667Z cpu23:32884)IRQ: 540: 0x4c <vmnic0-5> exclusive, flags 0x10

2015-01-28T09:29:18.667Z cpu23:32884)VMK_VECTOR: 218: Registered handler for  interrupt 0xff4c, flags 0x10

2015-01-28T09:29:18.668Z cpu23:32884)IRQ: 540: 0x4d <vmnic0-6> exclusive, flags 0x10

2015-01-28T09:29:18.668Z cpu23:32884)VMK_VECTOR: 218: Registered handler for  interrupt 0xff4d, flags 0x10

2015-01-28T09:29:18.668Z cpu23:32884)IRQ: 540: 0x4e <vmnic0-7> exclusive, flags 0x10

2015-01-28T09:29:18.668Z cpu23:32884)VMK_VECTOR: 218: Registered handler for  interrupt 0xff4e, flags 0x10

2015-01-28T09:29:18.669Z cpu23:32884)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_create_rx_standard_queues: Configuring net queue steering

2015-01-28T09:29:18.671Z cpu23:32884)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_apply_all_filters: apply previous filters to all queues

2015-01-28T09:29:18.680Z cpu9:33587)<6>NETQ mlx4_en_netqop_get_version: version is: 2.0

2015-01-28T09:29:18.680Z cpu9:33587)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_netqop_get_features: get features returned RX/TX QUEUES

2015-01-28T09:29:18.680Z cpu9:33587)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_netqop_get_queue_count: num rx queues supported is 7

2015-01-28T09:29:18.680Z cpu9:33587)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_netqop_get_filter_count: num rx filters is 32

2015-01-28T09:29:18.680Z cpu9:33587)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_netqop_get_default_queue: default rx napi ptr 0x410ad5264a30, queue id 0-[65536]

2015-01-28T09:29:18.680Z cpu9:33587)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_netqop_get_supported_feat: rx queues support none queue features

2015-01-28T09:29:18.680Z cpu9:33587)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_netqop_get_supported_filter_class: rx queues support filter class by MAC addr

2015-01-28T09:29:18.680Z cpu9:33587)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_netqop_get_queue_count: num tx queues supported is 7

2015-01-28T09:29:18.680Z cpu9:33587)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_netqop_get_default_queue: default tx queue mapping 0, queue id 0-[131072]

2015-01-28T09:29:18.680Z cpu9:33587)Uplink: 6529: enabled port 0x2 with mac f4:52:14:01:03:00

2015-01-28T09:29:18.680Z cpu23:32884)NetPort: 1632: disabled port 0x2

2015-01-28T09:29:18.680Z cpu23:32884)<6>NETQ mlx4_en_netqop_get_version: version is: 2.0

2015-01-28T09:29:18.680Z cpu23:32884)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_netqop_get_features: get features returned RX/TX QUEUES

2015-01-28T09:29:18.680Z cpu23:32884)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_netqop_get_queue_count: num rx queues supported is 7

2015-01-28T09:29:18.680Z cpu23:32884)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_netqop_get_filter_count: num rx filters is 32

2015-01-28T09:29:18.680Z cpu23:32884)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_netqop_get_default_queue: default rx napi ptr 0x410ad5264a30, queue id 0-[65536]

2015-01-28T09:29:18.680Z cpu23:32884)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_netqop_get_supported_feat: rx queues support none queue features

2015-01-28T09:29:18.680Z cpu23:32884)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_netqop_get_supported_filter_class: rx queues support filter class by MAC addr

2015-01-28T09:29:18.680Z cpu23:32884)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_netqop_get_queue_count: num tx queues supported is 7

2015-01-28T09:29:18.680Z cpu23:32884)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_netqop_get_default_queue: default tx queue mapping 0, queue id 0-[131072]

2015-01-28T09:29:18.680Z cpu23:32884)Uplink: 6529: enabled port 0x2 with mac f4:52:14:01:03:00

2015-01-28T09:29:18.680Z cpu23:32884)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_alloc_tx_queue: allocated tx queue 1-[131073], queue mapping is 1

2015-01-28T09:29:18.680Z cpu23:32884)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_alloc_tx_queue: allocated tx queue 2-[131074], queue mapping is 2

2015-01-28T09:29:18.680Z cpu23:32884)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_alloc_tx_queue: allocated tx queue 3-[131075], queue mapping is 3

2015-01-28T09:29:18.680Z cpu23:32884)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_alloc_tx_queue: allocated tx queue 4-[131076], queue mapping is 4

2015-01-28T09:29:18.680Z cpu23:32884)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_alloc_tx_queue: allocated tx queue 5-[131077], queue mapping is 5

2015-01-28T09:29:18.680Z cpu23:32884)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_alloc_tx_queue: allocated tx queue 6-[131078], queue mapping is 6

2015-01-28T09:29:18.680Z cpu23:32884)<6>mlx4_en: 0000:11:00.0: Port 1 : NETQ mlx4_en_alloc_tx_queue: allocated tx queue 7-[131079], queue mapping is 7

2015-01-28T09:29:18.680Z cpu9:33587)<6>mlx4_en 0000:11:00.0: Activating port:2

2015-01-28T09:29:18.681Z cpu9:33587)<4>mlx4_en: 0000:11:00.0: Port 2 : Initializing port

2015-01-28T09:29:18.681Z cpu9:33587)<6>mlx4_en: 0000:11:00.0: Port 2 : NETQ mlx4_en_netq_register: registered net queue done

2015-01-28T09:29:18.681Z cpu9:33587)<4>mlx4_en: 0000:11:00.0: Port 2 : Using 16 TX rings

2015-01-28T09:29:18.681Z cpu9:33587)<4>mlx4_en: 0000:11:00.0: Port 2 : Using 8 RX rings

2015-01-28T09:29:18.681Z cpu9:33587)VMK_PCI: 395: Device 0000:11:00.0 name: vmnic0

2015-01-28T09:29:18.681Z cpu9:33587)LinNet: create_pseudo_dev_name:3925: maxCounter 1000100

Thanks.

Reply
0 Kudos