VMware Cloud Community
dwchan
Enthusiast
Enthusiast

Can we aggregate multiple NIC and multiple VLAn

on our VMWare host, we have 2 - dual NIC controller. It is possible to aggregrate all of those NIC to a single pipe (i.e. 4 x 100M), so that it is it now 400M through put total. In additional, once the new aggregrate is setup, we will assign multiple VLAN on that single aggregate (trunking). Can this be setup using a single virtual switch. So in a short, all VM guest connect to that virtual switch, will have the capacity of 400M, and if one nic fail, it can still go open the other 3 NIC. In addition, the VM guest on that single virtual switch can from difference VLAN.

dwc

Reply
0 Kudos
17 Replies
dominic7
Virtuoso
Virtuoso

Most of what you want is simple:

Create a vSwitch with 4 uplinks, on the phsyical switch side set up a "trunk" 802.1q or 802.3ad with the mulitple VLANS that you want to use. Then set up portgroups on the virtual switch that correspond to the VLANs assigned in the trunk. You will have 4 physical NICS in a bond that will provide both load balancing and redundancy.

Guests themselves cannot "switch" from one VLAN to another, but you can adjust which vlan they are on by editing the network properties of the VM. If you really want the VMs to be able to switch from one vlan to another you may have to use guest based VLAN tagging which is based on features of the OS on the guest.

Reply
0 Kudos
esiebert7625
Immortal
Immortal

Yes you can, you need to use 802.1Q VLAN tagging if your physical network switch supports it. You can add all 4 NIC's to the vSwitch for 400Mbps total bandwidth but a VM can never access more then one NIC at a time so it will be limited to 100Mbps.

See these links...

VMware ESX Server 3 802.1Q VLAN Solutions - http://www.vmware.com/pdf/esx3_vlan_wp.pdf

Networking Virtual Machines - http://download3.vmware.com/vmworld/2006/TAC9689-A.pdf

Networking Scenarios & Troubleshooting - http://download3.vmware.com/vmworld/2006/tac9689-b.pdf

ESX3 Networking Internals - http://www.vmware-tsx.com/download.php?asset_id=41

High Performance ESX Networking - http://www.vmware-tsx.com/download.php?asset_id=43

Network Throughput in a Virtual Infrastructure - http://www.vmware.com/pdf/esx_network_planning.pdf

fyi...if you find this post helpful, please award points using the Helpful/Correct buttons...thanks

Reply
0 Kudos
dwchan
Enthusiast
Enthusiast

What do you mean a VM guest can only access one NIC at a time? 2 situation.

1. If I have a single NIC assign to a VM guest, but the Virtual Switch has 4 (100MB) NIC bond to its (setup for aggregation on the backend), what is the maximum through put between the VM guest NIC and the virtual Switch? Would it be 100 or 400? Or is the VM guest traffic only go out to one of the NIC on the virtual switch?

2. What if I assist 2 NIC to my VM guest? Can you bond Virtual NIC inside a virtual guest?

dwc

Reply
0 Kudos
esiebert7625
Immortal
Immortal

It's 100, a VM can only be connected to 1 pNIC at a time so a VM can never use more than the Bandwidth of 1 NIC.

http://www.vmware.com/community/thread.jspa?messageID=632932&#632932

If you assign two NIC's to a VM you would have to have software on the VM's that supports bonding, never tried it but check out this post...

http://www.vmware.com/community/thread.jspa?messageID=561718&#561718

Reply
0 Kudos
Paul_Lalonde
Commander
Commander

NIC Express from Falconstor allows teaming of network adapters (virtual or physical) on a Windows or Linux system.

It works with any kind of network adapter.

http://www.falconstor.com/en/solutions/?pg=Products&sb=NIC

Not sure if bonding from a VM standpoint will offer any additional value, though.

Paul

Reply
0 Kudos
hicksj
Virtuoso
Virtuoso

You may also want to check out the following... it was just posted this week:

Multi-NIC Performance Tech Note

http://www.vmware.com/pdf/Multi-NIC_Performance.pdf

Reply
0 Kudos
Mike_Fink
Enthusiast
Enthusiast

Actually, one small issue here. DW asks what the bandwidth is between the VM and the virtual switch.

In that situation, the uplink speed does not matter. The speed between the VM and the VSwitch is reported as 1GB, and it is actually faster then that. How fast? It's backplane speed, as fast as the host can move data.

The ONLY time you would ever see that is if you have 2VMs on the same VSwitch communicating with one another. In that situation, the PNICs don't matter, as the traffic will not uplink at all, it will all pass across the VSwitch at backplane speed.

I think there is a way to bond the NICs inside the VM. And, yes, in that situation, you could get 200mb/s. However, I do not know if it is a supported configuration or not, or if there is a way (other then creating 2 VSwitches) to absolutely say that your going to get 200Mb/s (becuase there is a chance that both virtual NIC addresses could ARP down the same PNIC).

The VSwitch is, in fact, a switch. It works just like a switch does when you trunk multiple ports together between switches. The aggregate bandwidth between switches can be greater then the bandwidth of any single connection. However, no single connection can exceed the bandwith of and single trunk link.

Reply
0 Kudos
esiebert7625
Immortal
Immortal

I guess that would be true if that is what he meant, he doesn't really say if his VM's would be talking to external hosts or other VM's on the same vswitch. The way I understand it is:

Same vSwitch is 'Routed Locally' doesn't hit any pNICs

Same vSwitch different PortGroup/VLAN same thing is 'Routed Locally' doesn't hit any pNICs

Between different vSwitches is 'Routed Externally' ie needs to leave one pNIC and come in the other pNIC of the other vSwitch.

Reply
0 Kudos
hicksj
Virtuoso
Virtuoso

I think there is a way to bond the NICs inside the

VM. And, yes, in that situation, you could get

200mb/s. However, I do not know if it is a supported

configuration or not

Its supported. See the tech note I posted above.

Reply
0 Kudos
dwchan
Enthusiast
Enthusiast

First, let me start of and say thanks a million for all the wonderful feedback, especially for a network rookie. However, there are still few things not quite clear to me

From what I gather in the discussion, I can either dedicate a entire virtual Switch to a VLAN, or assign port groups to the virtual Switch so that a single virtual Switch can handle multiple VLAN. So basically, depend on which port groups the virtual guest is connected to, that is the vlan it belong to. However, I am still confuse about the trucking portion.

1. On a given virtual switch, let say I have 4 physical NIC (gigabit) connect to its, and it has 2 port groups (vlan 100 and 200). The problem we are seeing, and think is by design, when the virtual guest power up, it will bind the virtual NIC from within the virtual machine to one of the 4 physical NIC and stay there. What we are seeing is that after a VM is power on, it will continue using a physical NIC that it bond to when it power on, ever through there are other physical NIC may be available or not being use. Are we doing something wrong? or will this only be enable after trucking is turn on back on the network switch?

2. In a physical world, if i have 2 NIC on my physical server, I will connect them back to the switch and use a OEM (in our case, HP) driver to bind the 2 NIC together for failover or aggregate the bandwith. I can easily add another virtual NIC to a VM guest, but how do I bond them together (or do I have to use a 3rd party software)? I know the connection between the virtual NIC and virtual switch is internal, but I presume a single virtual NIC inside a VM guest is only 1.0G since the physical NIC itself is only gigabits.

dwc

Reply
0 Kudos
esiebert7625
Immortal
Immortal

For #1 you need to change your load balancing settings from port group/mac to ip hash. Look under your vswitch properties on the NIC teaming tab.

For #2 you would have to use 3rd party software to bond them. The vswitch already has built in failover if you have multiple NIC's so you really do not need a OS bond for that. Also would you really use more then 1GB of bandwidth on the VM?

http://www.vmware.com/community/thread.jspa?messageID=580796&#580796

fyi....don't forget to award points for helpful/correct answers...thanks

Reply
0 Kudos
dwchan
Enthusiast
Enthusiast

we did set the network to ip hash. Basically, once the VM guest is power on, does it move from one physical NIC to another? or it will remain the same until a fail over?

dwc

Reply
0 Kudos
esiebert7625
Immortal
Immortal

With IP hash you have the best load balancing, here's how each method works and how it will select which NIC to use:

Route based on the originating port ID (default) \[PortBased]: With this method, VM’s outbound traffic is mapped to specific PhyNIC based on the portID of Virtual Port, to which this virtual machine is connected. This method is simple and fast and doesn’t require VMKernel to examine the frame for necessary information.

Route based on IP hash \[IPBased]: In this case NIC card is choosing based upon source and destination address of the outbound packet. This method has slightly overhead and is not compatible with all switches. but traffic is distributed across NICS

Route based on source MAC hash: Each virtual machine outbound traffic is mapped to phyNIC based upon the VM’s NIC MAC address. It has low overhead and is compatible with all switches but may not spread traffic evenly.

Reply
0 Kudos
dwchan
Enthusiast
Enthusiast

thank you for the information, but in all 3 cases, once the vm guest is power up, does the traffic go back and fore between all of the possible physical NIC that is tie to the Virtual Switch? What we are seeing is that once the VM are power up, it lock into just one physical NIC (base on the ARP entries on the switch) and never move

dwc

Reply
0 Kudos
esiebert7625
Immortal
Immortal

You also need to configure your physical switch to support whatever load balancing scheme you choose, see these posts...

http://www.vmware.com/community/thread.jspa?messageID=629827&#629827

http://virtrix.blogspot.com/2006/11/vmware-switch-load-balancing.html

Reply
0 Kudos
hicksj
Virtuoso
Virtuoso

Yes, with any of these three schemes, traffic between two devices is only going to take one path. (one scheme based on portID, one based on MAC, or one based on IP... any which way, it makes a decision and sticks with it)

Thus, I call it "load sharing" not load balancing. If two VM's run hot on the network and they happen to share the same pNIC (based on the above decision), the kernel will not shift one of the devices to another port. So you will still end up with potentially idle NICs while another is running full capacity.

Reply
0 Kudos
ynr78
Contributor
Contributor

Hi,

does any of you know how much nic's can I actually use within a NIC TEAMING policy ?

what would be the maximum number of nics I can use ?

also, what other limitiations do I have in a nic teaming policy (like Total Bandwith, using nics within one collosion domain, etc...)

Many thanks,

Yaniv R.

Reply
0 Kudos