Networking in ESXi, would like to do Cisco Etherchannel

[macpiano]

I have 8 nics in the server. 1 for management, 2 for iSCSI and since I don't have another ESXi server in the building I would like to do the other 5 with  Etherchannel on a Cisco 4500 switch. I know to use the load balancing in Vmware as IP Hash in the vmware network settings. In fact I did that and went to the switch and the system events said that those ports were flapping. I guess because I don't have it configured correctly.  The problem is the Cisco Switch. It would not allow me to create an etherchannel because it said it had static ip addresses attached and the choices well frankly I had no clue.

My VMware instructor said Etherchannel was the only way to go. The VM docs say to assign VM's to different physical switches but I think they take the lazy way out.

Any pointers out there?

[tidaltides]

I haven't tried Cisco products with VM networking. Can I ask for any feedback regarding it?

[jof]

Hi,

Why have you decided to start using etherchannel? Is it just because your instructor mentioned it?

Using IP-Hash\etherchannel can be of benefit in some cases but it is not true load-balancing  and may not solve your issues (if there are any). If you do want to use etherchannel you should assign Layer 3 addresses to the port channel logical interface, not to the physical interfaces in the channel.

As far i know as the optimal port numbers for etherchannel are 2, 4 or 8 due to the way the traffic is dealt with by the switch.

If you use 5 ports, 3 of the ports will be used twice as much as the other 2 (unequal load balancing). With 4 port ports they would all be utilised the same amount (so if you decide to go with it keep this in mind). Also, with a 5 port etherchannel you won't have a "5 Gbps" link you will still have 5 x 1Gbps links any one of which can still get congested and two of which will always be under utilised.

What i am trying to say is, have a reason for implementing this other than someone told you it would be good.

Just an opinion

[timmehb]

macpiano,

Can you do a "show int" for the interfaces you are wanting to create an etherchannel with.

Whilst you can make this a layer 3 etherchannel (routed) I would have thought you'd want a layer 2 (switched) bond. The ports that you are trying to bond may be set up as layer 3 at the minute but can be changed.

[macpiano]

It was explained to me that if you use multiple nics without load balancing that a VM will pick one of the nics to do its business and will continue to use that until the server is rebooted. So if your highest bandwidth using VMs happen to pick the same nic then your networking is not optimized so to speak. I will even out the number of nics to 4 or 6 but when done correctly etherchannel should act as if it is a 4 gig or 6 gig connection.

[jof]

IP Hash\etherchannel is not true load balancing either in that it choses a route based on the source and destination IP adresses not on the load on a particluar link.

It can give you a better spread of traffic over your links if you have a VM talking to multiple other servers as the IP hash for each would be different. But if you have for example an app server communicating with a DB server it will pick a link and use this link for communication and no other link (one to one communication will always use the same link). Also, if the link it happens to chose is the most heavily used link, it will still use it as it does not care about load.

It also adds a small overhead for this calc and you have to configure your Cisco with etherchannel. I'm not saying don't do it but if you are not experiencing any network performance issues will this give you real benefits?

If you do decide to do it, have 2, 4 or 8 links because the Cisco switch will split the traffic into "8 buckets" so to get an even load and the best use of your ports you will need 2, 4, or 8 ports in the etherchannel.

[macpiano]

So you are saying to not use Etherchannel? If not exactly what are you suggesting is the best way to setup multiple nics in VMware and get the best utilization of them? I could set up more switches inside esxi and manaully assign what I think would be the best VMs for each nic. I would rather not use Etherchannel if I don't have to.

[jof]

I am not saying don't do it I am asking why you feel you need to do it? Are you currently experiancing network performance issues on your VMs with your current Port ID based routing? If not is it worth the effort to reconfigure your environment to solve a problem you don't have.

If the reason is just that you feel you want to get the best utilisation from the NICs from a load balancing perspective, then using IP Hash doesn't necessarily mean that this will happen, although it may be slightly better. To get load balancing based on utilisation you could use Load Based Teaming (LBT) but this requires that you are using a dVS.

I'm asking the question is it worth the effort to reconfigure your environment to solve a problem you don't have, only you can answer this. If you are confortable with Cisco switching it's not a big deal to configure but if you are not why give yourself needless headaches.

[macpiano]

Thanks, in reality it is probably not worth the bother. I guess it's one of those if I can make it work then I can sleep at night knowing I have it configured. After seeing your advice I can sleep both ways so to speak.

[macpiano]

I did get Etherchannel setup and it looks like it is working fine. I saw this doc before from Vmware but it was a little sketchy there in the middle of the doc.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100404...

It looked a little cartoonish and I kind of recognized that if you follow it it will set it up. I did the commands and went to Cisco Network Assistant and sure enough the etherchannel was there and configured with Mode on no LACP. The caveat is that CNA will not configure etherchannel correctly if at all. The other caveat is we only use VLAN 1 in our whole domain so when I followed the last step to configure VLAN 1 no ip address I lost my connection to manage the Cisco switch. Fortunately I was in the same building and went down with a laptop and console cable.

If I see any issues I will post back. This is my new ESXi host that I'm building so I can blow it up if I need to.

[rickardnobel]

macpiano wrote:

Thanks, in reality it is probably not worth the bother.

I agree that if you have no bandwidth problems at the moment there is no real gain in using Etherchannel / IP hash. It will complicate your setup and also have a higher risk of later failure (if something would change on the physical switch for example.) A single incorrectly attached cable could cause a lot of damage.

If using default Port ID you could also connect your vmnics to two physical switches and gain very important high availability.