VMware Cloud Community
donikatz
Enthusiast
Enthusiast

IP Hash and standby NICs

Just upgraded to vCenter 4.1 and for the first time am seeing a warning in our ESX network config: "The IP hash based load balancing does not support Standby uplink physical adapters. Change all Standby uplinks to Active status."

Because our switches are not stacked but we want to use port channels, we have a 4x pNIC vSwitch for VM traffic, two NICs on a port channel on pSwitch1, two NICs on a port channel on pSwitch2. The two connected to one pSwitch are Active, the two connected to another pSwitch are Standby to avoid flapping (the active/standby choice is manually alternated per VLAN port group to balance across the switches).

This seems to work fine in pSwitch redundancy tests and I'd never seen such a warning message prior to vCenter 4.1. I assume the warning is just because in most situations Active/Passive with IP hash is incorrect. But any reason this is now a problem in our case? Thanks

Tags (1)
Reply
0 Kudos
21 Replies
vtoxbw
Contributor
Contributor

Did you ever find the answer to this question? I'm now seeing the same message and wondering if it's ok to configure it this way.

Reply
0 Kudos
donikatz
Enthusiast
Enthusiast

Ah, got so wrapped up in things, I don't recall if I ever actually put in a VMW support ticket (I'll have to check)*. After a VMW consultant told me they also continue to use the same config at customers without problems, I just went forward with it as planned and as usual.

I assume the warning was added because it's technically correct: the underlying port groups aren't load balanced by IP hash across the configured vSwitch's 4-port team, instead I'm manually splitting and balancing the team's underlying port groups 2+2 (so the overall load balance isn't absolutely by IP Hash nor optimal). But if you're smart enough to cheat by manually setting the teams in the underlying port groups, I don't see what the problem is. Seems to me it's a CYA by VMW from what their support logs are telling them. But I don't know for sure.

*Edit: Confirmed I never did get around to putting in an SR. But I just did now. Thanks for the reminder! I'll post back when I hear something.

Reply
0 Kudos
athlon_crazy
Virtuoso
Virtuoso

I forgot where I read before (perhaps network best practice documentation by vmware), IP hash for vNetwork require all vmnic adapters participated for load balance must "active"

http://www.no-x.org
Reply
0 Kudos
donikatz
Enthusiast
Enthusiast

Clearly vCenter is programmed to agree with you, but since Active/Passive with IP Hash has worked and continues to work, what's the technical info behind the warning (or "best practice" if indeed it is)? I don't see any reason why there would be a problem using IP Hash with manually designated Active/Passive on the port group level. This allows you the benefits of port channel + physical switch redundancy; what's the downside? If you can find where you read this and a technical explanation, that would be helpful. Thanks

Reply
0 Kudos
athlon_crazy
Virtuoso
Virtuoso

Hope this will help you. I found it in advance networking troubleshooting doc :

http://www.no-x.org
Reply
0 Kudos
donikatz
Enthusiast
Enthusiast

Ah thanks, hadn't seen this. But that isn't true when the standby NICs are on a different physical switch. The switch with the Active NICs doesn't have the Standby NICs as part of the LAG, only the ESX does. In that case, neither the switch nor the ESX will attempt to use those standby NICs. So again, I think VMware's docs and warnings are assuming you're doing a normal config, rather than "cheating" with two switches and manual port group override. I still don't see any technical problem with it, and it works in production.

Reply
0 Kudos
vtoxbw
Contributor
Contributor

I opened a case yesterday and spoke with a couple different engineer's on this. The answer I got was that it's not supported and never has been. I had spoken to engineer's in the past and they had said this was a perfectly fine way of setting up the network for redundancy so I was a little annoyed at this answer. The reason that the engineer gave was that if it is setup with two switches and say 4pnics where 2 pnics (active etherchannel and IP Hash routing) go to switch A and 2 pnics (standby etherchannel and IP Hash routing) go to switch B. (As in the Drawing below) Now you have one of the pnics fail in the active etherchannel, then one of the pnics on the other switch that was standby will be turned to active. At this point you would have 1 pnic in each etherchannel going to each switch and assuming you have the switches trunked together you will start seeing mac address flopping. My point was that I understood this and created it this way as the switches are more likely to fail than the Nics in the server, and we are more likely to be paged when a nic fails in a server so that an engineer can fix this simply by disabling the active pnic on the original failing etherchannel. In my testing our environment may be a little slow during this time, but everything stays up and running. The engineer's suggestion is to run all 4 pnics active to both switches using "route based on originating virtual port ID". So after I got off of the phone with them I realized that I wasn't sure if this would work with our particular setup (though no reason comes to mind yet), and will need to do some testing. It doesn't seem as though we would lose much functionality with the exception that no single server can flood more than 1 GB of data to the switches where before a server could send up to 2GB of data simultaneously. This shouldn't be a problem in our environment, but it could be especially in larger environments.

I am going to switch our environment to the VMWare supported way just because I don't like being on a support call where the engineer can say it's because you have done something that isn't supported. I will post back to this thread if I run into any problems or caveat's setting it up this way.

Reply
0 Kudos
donikatz
Enthusiast
Enthusiast

I just got off the phone with a VMW support as well. He said it's not a supported config, but it certainly can work, they're aware of people using it, and I can just ignore the warning. But I inferred from some other comments that he probably wasn't as savvy as your tech, so I'm not too confident in his answer.

Your point re NIC failure vs switch failure is an excellent one and clarifies why VMware would warn not to do it and would not support it (would be nice if they'd simply explained that in the warnings). I hadn't specifically thought of that, but a way to even further mitigate that risk would be to have separate NIC cards for each pair of ports. The odds of one interface on a NIC (and not the whole card) going down are slim. Of course your hardware arrangement would need to allow for that without compromising other redundancy, such as SC.

Yes, you could certainly switch back to the default of route based on virtual port ID, but the downside is that you would lose load balancing and only have redundancy. That algorithm means the VM attaches to the first available NIC on boot and will only switch if that NIC becomes unavailable. So you will likely wind up with some of your NICs heavily loaded while others are underutilized. (Of course you could always manually Active/Passive on the VLAN port group level like we're doing for IP Hash to manually load balance to a degree.)

The ideal scenario would be two physical switches that act as a logical switch, such as a stack, but stacks of course have their own pros/cons vs independently redundant switches. In other words, no scenario is ever perfect and we have to just do our best to meet out specific needs.

Reply
0 Kudos
rickardnobel
Champion
Champion

donikatz wrote: >The odds of one interface on a NIC (and not the whole card) going down are slim.

It could perhaps happen that someone at some time would disconnect one the network cables for some reason. If that would happen then your whole network access would suffer a lot since the MAC flapping will be massive on the physical switches. Depending on the vendor it could even temporarily shutdown the ports.

I belive there is a quite high risk to have IP Hash with four cards going to two pSwitches even with Active/Standby.

Either if your switches support some vendor specific Multi Switch Link Aggregation or use Port Based ID load balancing. The only thing you lose with Portbased is the possibility for a single VM to consume more than one pNIC at the same time.

(However that will only be under certain condition and the Load Balancing is not really balancing on the load, but in effect a random distribution.)

My VMware blog: www.rickardnobel.se
Reply
0 Kudos
donikatz
Enthusiast
Enthusiast

Either if your switches support some vendor specific Multi Switch Link Aggregation or use Port Based ID load balancing. The only thing you lose with Portbased is the possibility for a single VM to consume more than one pNIC at the same time.

Again, that's not completely accurate. "Route based on originating virtual port ID" does not provide load balancing, it provides fault tolerance. Each VM is assigned a NIC on boot and continues to use that NIC unless there is a failure. Because of this, your network traffic can be significantly imbalanced in many scenarios. IP Hash is more significant than just being able to aggregate more than one pNIC's worth of traffic at a time, it balances traffic so that one NIC isn't oversubscribed while another is undersubscribed.

I still maintain it's a risk analysis everyone can make for their own environment: Do you stick with an inferior teaming method 100% of the time because of a .01% risk? Or do you take the risk that you'll have to manually correct if the problem occurs, in order to enjoy better load balancing 99.99% of the time? That decision may have to do with your available resources and the needs of your environment.

Reply
0 Kudos
rickardnobel
Champion
Champion

>IP Hash is more significant than just being able to aggregate more than one pNIC's worth of traffic at a time, it balances traffic so that one NIC isn't oversubscribed while another is undersubscribed.

I would say that this is not the way it works. The balancing is in practise totally random and all traffic between two IP endpoints goes over one pNIC, until that fails, just like Port ID. It has no sense if one NIC is more loaded than another. If you have a single VM with lots of incoming sessions from different IPs that needs more than the bandwidth of one pNIC then IP hash is better, otherwise not. This is not often the case.

>Or do you take the risk that you'll have to manually correct if the problem occurs, in order to enjoy better load balancing 99.99% of the time?

I agree with you that this is everyones own decision, but I will not agree that IP hash will work better in 99.99% since it is not doing any superior load balancing and will help very little for most VMs.

My VMware blog: www.rickardnobel.se
Reply
0 Kudos
donikatz
Enthusiast
Enthusiast

I would say that this is not the way it works. The balancing is in practise totally random and all traffic between two IP endpoints goes over one pNIC, until that fails, just like Port ID. It has no sense if one NIC is more loaded than another.

It doesn't have logic to know which is more loaded and to actively balance, but because it distributes the load across multiple rather than sticking with one, it de facto provides this balance. It doesn't matter if it's random or not, it's distributed. I disagree that it has no use other than for a single VM with high load, it is also superior if you have several VMs with traffic flows with high peak periods (which is not uncommon), where you could easily have competing peak VM traffic on the same assigned NIC via virtual port ID, whereas with IP Hash that peak would be randomly distributed.

And my point was not that IP hash would work better in 99.99% of cases, it's that if you're in a situation where IP hash does work better for you, you would be able to take advantage of that within the 99.99% period of uptime vs that slight chance of failure period. But again, it's a subjective decision.

Reply
0 Kudos
rickardnobel
Champion
Champion

donikatz wrote: It doesn't have logic to know which is more loaded and to actively balance, but because it distributes the load across multiple rather than sticking with one, it de facto provides this balance.

I would not say that it does, not in the sense that it is in some fundamental way better than Port-ID. Both methods does in effect randomly distribute the overall network traffic across several NICs without caring or adjusting for actual load.

You could be unlucky with Port ID and have two VMs with high network IO on the same pNIC, but the very same is very likely with IP hash, not the least when you just use two active pNICs.

donikatz wrote: whereas with IP Hash that peak would be randomly distributed.

It is in practise randomly distributed, but in theory based on an algorithm which will always put the same IP endpoints on the same pNIC. So for any workload that could actually drive large amount of network IO for some time, perhaps a backup, it is still a risk for this traffic from several VMs to go through the same pNIC.

My VMware blog: www.rickardnobel.se
Reply
0 Kudos
donikatz
Enthusiast
Enthusiast

Peak load doesn't mean fixed endpoints. Server network loads can often be heavier during specific business hours, such as 8 AM when clients all connect, or perhaps when there is new content published & advertised on the server, or many other reasons I can think of that still involve multiple endpoints. The bottom line is there are definitely scenarios where IP hash can have enough advantage over virtual port ID where one might choose to take the slight risk in the implementation. If it doesn't in your environment, if there's no upside for you, than there's no point in risking any possible downside when possible, fair enough.

Reply
0 Kudos
reubensi
Contributor
Contributor

I just wanted to let anyone reading this know that in the end in order to get the best redundancy and performance we ended up upgrading to Enterprise plus licensing and enabling distributed switching.  The reason that we did this is because when using a distributed switch and setting each pNic to Route based on the originating virtual port ID the distributed switch will load balance the traffic automatically instead of binding a single VM to a single pNic/port.  This approach allows all of the traffic to flow over all of the links and doesn't care which network connection it is using.  Thus a switch or network port can go down and it's just rerouted.  The load balancing is done from the VMWare/host side instead of the switch side which in theory could be an issue but in practice and testing has proved to not be an issue at least in our environment.  This eliminates the issues described in the beginning of the post and allows for proper load balancing under all circumstances.  This is a better way to architect this in the end, but does require the high end and more expensive licensing.

Reply
0 Kudos
eliot
Enthusiast
Enthusiast

Just set this design up myself for my NFS storage network. Very similar to above, two independent switches (which i do have the option to stack, but want to keep them separate to facilitate switch maintenance)

VMKernel with 4 nics, 3 x 1Gig as IP Hash to a single Cisco 3750 and the 4th card as standby to a 2nd 3750 - with a port channel between the two 3750's

Observations:

  • In normal operational mode all traffic goes down the active etherchannel and nothing (as you would expect) goes down the standby card.
  • If you drop the port-channel all traffic fails over to the standby card and carries on working.
  • BUT as described above - if you drop a single card on the port-channel the standby card kicks in and you get mac-flaps reported on the switches - although my VM's using that networking via NFS carried on working.

I'm now in a rock and hard place, because IP-HASH is the only method that will even vaguely distribute connections (I'm avoiding the term load balance, because that isn't what any type of port channel does) - mac-hash and port based both pin all the traffic to a single card.

This is the point where i dropped one nic on the port-channel and traffic flows over the standby card giving mac-flaps:

Capture.JPG

(Cacti weathermaps are perfect for this sort of testing)

Reply
0 Kudos
rickardnobel
Champion
Champion

The setup is unsupported both from VMware and also dis-allowed in 802.3ad which says that link aggregation must be configured between only two devices, for example between a virtual switch and a single physical switch. (If the physical switch with some propriarity multi-switch-stacking appears as one switch it is ok.)

My opinion is still that this is an unsafe way to build the network.

Is this for your NFS traffic only? Do you have multiple IP addresses on the NFS server?

My VMware blog: www.rickardnobel.se
Reply
0 Kudos
eliot
Enthusiast
Enthusiast

Yes i do have multiple alias IP's on the storage and have multiple volumes mounted against those aliases in order to force the algorithm to use more than one card. My testing has shown that only IPHASH actually balances the traffic, mac and port result in a single card being used:

Mac-Hash and port based:

Capture1.jpg

IP-HASH: (also clearly showing i need 4 ports in the cross switch link)

Capture2.JPG

Reply
0 Kudos
rickardnobel
Champion
Champion

With multiple IP addresses (on different IP subnets) on several VMKs with some active/passive setup together with Port ID you should be able to get the hosts to use several interfaces at the same time, if the NFS server has storage presented on different IP address from different subnets.

My VMware blog: www.rickardnobel.se
Reply
0 Kudos