Does DRS Check Affinity Rules Immediately After HA Event?

[StephenMoll]

The system I working on has some strict requirements resulting in a set of affinity rules.

We have VM-VM affinity rules to keep groups of VMs in pre-defined Blade workloads.

One VM from each Blade work load is designated as an anchor for the set, and the anchors from all sets are members of a VM-VM anti-affinity rule.

Some workloads can only run in certain sections of the cluster, so the VMs in those sets are in VM-Host Affinity rules.

Now when a blade fails, HA kicks in to restart the lost VMs... great.

HA only obeys the VM-Host affinity rules and VM-VM anti affinity rules. So as HA restarts the VMs, the only one in the set that be guaranteed to startup in the right place is the anchor from the set. The other VMs from the set, well some of them will start in the right place, because the new blade is relatively lightly loaded. After a couple of VMs have been initiated however, presumably HA now sees that blade as having a higher load than some of the other blades which already have sets running, but are the VMs are idle. So HA will start some VMs on blades other than the one with the anchor.

We see DRS kick in and sweep up the misplaced ones as it resolves the rules that HA ignored, so the end result is correct.

Now DRS normally only checks the rules every 5 minutes, is that correct?

After a few tests, I haven't seen the gap between HA events and DRS tidying things up be more than 20 seconds or so. I can't find anything to confirm this, but it seems to me that DRS might be triggered to check rules out of its normal schedule following an intervention by HA. Is this true, or have I just been lucky to always run a test shortly before DRS ran its normal cycle anyway?

[daphnissov]

Now DRS normally only checks the rules every 5 minutes, is that correct?

It checks for cluster imbalances every five minutes, yes.

After a few tests, I haven't seen the gap between HA events and DRS tidying things up be more than 20 seconds or so. I can't find anything to confirm this, but it seems to me that DRS might be triggered to check rules out of its normal schedule following an intervention by HA. Is this true, or have I just been lucky to always run a test shortly before DRS ran its normal cycle anyway?

I also don't have confirmation, but my observation has been similar in that DRS appears to evaluate the affinity rules after an HA event and the 5 minute timer sits therefore outside of its realm.

Also, there is an advanced setting possible as of vSphere 6.0 which allows HA to respect VM-VM affinity rules through the option das.respectVmVmAntiAffinityRules. Setting this to "true" should suit your needs HOWEVER be aware that if you have antiaffinity rules configured that this option could result in HA *not* restarting VMs if a valid host cannot be located.

In contrast to this, there is also the advanced option called das.respectVmHostSoftAffinityRules. Set to "false" by default, this is a softly-evaluated rule which may result in violation if HA cannot find a suitable host, i.e., it will restart a VM on an invalid host rather than not restarting it at all.

These tidbits and more can be found in depping​ 's great (and free!) resource VMware vSphere 6.0 High Availability Deepdive book. These advanced settings are specifically referenced on p53 of the PDF version.

[StephenMoll]

We have:

    "das.respectVmVmAntiAffinityRules set to "True"

    "das.respectVmHostSoftAffinityRules" set to "True"

VMs are not permitted to start on a "wrong" host, i.e. one that already has a VM set and an anchor. This is a requirement we need to meet. I am evaluating options to try and prevent this at the moment, including:

• Using the anchor VM to pad the resource allocation on the blades so HA has no option to start additional VMs there. So in theory HA would only be able to start the a VM set on what were empty blades. If there were more than one spare (likely) then DRS would following up and move the set together.
• Using our cluster management application to kill VMs that start without their anchor being present. Don't fancy this at all.

We did consider have vm-vm anti affinity for all VMs but the number of rules that would require gave us concerns it may harm the response time of HA. We have over 300 VMs with on average 8 VMs per set. Each one would require a rule that stops it co-residing with each VM that isn't in the same set, so in rough number 300 x 300 rules, so of the order of 100,000 rules!!! So we didn't even attempt that one! We reckon there is likely to be a practical limit to the number of rules that can be processed, although I can't find any evidence of what this number is likely to be.

The way the system is working at the moment is largely satisfactory. It will get us through our first delivery drop. The expected behaviour is being written into user guides. This query was largely about making sure I have done as much as I can to reduce the time the system is in a non-compliant state. We were thinking of triggering a DRS rule check at a higher rate, or immediately a configuration change occurs (like an HA event), but at the moment it looks as if we might not need to.

My next consideration will be whether or not I have a feature implemented in our software to turn off HA on VM sets that would have no where to go if a failure occurs, but this puts us in the realms of actively interfering in something that operates autonomously (HA) through something which itself runs on a protected VM.

p.s. Congratulations on becoming a "Community Warrior"!  😉

[daphnissov]

What you may need to do if this behavior still isn't satisfactory is to have different compute clusters which serve as a boundary for your VM sets. This would also reduce the complexity of DRS rules you have to manage. If you wanted to move VMs around those clusters, you could use vROps as an overlay with a virtual datacenter construct which creates a logical environment spanning those clusters. This obviously necessitates a vMotion network with connectivity across clusters, but it's one possible avenue for you.

And thanks for the congratulations

[StephenMoll]

Not sure I fully understand this. I'll have to study at another time.

In any event, the cluster design is set, and I think the powers-that-be would be reluctant to add another server or service into the mix.

What we have is working satisfactorily. I just wanted to find out if we can (or need) to do anything to prevent too much of a delay between HA finishing a task and DRS checking the affinity rules are being met.

It would seem at the moment that we can't be sure that DRS is being triggered automatically when a HA task has completed.

[StephenMoll]

We still don't see any appreciable delays between HA finishing its thing and DRS tidying up any mess.

I am convinced that something in vSphere is causing DRS rules reassessment to be triggered immediately following an HA event, but would appreciate knowing if there is anything that documents this as fact.

[StephenMoll]

We has a visit from our TAS a couple of weeks ago, along with a couple of other engineers.

I posed the question to them, and although there hasn't been confirmation, the feeling (assumption) at the moment is that DRS does reassess rules soon after HA has completed its actions.

On the matter of "das.respectVmVmAntiAffinityRules" and "das.respectVmHostSoftAffinityRules" these are now set to "true" by default, and more disconcertingly HA can break these rules still anyway, and probably always could. Information here VMware Technical Partner Roundtable: Where’s the HA enforce VM-Host and Affinity rules option in vSp... This came up in my other discussion Where is "das.respectVmHostSoftAffinityRules" documented?

[StephenMoll]

Well here I am at VMWorld 2018 in Las Vegas and as I passed the "Meet The Experts" desk they seemed a bit quite, so as I had the attention of three of them, I asked them.

Yes, DRS rules check is run after an HA event. It might be that this detail is not in the documentation, or if it is, it isn't clear and obvious. So I was advised to raise an SR which may lead to clarification or a KB article.