vRA 7.4 upgrade and Load Balancer

[michael_stefani]

In the VMware documentation an upgrade from 7.x to 7.4 has a step that literally just says "disable load balancer" with no additional info or links.  There's also some documentation in the 7.4 release notes that mentions no need to disable health checks for upgrades in a distributed  environment.  Very confused as to whether I need to do anything with the load balancers for this upgrade or just leave them and if so what exactly does "disable" them mean?  Remove health check, take out secondary nodes, etc.  Thanks for any info

[daphnissov · Accepted solution]

This is where the documentation needs work. It's something I've reported already. You need to disable health checks on your load balancer for sure but not disable the load balancer entirely (how would they seriously think this could work??).

[michael_stefani]

Thanks, I saw that in some of hte older documentation so that's what I was leaning towards.  Was just hoping for some confirmation.  So OK to leave both nodes in each of the LB pools (web, appliance, manager) but just disable the health checks so they don't fail over at all during hte upgrade?

[daphnissov]

I have not done a fully-distributed and redundant upgrade to 7.4, but this should be fine to leave both nodes in the pool. As always prior to these scary vRA upgrades, snapshot everything. And if you want to be really really safe, snapshot them after a full stack shutdown (at powered off state). Take a standalone full backup of your IaaS SQL database as well.

[michael_stefani]

Thanks, yeah I had some catastrophic ones in 6.x.  Have only done one upgrade in 7.x and it went surprisingly smooth.  Crossing my fingers 7.4 goes that well.

[daphnissov]

Make sure you read the release notes and documentation very, very diligently before proceeding, and check out the known issues discussion here in the channel.

[prestonville]

As others have stated I would always do a full snapshot with powered down VM's and a quiesed SQL backup. I have found our distributed platform upgrades difficult and I also run it on our test platform first which is a copy of the distributed production. I have upgraded 7.01 to 7.1. to 7.2 to 7.3 and on my test platform it has never worked first go. Upgrades are getting easier as more appliance based but even with the test platform I missed in the last upgrade testing from 7.2 to 7.3 that the variable handling has change in vRO 7.3 which has "broken" the Netapp catalog items for WFA. We needed to recode some stuff in vRO. Thankfully this wasn't a major issue.

[michael_stefani]

FYI - we ran into multiple issues with our upgrade.  The first issue I saw documented on this forum where the management agents "aren't responding".  Had to manually remove / reinstall them on all the IAAS boxes and then the upgrade actually started for us.

Second issue we ran into was secondary appliance couldn't be upgraded.  Support sent me a KB saying to roll the whole thing back and make some changes to a certificate file, but 6 hours into a problematic install I'd rather not roll the damn thing back and start over.  Right now we're running with a single appliance waiting for word from support if there's any alternatives. 

Third issue was the IAAS upgrade failed halfway through with a timeout error.  This one was pretty easy, just had to manually finish the IAAS agent upgrades on the hosts that were still running 7.3 agents.

Not a fun install to say the least and I'm concerned support's not going to have a good solution for the failed replica node and tell me to roll the whole thing back.

[daphnissov]

I was afraid of this, unfortunately. These upgrades are still notoriously wonky and are still highly error prone. I hope at least with snapshots you took that you have a good revert point if need be.

[michael_stefani]

Just an FYI for anyone reading this thread, we opted to just kill the replica appliance and deploy a new one.  It was definitely a faster solution than following the KB which wanted to roll back the entire upgrade and then fix some cert file.  Ripped out the old replica appliance, deployed a new one, and then sync'd them up.  Worked pretty good, only had a few things to clear up after it was joined to the cluster.