Possible to update vCenter without taking things down?

[emcclure]

Hello,

So I'm trying to see if there's a way to solve a particular problem.  To start we have 3 different vCenters, all are 6.7 U2, all have multiple clusters with multiple hosts doing different things.  2 of the vCenters are for production, 1 for staging.  Patching ESXi hosts is no problem, put a host in maint mode, watch the VM's migrate, update, reboot, nobody is the wiser.  However when updating vCenter it requires downtime.  Is there a way to update vCenter without causing downtime to prod or staging?  Is there a certain kind of setup needed with vCenter (I believe they're all stand-alone vCenters, no PSC linking them together)?  Is it just having some sort of API's that would somehow tell prod for example to open new requests on a certain vCenter and once we've verified there's nothing happening on the target vCenter we update it, then somehow have things moved back to the just updated vCenter, or is that going to be slower process where we're just updating one vCenter at a time?

[daphnissov]

No, there's no way to do that currently. vCenter is a single management application that does not have a clustered ability (vCHA is not this) so you will require some downtime for vCenter for a patch or upgrade. But it sounds like you have different stuff going on against these three vCenters, like a cloud management platform or automation that speaks to one. If that's the case, and presuming prod or staging are equal targets for a certain type of deployment, you could always write in a health test to ensure the candidate vCenter is online prior to making a deployment. Lots of options here but more details would be required.

[emcclure]

Ok I'll try to give as much info as I can without revealing certain company things.

A customer (internal) goes to a website (custom website built by us) to request an environment.  They can choose what VM's they need, what internal products they need, etc.  They submit it, it does its thing thru various API's or whatever.  One of the production vCenters gets the request and builds the pod from images that they have.  They create a new folder, clone VM's, rename, join the domain, install our products, etc.  When a environment is complete it is destroyed unless the customer asks for more time.

I'm not familiar with vCHA, but I can look at it.  Is it something that I can add to the current setup, or would we have to rebuild from scratch?  If we had to rebuild is there an easy way to transfer from what we have to vCHA or is that a huge process?

[daphnissov]

Jeez, so you guys built your own true cloud management platform from scratch. Must be a hell of a chore to maintain and extend!

If you have multiple, different production vCenters, you could eval their health by using the appliance management API presuming you're on 6.5 or later. When your platform gets the request and determines the destination is prod, you build a list of candidate endpoints. You then insert that liveness probe after one is chosen. If that probe returns that it's down, you loop back and choose another vCenter. As long as there is always one "alive" vCenter, your user requests succeed.

[emcclure]

Yep that's what they did.  I don't think they knew about vRA at the time.  I was starting to work on vRA, but I had another project that just ate up my time and I could never get into it.  By the time I showed them vRA and some of what it could do they just wanted to stick with what they have.

We're using vCenter 6.7 U2.

I'm betting they can figure out the API's for that.  The only issue could be is if some of the environments are in the vCenter that's going down for maintenance.  That's the part I'd like to avoid.  I'm sure they can use an API to point everyone who builds an environment to a particular vCenter, but if you have too many pods and not enough resources then people can't build environments, etc and my goal is to be able to do these updates without any of the customers knowing we're ever doing this, to minimize downtime as much as possible.

[daphnissov]

The only issue could be is if some of the environments are in the vCenter that's going down for maintenance.  That's the part I'd like to avoid.

Well, if those "environments" (however that's described; you didn't say) are located in one and only one vCenter, and you upgrade that vCenter, then there will be temporary unavailability of that vCenter. There's nothing you can do about that, vCHA or otherwise. You can only minimize the impact by scheduling the upgrade during non-peak business hours, taking the necessary precautions, and having a solid roll-back plan.

my goal is to be able to do these updates without any of the customers knowing we're ever doing this, to minimize downtime as much as possible.

And minimizing it may be the best you can do in this situation, but some downtime may be unavoidable.

Since you're dealing with a home-grown, roll-your-own CMP, it really comes down to how many eligible endpoints (vCenters) you have and the environments within them. From the custom perspective, they shouldn't care about where their infrastructure is being built. They just need those resources.