VMware Cloud Community
tbone11
Contributor
Contributor

Various VMware Questions

I have several question, let me give you a little background first:

Resently we had a power outage in our datacenter at work. Everything in the datacenter went down hard. This includes all VM's, all ESX hosts and the Virtual Center server. When we got everything back up, the virtual center database was corrupt. After spending several hours on the phone with VMware, we were still unable to recover it. For reasons that don't matter to this thread and that I'll leave left unexplained we had to start from scratch with the database.

While we were working on that, other admins were powering up ESX hosts. HA was enabled on these hosts so VM's just start powering up when the hosts came up. The problem though was that several VM's were powered up and were on one ESX host, but were listed as powered off on another ESX host. I'm guessing this happened because all primary nodes in each cluster went down and without the virtual center database, hosts didn't know which one had which VM? That's my first question.

My second is hypothetical, what would have happened if HA was not enabled on those clusters? When the hosts powered on, would they have even seen the VM's anymore, or would we have had to re-register all of them and then power them on? Similarly, in a more normal situation, if HA is not enabled on a cluster and one host fails, what happens to the VMs? Obviously they will shut off, but how would you go about powering them back on? Would you just have to find them in each datastore and re-register them and power them on? I ask these questions because some in my organization are now questioning the efficacy of VMware HA and I want to make sure I fully understand the concequences of having HA enabled/disabled before I argue one way or another.

My last question is this - given the datacenter fiasco mentioned above, we're now installing a better generator to prevent this from happening again. Two weeks from we're going to have to gracefully shut down everything in the datacenter though to install it. What is the best way to do this with VMware without resulting in another split-brain scenario? Our virtual center database resides on a VM and our virtual center server is physical. I figure we should shut down each VM first, then the hosts, then the VC server. But how should we go about bringing it all up so all the hosts know exactly what to power on? Should I isolate the VM that hosts the VC database to one host, bring that host and VM online and then the VC server and then the rest of the hosts and VMs?

0 Kudos
6 Replies
weinstein5
Immortal
Immortal

Welcome to the Forums - I will try my best to answer your questoins -

While we were working on that, other admins were powering up ESX hosts. HA was enabled on these hosts so VM's just start powering up when the hosts came up. The problem though was that several VM's were powered up and were on one ESX host, but were listed as powered off on another ESX host. I'm guessing this happened because all primary nodes in each cluster went down and without the virtual center database, hosts didn't know which one had which VM? That's my first question.


I am not sure what is happening here - but I think what is happening your ESX servers are configured to autostart your VMs and it is this that is starting your vMs - with VC down you can safely remove the VM from the ESX server where they showed power off -




My second is hypothetical, what would have happened if HA was not enabled on those clusters? When the hosts powered on, would they have even seen the VM's anymore, or would we have had to re-register all of them and then power them on? Similarly, in a more normal situation, if HA is not enabled on a cluster and one host fails, what happens to the VMs? Obviously they will shut off, but how would you go about powering them back on? Would you just have to find them in each datastore and re-register them and power them on? I ask these questions because some in my organization are now questioning the efficacy of VMware HA and I want to make sure I fully understand the concequences of having HA enabled/disabled before I argue one way or another.


Depending on how your hosts are ocnfigured - you can have VMs autostart when the ESX hosts start -




My last question is this - given the datacenter fiasco mentioned above, we're now installing a better generator to prevent this from happening again. Two weeks from we're going to have to gracefully shut down everything in the datacenter though to install it. What is the best way to do this with VMware without resulting in another split-brain scenario? Our virtual center database resides on a VM and our virtual center server is physical. I figure we should shut down each VM first, then the hosts, then the VC server. But how should we go about bringing it all up so all the hosts know exactly what to power on? Should I isolate the VM that hosts the VC database to one host, bring that host and VM online and then the VC server and then the rest of the hosts and VMs?

</div>

I would first disable HA - then shutdown the VMs except the one hosting the database - shut down VC and then shut down your ESX servers -

Theone recommendation Iw ould make is virtualize your VC server and make your SQL database physical - you will still be able to have HA working even if the ESX server hosting your VC VM fails - the database is the critical component - all informaiton about your VC environemnt is tored there -

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful
0 Kudos
tbone11
Contributor
Contributor


I am not sure what is happening here - but I think what is happening your ESX servers are configured to autostart your VMs and it is this that is starting your vMs - with VC down you can safely remove the VM from the ESX server where they showed power off - {/quote}



Where would they be configured to autostart the VMs? Isn't that just if HA is enabled? And why would that have anything to do with VMs being powered on on one host and powered off on another? And what would have happened had HA not been started? Would we have even seen the VMs or would we have had to re-register them all?



I would first disable HA - then shutdown the VMs except the one hosting the database - shut down VC and then shut down your ESX servers -

Why would you disable HA? And why shut down the hosts last? It seems to me that VC should be shut down last.

And what order would you start things? It seems to me that we should start the host with the VC database on it first, then VC and then the rest of the hosts and VMs. What do you think?

0 Kudos
tbone11
Contributor
Contributor

...I realize I asked a ton of questions, but I would really appreciate some more help with at least how to best handle the upcoming planned outage so we can ensure there will be no issues this time. Thanks!

0 Kudos
weinstein5
Immortal
Immortal


Where would they be configured to autostart the VMs? Isn't that just if HA is enabled? And why would that have anything to do with VMs being powered on on one host and powered off on another? And what would have happened had HA not been started? Would we have even seen the VMs or would we have had to re-register them all?



I am not sure what happened here - you set the auto start on the ESX hosts - I have not seen this occur before -



Why would you disable HA? And why shut down the hosts last? It seems to me that VC should be shut down last.
And what order would you start things? It seems to me that we should start the host with the VC database on it first, then VC and then the rest of the hosts and VMs. What do you think?

Actually better plan would be to place your hosts into Maintenance Mode - yes VC should be shut down last but if you shit down the host that is running the database VC will crash - so hut VC down before you shut the host down running the VC databaae - your start up sequence sounds good -

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful
0 Kudos
RParker
Immortal
Immortal

but if you shit down the host

</div>

The throat or just the neck? And would this be before going up the peverbial rope... Smiley Happy

0 Kudos
weinstein5
Immortal
Immortal

hey - its early ok -

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful
0 Kudos