Cluster HA and Virtual Machine Startup/Shutdown

[hstorn]

Hello,

In the documentation, it is written :

"NOTE The Virtual Machine Startup and Shutdown (automatic startup) feature is disabled for all virtual

machines residing on hosts that are in (or moved into) a VMware HA cluster. VMware recommends that you

do not manually re-enable this setting for any of the virtual machines. Doing so could interfere with the actions

of cluster features such as VMware HA or Fault Tolerance."

When all ESX servers, how to set a startup order for virtual machines ? It's impossible ?

Thanks,

Regards.

[marcelo_soares]

The problem is that the option will never be used if the host is on HA cluster. IF you have VMs that are not affected by HA (i.e. local datastore ones) you can manually enable this option and enable for this specific VMs. All other VMs residing on shared datastores will start on other hosts in the failure moment, and you can set start priorities on the HA cluster level.

Marcelo Soares

VMWare Certified Professional 310/410

Virtualization Tech Master

Globant Argentina

Consider awarding points for "helpful" and/or "correct" answers.

[hstorn]

I think the startup order for virtual machines is essential even in a cluster HA.

In my IT team, we are only two people responsible for the virtual infrastructure.

After a power failure, the others must be able to start physical servers and virtual machines, without specific knowledge.

So, it's impossible in a cluster HA.

[marcelo_soares]

Didn't understand your concern. If you have a host power failure, your VMs will startup automatically. You can change the order on HA settings. Why you are thinking they won't and you will need intervention?

Even if you reboot a server the behavior intentionally will be the same - obviously you will be carefully enough to not reboot a server with VMs running o it, will vmotion them or cold migrate before the planned maintenance...

Marcelo Soares

VMWare Certified Professional 310/410

Virtualization Tech Master

Globant Argentina

Consider awarding points for "helpful" and/or "correct" answers.

[hstorn]

During a long power failure in the in the servers room, all ESX servers can be turned off.

It should restart all physical servers and all virtual machines.

Certain services and virtual machines must be started before other (DNS, LDAP directory, databases, ...).

If all ESX servers are in a cluster HA, I can not set the boot order of virtual machines. Virtual machines will be started immediately without startup delay.

This is my problematic.

Thanks.

[golddiggie]

I don't have a cluster to confirm with at the moment, but I believe you can set a boot priority within the environment. Such as attach highest priority to the DC's, email, or servers that need to start first, then lower priorities to the remaning servers, with the lowest to the last ones you want to start up again. I would assign one of the higher priorities to the vCenter server so that you can log in and start powering on any remaining servers too.

I would also make sure that email alerting is enabled, and working properly, so that you know the instant something happens. Configure the alerts so that you know when you're running on battery backup power too (I'm assuming you have decent UPS units in place). The physical server startup (such as the host) is often a bios setting... Something akin to behavior after a power failure. I typically set that (in desktops and servers alike) to 'last state' this ensures that if a server was powered on before power was removed/lost, it will turn itsefl back on as soon as power is restored. If you have UPS units that have direct interface capabilities to servers, use those too. Even in my home lab, I have TrippLite units that have the USB ports on them, which are connected to the ESXi host as well as one of my other systems (the one I use every day). This will (or should) be detected so that you can make additional option settings for the host.

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

[hstorn]

In a cluster HA, there are only 4 restart priority (high, medium, low, disabled).

With lots of virtual machines, it must also take into account the startup time of virtual machines.

With a cluster HA, it is not possible to define a order and a specific startup delay between each virtual machines. So, I am not possible to restart automatically ESX servers and virtual machines, simply starting the physical servers.

[depping]

No unfortunately you can't set an order in which it should restart. you can use the 4 different options you would have but that still wound end up being less than you probably would like to have.

Duncan

VMware Communities User Moderator | VCP | VCDX

-

---

Now available: <a href="http://www.amazon.com/gp/product/1439263450?ie=UTF8&tag=yellowbricks-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=1439263450">Paper - vSphere 4.0 Quick Start Guide (via amazon.com)</a> | <a href="http://www.lulu.com/product/download/vsphere-40-quick-start-guide/6169778">PDF (via lulu.com)</a>

Blogging: http://www.yellow-bricks.com | Twitter: http://www.twitter.com/DuncanYB

[domboy]

I just want to confirm regarding this statement:

"NOTE The Virtual Machine Startup and Shutdown (automatic startup) feature is disabled for all virtual

machines residing on hosts that are in (or moved into) a VMware HA cluster. VMware recommends that you

do not manually re-enable this setting for any of the virtual machines. Doing so could interfere with the actions

of cluster features such as VMware HA or Fault Tolerance."

This means that each host server in a cluster with HA enabled should have the check box "Allow virtual machines to start and stop automatically with the system" unchecked on the Virtual Machine Startup and Shutdown properties page, correct? With VMware HA enabled on the cluster the virtual machines will still start automatically with the servers on boot up, correct? What I'm trying to make sure of is when we have a system-wide failure, say during a power outage, causeing all hosts and vcenter go down, when power resumes and the hosts are powered back on, HA will power on the virtual machines in a similar fasion to what the aforementioned check box setting would do for an individual host not in a cluster. Do I understand this correctly? Thanks!

[ssingla]

I am having the same concern. Did you get any answer to this?

[domboy]

ssingla wrote:

I am having the same concern. Did you get any answer to this?

No I did not. However, I recently had to shutdown our virtual datacenter due to required SAN maintenance, so I similated a power outage. The VMs that were powered on before the shutdown came back up. However, I noticed that the VMs one of our MS Admins manually shutdown before the downtime did not boot back up, even though the cluster VM restart priority setting was enabled (medium) for those. Interesting. Anyway, I think I answered my question.

[Gleed]

To the point on why you don't want to austostart VMs in a cluster: 

When you have a standalone host, it makes sense to enable autostart because since there is no cluster, the VMs are only registered with the one host, meaning they can only run on one host, so there's no concern that another host may already be running/accessing the VMs when the (re)boot occurs.

However, once you put the hosts in a cluster there are now multiple ESX hosts capable of running the VMs.  As such it no longer makes sense to auto start VMs when a host (re)boots because when the host went down the running VMs would have either been (1) evacuated or (2) failed over.  Meaning they would already be running somewhere else when the host (re)booted so no need to try and autostart.

Once a VM is running, HA will do everything it can to keep the VM running.  The HA primaries keep track of what VMs are running on which hosts, if a hosts goes fails, then HA will attempt to restart all the affected VMs that were running.

Note that if a VM is manually shotdown/powered off/not running, then HA assumes there is a reason for this and in this case will not automatically start the VM when the host boots.  If the admin shuts the VM down, then HA has to assume the admin wants the VM to stay down until it gets manually restarted.  Once it has been (re)started, HA will do everything it can to make sure it stays running.

HTH,

-Kyle

[depping]

I wrote a short article on this topic a while back: http://www.yellow-bricks.com/2010/10/22/did-you-know-all-hosts-failed/

Duncan (VCDX)

Available now on Amazon: vSphere 4.1 HA and DRS technical deepdive

[denisbaturin]

Duncan,

your article correct for failed cluster. As you wrote HA "restarts based on the last known state". What if all VMs were shuted down gracefully by powerchute or UPS shutdown script... VMs won`t start!!!

[depping]

No of course they won't as HA assumes you powered down all VMs for a good reason, it cannot assume anything else. If you want to power them up automatically you will need to write a script or just click through all the VMs in the proper order.

Thanks,

Duncan (VCDX)

Available now on Amazon: vSphere 4.1 HA and DRS technical deepdive

[denisbaturin]

vSphere already has builtin script(/sbin/vmware-autostart.sh) to start virtual machines and cool interfaces to manage it. 

But VMware recomend not to use autostart feature. :smileyplain:

[domboy]

Denis Baturin wrote:

Duncan,

your article correct for failed cluster. As you wrote HA "restarts based on the last known state". What if all VMs were shuted down gracefully by powerchute or UPS shutdown script... VMs won`t start!!!

My solution is to install the UPS shutdown software on the host servers and not on the VMs. Incidentally, this is why I'm not in any hurry to transition from ESX to ESXi... the service console is very handy in that it will run most linux programs, such as our UPS shutdown software. Anyway, we haven't had a power outage to test it yet, but based on the HA behavior described in the article and my own simulated outages it should work.

[depping]

On the host or VMs, it is all the same. When the Host powers off but all the VMs are down, HA will never power on the VMs as the last known state more than likely is the situation where all VMs were powered down.

Duncan (VCDX)

Available now on Amazon: vSphere 4.1 HA and DRS technical deepdive

[domboy]

Duncan wrote:

On the host or VMs, it is all the same. When the Host powers off but all the VMs are down, HA will never power on the VMs as the last known state more than likely is the situation where all VMs were powered down.

Duncan (VCDX)

Available now on Amazon: vSphere 4.1 HA and DRS technical deepdive

Maybe I wasn't clear. By installing the UPS shutdown software on the hosts and not on the VMs, only the host servers will be instructed to shutdown in the event of a power failure. The VMs will all still be running when the hosts begin to shutdown. I have attempted to simulated this with manually shuting down all hosts of a cluster simultaniously from the console while the VMs are running. When the hosts were brought back up all VMs that were running at the time I initiated the shutdown were started back up. I should think the same would apply when the UPS software is initiating the shutdown.

[FredericNass]

Hi domboy,

I'm hoping you'll read that post even if it's a year and a half old...

I tried what you did on a test host and it works fine the way you said. But this way, it seems to me that the host crashes the VMs when it powers down.

What about this ? How would you be sure all VMs would restart correctly at next boot ?

I opened a thread here : http://communities.vmware.com/thread/299757?tstart=0.

I think the best practice in that case of power failure would better be to :

1. Record each VM state on a host when the power cuts off in a log file,

2. Gently shutdown each running VM on the host,

3. Shutdown the host,

4. Restart each VM with each hosts when the power returns, using the log file from step 1.

I don't know why VMware does not provide any mechanism / tool to automate this... A simple script "emergency_shutdown" would do that fine with asking the host to restart running VMs on next boot.

Any relevant trick from anyone's experience ?

Mini.