Are these windows 2003 machine... also is VMware tools upto date.
They are mostly 2008 and mostly are out of date. Some are 2003 that have current tools that are failing.
I don't think that is the issue though since I was able fix a couple of these by changing the time out for the machine directly.
It appears the settings I've changed at the site level are not working as I still get these errors "Error - Timed out waiting for VMware Tools after 300 seconds".
The problem is that some machines desired final state is Powered off so there's no way to set this at the machine level.
I think you upgrade the VMware tools on production VM and do test failover with increased timeout for Vmware tools set.
I really don't think that will fix the issue. All the vms I set the timeout on manually now work without a tools update.
The problem is the advanced recovery settings don't seem to being applied.
I believe I'm going to have to open a call with VMware.
You may want to check the isolation.tools.setinfo.disable value on the VMs. If set to true (once was a CIS recommended setting), it will prevent SRM from talking to the VMTools. This setting caused me issues when SRM would try to reconfigure the VM's network on the recovery side. Set the value to false.
This option isn't set to true. In fact it doesn't even exist on these machines.
The workaround VMware support came up with is to Configure the virtual machines and change the Startup Action to Power on, change the Timeout to wait for VMware tools and save. Then go back into the configuration and change the the Startup Action to Do Not Power On.
This appears to work. I never got an explanation as to why setting the recovery.power.OnTimeout under recovery in Advanced Settings of the site didn't seem to work.
Just ran a quick test in one environment and it appears as though the timeout value assigned under the site Advanced properties (recovery.powerOnTimeout) is only used when you initially configure the protection of the VM. If the VM has already been configured, you will need to manually assign the timeout to the particular VM in the Recovery Plan.
A quick method to change the timeouts to many VM's would be to remove protection and then reconfigure protection again with the advanced setting changed for the remote site (a restart of the services is required). This, of course, will make you lose specific VM configurations in the recovery plan and you may need to manually set the timeout for each VM if you have many customizations.