We have a 3 ESX host system connected to a redundant path HP SAN.
We ugpraded from ESX 4.0 U2 to 4.1. We normally update vmware tools within 48 hours of an ESX upgrade however before we had the chance we had a complete storage outage due to ESX making a reservation on a lun and then never removing it during an exchange backup that uses windows shadow copy. This caused all of our VMs that were on that LUN to stop functioning. Ultimately we had to take everything offline and restart everything.
It happened 2 nights in a row at exactly the same time down to the minute. It coincided with our TCP/IP exchange backup(not a vm snapshot) using backup exec Veritas 12.5 SP4 and a backup exec remote agent. This setup has been working in our environment for 2.5 years.
Both nights I worked with vmware enterprise support and they were unable to determine the problem(they did not look very hard and brushed me off to HP support regarding our SAN equipment). HP confirmed no issues on our SAN or related equipment.
After the 2nd outage in row we immediately updated vmware tools and have not had the problem reoccur.
Vmware support wants me to try and reproduce the issue now that they are thinking the issue could be related to vmware tools being out of date. I would not allow that to happen again on my production environment so I don't think the support case will go further.
This is just a warning to everyone that you should update your VMWARE tools ASAP after upgrading ESX to 4.1.
This is not behavior I would normally expect to occur from an out of date vmware tools so BE CAUTIOUS!
