Consolidated Backup and Guest DB's hanging

[jbsengineer]

We have had issues with 3 servers that run an Adaptive Server Anywhere Sybase Database and running a VCB backup nightly. It's sporadic issue however we have found situations where we can increase the likelihood. The problem is that during the VCB backup snapshot creation while the disks are starting to be copied off the VMFS partition the Sybase DB comes to a halt. Its error is “Fatal error: cannot write to transaction log.” I have also seen a “Fatal error: unknown device”. 1 server is running a different version of ASA while the remaining 2 run and identical version. We do have other ASA DB’s in our VM environment as well as MySQL, and MSSQL. But these 3 servers are by far the most active. They are Point of Sale systems. We live on a large University so transactions are constant throughout the night as we have online vending, laundry, dining, snack, etc transactions.

Right now it seems to occur once a week. But we can make it occur almost nightly by aligning the VCB backup with each server local DB backup, or running a network backup such as Retrospect at the same time. We have found this over the past year. So now we have our backups tuned so nothing interferes with VCB. However we still randomly see the failure. I can only guess ASA is very sensitive and when it tries to write a transaction and if it can’t it’s going down. It seems this occurs exactly when the snapshot on the server is created or if the traffic on the disk is high. Performance does not seem to be an issue.

Apart from trying to find the perfect time when the server is close to idol what can I look at? Has anyone else experienced this with VCB?

Environment:

CX300 arrays

ESX 3.5 and VC 2.5

VCB 1.1

6 ESX Servers

Thanks!

[Pmarsha1]

We have had problems with snapshots and it came down to two things:

1) VMDK files with the same name, i.e a boot disk and paging disk having the same name though on different Datastores

2) HP Management agents need to be stopped whilst carrying out VCB operations

VMware also told us that snapshots can fail if activity is high and you are using to option to quiesce the operating system.

They told us to uninstall the sync driver in VMware Tools so as not to quiesce the OS.

In you case I think you may have to shut down the DB using a pre freeze script and restart it using a post freeze script

HTH

Paul

[depping]

most databases don't like quiescing. disabling the driver or disabling quiescing completely would solve this.

Duncan

My virtualisation blog:

[jbsengineer]

How can I disable quiescing to test this out? Also what will be the negatives of doing so?

Thanks!

[Pmarsha1 · Accepted solution]

Method 1: Online, Crash Consistent SnapshotDisable the VCB Sync Driver (LGTO_Sync). In Device Manager:

- Select View | Show hidden devices- Expand Non-Plug and Play Drivers- Right click on Sync Driver and select Disable from the context menu- Click Yes to disable the device- Click Yes to restart the computer

Alternatively don't install the Sync driver when installing VMware Tools.

Using the command line:

msiexec -i "VMware Tools.msi" ADDLOCAL=ALL REMOVE="Sync" /qn reboot=supress

Using a transform:

- Install Orca ()- Right click on the VMware Tools.msi and select Open with Orca- Select Transform | New Transform- Click on the Feature table in the left pane- On the Sync feature set the level from 1 to 0- Select Transform | Generate Transform- Save the transform as "VMware Tools.mst"

Install VMware Tools with the following command:

msiexec /i "VMware Tools.msi" /qb TRANSFORMS="VMware Tools.mst"

The Sync driver makes the backup consistent by writing all memory to disk and all new writes get ewritten to a new VMDK file. Thus the full machine state is captured.

Not using the Sync driver means the backup is similar to powering off a machine. i.e. It crashed. 99 times out of 100 it will recover with no intervention.

The alternative is to use a pre and post freeze script like this one for Exchange:

C:\Windows\pre-freeze-script.bat@echo offNet Stop MsExchangeSA /Y

C:\Windows\post-thaw-script.bat@echo offNet Start MsExchangeSANet Start MsExchangeISNet Start MsExchangeMTA

[jbsengineer]

I have disabled the Sync Driver on 1 of the 3 servers and it looks like this has corrected the issue. On the next maintenance period I will disable it on the remaining 2. The Sync Driver requires a reboot after it's disabled. Thanks for the help!!