Did you have any luck with this? We're in a very
similar position, using RDM luns on NetApp. I want to
use NetApp snapshots, but need a way to quiesce the
VM first.
All I want to do is:
quiesce VM IO
take NetApp snapshot
resume normal operation
After speaking to the VCB design team lead at VMWorld, I am now confidant that I have this problem licked.
Using NetApp's article #3393 which was written for ESX 2.5 as a guide, I have updated the process to work with ESX 3.0. Of course, there is a rumor that NetApp is developing a "Snapmanager for VMware" which may replace this entire process.
Below is my complete script that quiesces the RDM LUN, takes the NetApp snapshot, then unquiesces the RDM LUN. But first, some important background information:
Although VMware changed some names and commands between ESX 2.5 and 3.0, the concepts and functionality are the same. Taking a VMware snapshot, whether from the GUI or the command line, does indeed quiesce the vmdk.
In the case of RDM LUNs, the metadata and change log for the snapshot are stored in the VMFS volume where the vmx file resides. But that’s ok- if you had to do a restore, you would be recovering to the point in time when the VMware snapshot was taken. Incidentally, even without quiescing the vmdk, your SAN snapshot copy would be crash consistent. For a Windows VM, Checkdisk would run if you booted from that state- but it would work.
In ESX 2.5, the change log was called a REDO log. ESX 3.0 calls them COW files. COW files store a copy of each changed block in 16MB increments. So for multiple VMware snapshots or VM’s with heavy disk I/O, the COW files can grow large quite quickly. But for our purpose of taking a SAN snapshot, this shouldn’t be a concern.
There IS a concern with the number of simultaneous VMware snapshots on a single VMFS volume on the SAN, with multiple ESX servers connecting to it. According to the VCB design team, too many simultaneous disk writes to VMFS from multiple ESX servers is BAD. I couldn’t get them to go on record with a specific number of VM’s, but it was suggested that above 50 VM’s on the same VMFS volume, all with VMware snapshots, you might see performance degradation, or worse: disk corruption or even ESX crashing. You could use DRS affinity rules to group VM’s, or use some other grouping criteria in SQL when you run your quiescing script.
Another problem to consider when automating VMware snapshots, is that in the world of VMotion, DRS, and HA; the host ESX server for a particular VM can change. The script needs to send the quiescing command to the correct host, but how will it know which host has a particular VM?
My script leverages the features of several utilities.
First, VCB comes with several nifty command-line utilities. In fact, the VCB product is just a set of scripts designed to plug-in to your backup software. But those scripts all use these core utilities. You can write your own scripts to use the VCB utilities for other purposes. The VCB utilities access Virtual Center’s SQL database to get information about the target VM. In particular we will use vcbsnapshot[/i]. This command locates the proper host for the VM and creates the Vmware snapshot:
“vcbsnapshot -h localhost -u username -p password -c moref:vm-611 quiesce”
In my setup, I have VCB installed on the same Windows server as my Virtual Center (thus the localhost). But the target hostname should be your Virtual Center server. The username and password must be an account with appropriate permissions in Virtual Center. You can store the username and password in a config file so that you don’t have to write them on the command line every time. The target VM is formatted “moref:vm-611” which means “machine object reference” and 611 is the record ID in SQL. “quiesce” is the name of the snapshot. Because you are sending the command to Virtual Center, and Virtual Center knows on which host the VM resides, it automatically forwards the command to the appropriate ESX host.
Then to remove the snapshot created above, the command is:
“vcbsnapshot -h localhost -u username -p password -d moref:vm-611 ssid:snapshot-619”
Yes, you DO need to know the snapshot ID.
So how do we get these ID numbers? The answer is OSQL. OSQL is a command-line utility that comes with SQL Enterprise Manager. You only need the osql.exe file. Just copy it from your installation of SQL. You can use osql to query the Virtual Center database and get the necessary ID numbers for both the VM’s and the snapshots:
osql -U username -P password -D VirtualCenter -Q "SELECT ID FROM vc_user.VPX_VM WHERE IS_TEMPLATE = 0"
The above command connects to the database referenced by the System DSN (in your ODBC connectors) called “VirtualCenter”. It logs into the database with the listed username and password, which must have rights on the database in SQL. Then the actual SQL query pulls the record ID number for each VM that is not a template. (You can customize the filter criteria here). In my example, “vc_user” is the database owner in SQL. VPX_VM is the SQL table that stores the list of all VM's.
The output from the above command looks like this:
ID
\----
108
110
…
1226
(19 rows affected)
Similarly, to get the snapshot ID, we use:
osql -U username -P password -D VirtualCenter -Q "select VM_ID,ID from vc_user.VPX_SNAPSHOT where SNAPSHOT_NAME = 'Quiesce'"
This query returns a list of the VM ID and Snapshot ID for each VM that has a snapshot named “quiesce”. The output looks like this:
VM_ID ID
\----
-
108 1228
110 1229
… …
1226 1246
(19 rows affected)
Now all we need to do is clean this up so that it can be used in a script. Add the –o switch to direct output to a text file. Add the switch “-h-1” to remove the column headers. Then, when you run the vcbsnapshot command in a FOR loop, you set the “(“ character as the EOL character. That will cause the FOR loop to skip the last line of your output that says “(## Rows Affected)”. More on the FOR loop later:
osql -U vc_user -P vmware -D VirtualCenter –h-1 -Q "SELECT ID FROM vc_user.VPX_VM WHERE IS_TEMPLATE = 0" –o output.txt
The last component is the set of commands to manage the NetApp snapshots. I use RSH to connect to NetApp. You will need a user account with RSH permissions in NetApp. It can just be a local user account on the Windows box where you run the script. Modify etc/hosts.equiv on NetApp to grant RSH permission.
RSH nameofNetApp snap create volname snapshotname
Similarly, there is also “snap rename” and “snap delete” to manage old snapshots.
Now, let’s put it all together. Here is the script:
\--
BEGIN SCRIPT \--
@echo off
echo Script Execution Commenced at %date% %time% >> snapall.log
c:
cd "\Program Files\VMware\VMware Consolidated Backup Framework"
REM Manage old snapshots on NetApp
RSH nameofNetApp snap delete volname vmware_snap4 >> snapall.log
RSH nameofNetApp snap rename volname vmware_snap3 vmware_snap4 >> snapall.log
RSH nameofNetApp snap rename volname vmware_previous vmware_snap3 >> snapall.log
RSH nameofNetApp snap rename volname vmware_recent vmware_previous >> snapall.log
REM Get list of VM's from SQL database
osql -U vc_user -P password -D VirtualCenter -h-1 -Q "SELECT ID FROM vc_user.VPX_VM WHERE IS_TEMPLATE = 0" -o vmlist.txt
REM quiesce all VM's by creating vmware snapshots.
for /F "eol=(" %%i in (vmlist.txt) do vcbsnapshot -h localhost -u username -p password -c moref:vm-%%i Quiesce >> snapall.log
if not errorlevel 0 goto :Error
REM create new snapshot on NetApp
RSH nameofNetApp snap create volname vmware_recent >> snapall.log
if not errorlevel 0 goto Error
REM Get list of vmware snapshot ID's from SQL database
osql -U username -P password -D VirtualCenter -h-1 -Q "select VM_ID,ID from vc_user.VPX_SNAPSHOT where SNAPSHOT_NAME = 'Quiesce'" -o sslist.txt
REM unquiesce all VM's by removing all vmware snapshots.
for /F "eol=( tokens=1,2" %%i in (sslist.txt) do vcbsnapshot -h localhost -u username -p password -d moref:vm-%%i ssid:snapshot-%%j >> snapall.log
if not errorlevel 0 goto :Error
goto End
:Error
echo An error occurred during script execution. See snapall.log for details.
:End
echo Script Execution complete at %date% %time% >> snapall.log
\--
END SCRIPT \--
Works like a champ!
You can now use a LUN clone to do a machine-level restore, or mount the LUN clone with SnapDrive to do file-level restores.
Go Virtual!
