Problems with VCB on VMs with multiple drives on different SAN LUNs

[bobrenzi]

We are attempting to use VCB to perform file level backups on Windows VMs using the TSM interoperability module and have been getting nowhere fast. Here is what we have done so far:

1. Installed and configured our VCB proxy to communicate with the VC server controlling our test VMs.

2. Installed an admin account on the VC server and individual ESX servers (I am not sure that this isn't overkill - but the documentation isn't clear) which will execute the backups.

3. Granted the VM Admin role to the backup account

4. Configured the TSMIM as prescribed in the readme file. We are doing a simple test on 1 VM.

5. Configured the TSM to execute the pre-/post- commands.

Before continuing with my tale, let me state that our VMs are configured with two virtual disks - a c: for system data and d: for user data) - but we are only interested in backing up the d: drive - so that is all that is the only DOMAIN specified in the dsm.opt file. Also, both of these virtual drives are stored on separate SAN LUNs. Other environment factors to consider is that we are using HA and DRS.

After several failed attempts that leave us with less than sufficient diagnostic entries in the log, I decided to try to perform the backup process in a stepwise fashion via browse-start command to see if I can mount the VM drives to the proxy. I run the following "browse-start testvcb1 vm_name" from a command window that is in the TSM directory containing the script. The command executes and reports the following in the browse-start.log file:

Port is not set. defaulting to 902

\[Time stamp 'APP' 1224 info] Current working directory is C:\Tivoli\TSM\baclient

\[Time stamp 'Baselibs' 3252 warning] \[vmdb_unset] Unsetting unknown path: /vmomi/

\[Time stamp 'vcbmounter' 1224 error] Error: Error while opening blklist://snapshot-3177 \[train-os-vol-02] vm_name/vm_name.vmdk@vc_srvr_name.dom:902?xxx\xxx A virtual disk could not be opened. Cannot open disk file: Error : unable to open blklist://snapshot-3177 \[train-os-vol-02] vm_name/vm_name.vmdk@vc_srvr_name.dom:902' Failed to configure scsi0:0.

\[Time stamp 'vcbmounter' 1224 error] An error occurred, cleaning up

\[Time stamp 'vcbmounter' 1224 warning] Snapshot deletion failed. Attempting to clean up snapshot database.... failed to prepare vm_name.dom for backup. PrepareForBackup() returned error 1

External command failed. See error above.

Exit Code: 1

At this point, it gets more interesting because now when I look at the VM from the VI Client, the LUN/volume that contains the virtual d: drive is no longer listed. I still see the virtual d: drive directory on the SAN LUN from the ESX servers, but not from the VIC. I suspect that this has something to do with the state of VM following the failure of the snapshot. Additionally, the VIC reports that a snapshot is in place and I verified presence of the delta files for the system (c:) drive in its SAN directory. Consequently, the state of my VM is all messed up at this point, so here are my questions.

1. How do I clean up the state of the VM at this point? Performing a "Delete All" from the snapshot manager seems to get me part of the way there - but the delta and sever numbered vmdk files are left behind.

2. Does VCB have a problem snapshotting a VM that has multiple virtual drives on different SAN LUNs?

Being that we are proto-typing this using "disposable" VMs I am going to attempt to do this one more time and see if the same issues surface. If anyone has anything constructive to suggest - I'd be glad to hear it.

Thanks In Advance

BR

[kix1979]

1. How do I clean up the state of the VM at this

point? Performing a "Delete All" from the snapshot

manager seems to get me part of the way there - but

the delta and sever numbered vmdk files are left

behind.

You have to power the VM off to clean it up properly in most cases.

2. Does VCB have a problem snapshotting a VM that

has multiple virtual drives on different SAN LUNs?

Yes/No... VCB works fine with multiple VMDKs across LUNs, BUT if you created them with the VI Client it will name them the same. This creates an issue when you snapshot the VM, so make sure the VMDKs are names uniquely. Once that is done you should be much better off

Kix

[bobrenzi]

Thank you - we do build the VMs from VI Client and it does in fact create identically named vmdk, albeit on differ volumes in different subdirectories. So the basic question is how do we rename the vmdk? If we do rename it, is there something that we need to do to keep its association with the VM?

Thanks Again

BR

[ZMkenzie]

You need to rename both the file.vmdk, the file-flat.vmdk and all the occurrences in your vmx file (usually it's just one under called scsi0:0 filename).

[weili]

There is a problem known by vmware when you have the latest patches installed. My problem is, that vcb stucks when reading disks. All VMs with only one vmdk or more on the same lun are working - the others don't. It is definetly a bug with the latest patches.

Michael

[davidbarclay]

Use vmkfstools -E instead, it takes care of everything for you

Dave

[davidbarclay]

I standardise on

_2.vmdk

regardless if they are on separate LUNs or not - that way you always have unique names and always know which disk is what in your VM (as _1 = "Hard Disk 1" and _2 = "Hard Disk 2" and so on).

Dave

[bobrenzi]

Here is an update -

1. I first attempted to rename the file using the vmkfstools -E command with the VM powered off. The rename took but VC complained "Unable to access file since it was locked" when I attempted to power on the VM and the command failed.

2. Next, after I renamed the file back and restarted the VM, I attempted to rename the file using the vmkfstools -E command with it powered on. The rename took but the VM would not recycle with VC complaining "Cannot open disk file /vmfs/volumes/wwid/vmbnane/vmvname.vmdk or one of the snapshot disks it depends on. Reason: device or resource busy". After I rename the disk back, the VM boots again.

In essence, both renaming scenarios don't work as I would have hoped. Any suggestions? Is there an intermediate step that I am not doing?

[bobrenzi]

Update to the update -

I took a closer look at the thread and thought I identified my missing step as I did not change the vmname.vmx file located in the primary VM directory. However, everytime I change the file on the ESX server and then reboot from VC, the reboot process resets the scsi0:1.filename back to its orginal name and I get the same locked message as before.

[admin]

Power off the VM. Commit any existing snapshots. Edit the VM settings and remove the disk you want to rename. Use vmfkstools -E originalvmdkname.vmdk newvmdkname.vmdk to rename the disk. Edit the settings of the VM, add a disk, select use existing disk file, then browse to your renamed VMDK file. Do this for each disk until all names are unique.

More detailed instrucations here: http://kb.vmware.com/kb/5096672

[bobrenzi]

Thanks - this did the trick in renaming the file. Now I can get back to working with VCB.

[GBromage]

Before continuing with my tale, let me state that our

VMs are configured with two virtual disks - a c: for

system data and d: for user data) - but we are only

interested in backing up the d: drive - so that is

all that is the only DOMAIN specified in the dsm.opt

file. Also, both of these virtual drives are stored

on separate SAN LUNs[/b]. Other environment factors to

consider is that we are using HA and DRS.

I suspect that this might be your problem. I recall reading somewhere (and I don't know where) that if you have two virtual disks, VCB doesn't like them being on different LUNs.

[admin · Accepted solution]

Lol, read the KB article in my post two above yours.

bobrenzi - glad that worked for you. Remember all disk names must be unique, even if the disks are on other LUNs, so when you add disks in future remember to name them manually. p.s. points? \*hint hint*

[bobrenzi]

Update - now that the second drive on the VM has been renamed and properly added, I kicked off the browse-start command again. The VIC reports that the snapshot for the two virtual drives completes and I see delta files for both virtual drives are stored in the datastore for the C: drive (no other files are added to the second datastore on the other LUN). However, the browse-start log reports that vcbmounter has an error 3856 and terminates without mounting the drives. It additionally leaves the VM in a "quasi" snapshot state. Looks like I am getting closer but still not there. Any ideas?

[bobrenzi]

Everyone -

I have received this notice from my VMware Sales Engineer stating the following:

"Nothing that helps you right this second but here is a little info with respect to Snapshots in VCB.

Known bug in ESX 3.0.x. Snapshots are horribly broken for VMs with disks on different SAN LUNs. CPD is targetting a fix for ESX 3.0.2

3.0.2 should not be that far out…,"

Looks like my issue will be resolved by the next point release.

[Byron_Zhao]

A simpler way to rename the second disk is to migrate the powered off VM to another LUN, and the migration process will take care of the renaming.