VCB, iSCSI and BackupExec 11d - Not Working - Please HELP!

[RDArmstrong]

Greetings,

I need assistnace figuring out why VCB will not work. Been working on this for a week straight and still nothing but problems. Here's my setup

ESX 3.0.2

VCB 1.0.3

VC 2.0.2

VCB Proxy - Windows 2003 Ent SP1 running Symantec BackupExec 11D fully patched

Equallogic PS300e iSCSI SAN

VCB Proxy is running MSiSCSI Initiator 2.0.4

I have a single 1GB interface (1 path) to the SAN from the VCB Proxy/BackupExec Server

My BackupExec Pre and Post Jobs are:

cmd /c "C:\Program Files\VMware\VMware Consolidated Backup Framework\backupexec\pre-backup.bat" VMBackup INVSTGASC02-FullVM

cmd /c "C:\Program Files\VMware\VMware Consolidated Backup Framework\backupexec\browse-end.bat" VMBackup

I'm using f:\mnt and when the Pre script runs, I do get a INVSTGASC02-FullVM folder and files get copied to it, but then the folder disappears and BackupExec job failes and reports that Pre job errored out. Here is the contents of the error log:

\[2007-08-22 11:30:50.312 'App' 664 info] Current working directory: E:\Program Files\Symantec\Backup Exec

\[2007-08-22 11:30:50.312 'BaseLibs' 664 info] HOSTINFO: Seeing Intel CPU, numCoresPerCPU 2 numThreadsPerCore 1.

\[2007-08-22 11:30:50.312 'BaseLibs' 664 info] HOSTINFO: This machine has 2 physical CPUS, 4 total cores, and 4 logical CPUs.

\[2007-08-22 11:30:50.312 'BaseLibs' 664 info] Using system libcrypto, version 90703F

\[2007-08-22 11:30:50.734 'BaseLibs' 4672 warning] \[Vmdb_Unset] Unsetting unknown path: /vmomi/

Copying "\[VMWare_SAN_VOL1] INVSTGASC02_1/INVSTGASC02.vmx":

0%=====================50%=====================100%

**************************************************

Copying "\[VMWare_SAN_VOL1] INVSTGASC02_1/INVSTGASC02.nvram":

0%=====================50%=====================100%

**************************************************

Copying "\[VMWare_SAN_VOL1] INVSTGASC02_1//vmware-1.log":

0%=====================50%=====================100%

**************************************************

Copying "\[VMWare_SAN_VOL1] INVSTGASC02_1//vmware-2.log":

0%=====================50%=====================100%

**************************************************

Copying "\[VMWare_SAN_VOL1] INVSTGASC02_1//vmware-3.log":

0%=====================50%=====================100%

**************************************************

Copying "\[VMWare_SAN_VOL1] INVSTGASC02_1//vmware-4.log":

0%=====================50%=====================100%

**************************************************

Copying "\[VMWare_SAN_VOL1] INVSTGASC02_1//vmware-5.log":

0%=====================50%=====================100%

**************************************************

Copying "\[VMWare_SAN_VOL1] INVSTGASC02_1//vmware-6.log":

0%=====================50%=====================100%

**************************************************

Copying "\[VMWare_SAN_VOL1] INVSTGASC02_1//vmware.log":

0%=====================50%=====================100%

*************************************************

\[2007-08-22 11:31:00.500 'BaseLibs' 4672 warning] \[Vmdb_Unset] Unsetting unknown path: /vmomi/

\[2007-08-22 11:31:01.938 'App' 664 error] No path to device LVID:44c3c8bc-bb8c49bf-db9e-0019bb251ea0/44c3c8bb-85a7f18a-538f-0019bb251ea0/1 found.

\[2007-08-22 11:31:01.938 'BlockList' 664 error]

\[2007-08-22 11:31:02.172 'vcbMounter' 664 error] Error: Failed to open the disk: One LUN could not be opened

\[2007-08-22 11:31:02.172 'vcbMounter' 664 error] An error occurred, cleaning up...

failed to prepare INVSTGASC02 for backup, PrepareForBackup() returned error 1

External command failed. See error above.

Exit Code: 1

Does anybody have any ideas why this is happening?

Any help will be greatly appreciated

Regards,

Rob

[alex2801]

i recently got the vcb working wih backupexec and equalogic. First you should test vcb with the command prompt on your vcb proxy. Use vcbmounter.exe to make a full image of a vm. If vcbmounter.exe does not work,BE surely won't work.

good luck

[woharrow]

How is your VCB server connected to your PS array?

also, how do you have the mapping set up in the array for the volumes?

another thing to make sure is that all the files for the VM are in the same folder and that you don't have a 2nd or 3rd disk in the VM as VMDKs on a different volume.

While I haven't tried it yet, I'm curious to see if you can run a script that will do a Pre-Freeze to the VM(s) followed by a Snapshot command to the array, with the Post-Thaw to the VM(s). Then just go mount the snapshot with your VCB host and back it up at will without having to share the volume with running ESX servers.

Let us know what works.

Thanks

Wade

[RDArmstrong]

Greetings, and thank you for the response. I'll attempt to answer your questions as best I can.

The VCB server (VCB proxy) connects to the PS array through a single GB interface and uses the MS iSCSI initiator. I have a separate SAN network that is isolated from my server subnet. From the VCB/BackupExec server I can browse and see and attach the LUN that stores the VMWare VMDKs.

All files for my VM guests are stored on the same volume in folders corresponding to the VM guest name.

The only odity that I can think of is at one point I had to add space to the volume on the SAN and then add an Extent in ESX Storage Adapters to extend the volume.

I don't understand the question about how mapping is setup in the array for the volumes. I'm not familiar with that aspect of the array configuration.

I'm also not sure how to accomplish the Pre-Freeze / Snapshot / Post-Thaw tasks you suggest in your last paragraph.

Sorry, this is all very new to me and I'm absorbing bits and pieces each day.

Regards,

Rob

[RDArmstrong]

Alex,

Was there anything special that you had to do on the Equallogic array in terms of LUN ID etc that I may have not done?

Regards,

Rob

[alex2801]

Rob,

On the Equalogic i left everything standard. You could use vcbmount.exe in the commandprompt with the debug logging option to identify the problem. note that If the pre script in BE returns any kind of error the job wil not run.

Alex

[RDArmstrong]

Alex,

I executed vcbMounter with the log level set to 6. Here are the results. It doesn't appear that vcbMounter can find the volume ID that is passed back to it from Virtual Center server (however, I'm not sure I'm reading the log correctly)

vcbMounter -h vcserv01.domain.inc -u AdminUser -p adminpwd -L 6 -a ipaddr:VSERVER01 -r F:\mnt\VSERVER01-fullVM -t fullvm

\[2007-08-27 09:26:12.177 'App' 4284 info] Current working directory: C:\ProgramFiles\VMware\VMware Consolidated Backup Framework

\[2007-08-27 09:26:12.177 'BaseLibs' 4284 info] HOSTINFO: Seeing Intel CPU, numCoresPerCPU 2 numThreadsPerCore 1.

\[2007-08-27 09:26:12.177 'BaseLibs' 4284 info] HOSTINFO: This machine has 2 physical CPUS, 4 total cores, and 4 logical CPUs.

\[2007-08-27 09:26:12.177 'vcbMounter' 4284 trivia] Initializing SSL context

\[2007-08-27 09:26:12.177 'BaseLibs' 4284 info] Using system libcrypto, version 90703F

\[2007-08-27 09:26:12.396 'App' 4284 info] Vmacore::InitSSL: doVersionCheck = true, handshakeTimeoutUs = 120000000

\[2007-08-27 09:26:12.396 'vcbMounter' 4284 info] Connecting to host vcserv01.domain.inc on port 902

\[2007-08-27 09:26:12.615 'vcbMounter' 4284 verbose] Connected to stream: \[classVmacore::Ssl::SSLStreamImpl:01676C18]

\[2007-08-27 09:26:12.615 'BaseLibs' 4512 warning] \[Vmdb_Unset] Unsetting unknown path: /vmomi/

\[2007-08-27 09:26:12.630 'vcbMounter' 4284 info] Authenticating user AdminUser

\[2007-08-27 09:26:12.849 'vcbMounter' 4284 info] Logged in!

\[2007-08-27 09:26:12.849 'vcbMounter' 4284 info] Performing SearchIndex find.

\[2007-08-27 09:26:13.052 'vcbMounter' 4284 info] Attempting index search.

\[2007-08-27 09:26:13.052 'vcbMounter' 4284 info] Found VM using index search.

\[2007-08-27 09:26:13.052 'vcbMounter' 4284 info] Got VM MoRef

\[2007-08-27 09:26:13.052 'vcbMounter' 4284 info] Got access method

\[2007-08-27 09:26:13.052 'vcbMounter' 4284 info] Got coordinator object

\[2007-08-27 09:26:13.052 'vcbMounter' 4284 info] Attempting data access.

\[2007-08-27 09:26:13.255 'vcbMounter' 4284 info] No snapshot info for this VM, nothing to do.

\[2007-08-27 09:26:13.255 'vcbMounter' 4284 info] Creating snapshot

\[2007-08-27 09:26:15.661 'vcbMounter' 4284 info] Snapshot created, ID: snapshot-406

\[2007-08-27 09:26:15.661 'vcbMounter' 4284 info] Mount operation created snapshot.

\[2007-08-27 09:26:15.693 'vcbMounter' 4284 verbose] Found a device: vim.vm.device.VirtualBusLogicController

\[2007-08-27 09:26:15.693 'vcbMounter' 4284 verbose] Found a device: vim.vm.device.VirtualDisk

\[2007-08-27 09:26:15.708 'vcbMounter' 4284 info] Mount operation obtained backup info.

\[2007-08-27 09:26:15.708 'vcbMounter' 4284 info] Creating export directory

\[2007-08-27 09:26:15.708 'vcbMounter' 4284 info] Writing restore config file

\[2007-08-27 09:26:20.693 'vcbMounter' 4284 info] Exporting VM config files

\[2007-08-27 09:26:21.099 'vcbMounter' 4284 info] Establishing NFC connection tohost invesxbl02.domain.inc on port 902

Copying "\[VMWare_SAN_VOL1] VSERVER01_1/VSERVER01.vmx":

0%=====================50%=====================100%

**************************************************

Copying "\[VMWare_SAN_VOL1] VSERVER01_1/VSERVER01.nvram":

0%=====================50%=====================100%

**************************************************

Copying "\[VMWare_SAN_VOL1] VSERVER01_1//vmware-1.log":

0%=====================50%=====================100%

**************************************************

Copying "\[VMWare_SAN_VOL1] VSERVER01_1//vmware-2.log":

0%=====================50%=====================100%

**************************************************

Copying "\[VMWare_SAN_VOL1] VSERVER01_1//vmware-3.log":

0%=====================50%=====================100%

**************************************************

Copying "\[VMWare_SAN_VOL1] VSERVER01_1//vmware-4.log":

0%=====================50%=====================100%

**************************************************

Copying "\[VMWare_SAN_VOL1] VSERVER01_1//vmware-5.log":

0%=====================50%=====================100%

**************************************************

Copying "\[VMWare_SAN_VOL1] VSERVER01_1//vmware-6.log":

0%=====================50%=====================100%

**************************************************

Copying "\[VMWare_SAN_VOL1] VSERVER01_1//vmware.log":

0%=====================50%=====================100%

*************************************************

\[2007-08-27 09:26:22.177 'vcbMounter' 4284 info] Exporting disks

\[2007-08-27 09:26:22.177 'vcbMounter' 4284 info] Filenames: blklst://snapshot-406\[VMWare_SAN_VOL1] VSERVER01/VSERVER01.vmdk@vcserv01.domain.inc?AdminUser/adminpwd --> F:\mnt\VSERVER01-fullVM\scsi0-0-0-VSERVER01.vmdk

\[2007-08-27 09:26:22.240 'App' 4284 trivia] Evaluating 1 paths.

\[2007-08-27 09:26:22.240 'App' 4284 trivia] Trying to open path
?\scsi#disk&ven_hp&prod_logical_volume&rev_1.20#6&28f4192b&0&000400#\{53f56307-b6bf-11d0-94f2-00a0c91efb8b}.

\[2007-08-27 09:26:22.240 'App' 4284 info] Now using Path
?\scsi#disk&ven_hp&prod_logical_volume&rev_1.20#6&28f4192b&0&000400#\{53f56307-b6bf-11d0-94f2-00a0c91efb8b}.

\[2007-08-27 09:26:22.240 'App' 4284 trivia] Reading 32256 bytes from offset 0.

\[2007-08-27 09:26:22.255 'App' 4284 trivia] Found 1 partition(s) on this device.

\[2007-08-27 09:26:22.255 'App' 4284 trivia] Evaluating 1 paths.

\[2007-08-27 09:26:22.255 'App' 4284 trivia] Trying to open path
?\scsi#disk&ven_hp&prod_logical_volume&rev_1.20#6&28f4192b&0&000500#\{53f56307-b6bf-11d0-94f2-00a0c91efb8b}.

\[2007-08-27 09:26:22.255 'App' 4284 info] Now using Path
?\scsi#disk&ven_hp&prod_logical_volume&rev_1.20#6&28f4192b&0&000500#\{53f56307-b6bf-11d0-94f2-00a0c91efb8b}.

\[2007-08-27 09:26:22.255 'App' 4284 trivia] Reading 32256 bytes from offset 0.

\[2007-08-27 09:26:22.255 'App' 4284 trivia] Found 1 partition(s) on this device.

\[2007-08-27 09:26:22.255 'App' 4284 trivia] Obtained list of SCSI disk devices.

\[2007-08-27 09:26:22.255 'BlockList' 4284 info] Establishing new connection for vcserv01.domain.inc:AdminUser

\[2007-08-27 09:26:22.255 'blocklist' 4284 trivia] Initializing SSL context

\[2007-08-27 09:26:22.255 'blocklist' 4284 info] Connecting to host vcserv01.domain.inc on port 902

\[2007-08-27 09:26:22.474 'blocklist' 4284 verbose] Connected to stream: \[class Vmacore::Ssl::SSLStreamImpl:016AE590]

\[2007-08-27 09:26:22.474 'BaseLibs' 6052 warning] \[Vmdb_Unset] Unsetting unknown path: /vmomi/

\[2007-08-27 09:26:22.490 'blocklist' 4284 info] Authenticating user AdminUser

\[2007-08-27 09:26:22.787 'blocklist' 4284 info] Logged in!

\[2007-08-27 09:26:23.709 'App' 4284 trivia] Attempting to open LVID:44c3c8bc-bb8c49bf-db9e-0019bb251ea0/44c3c8bb-85a7f18a-538f-0019bb251ea0/1.

\[2007-08-27 09:26:23.709 'App' 4284 error] No path to device LVID:44c3c8bc-bb8c49bf-db9e-0019bb251ea0/44c3c8bb-85a7f18a-538f-0019bb251ea0/1 found.

\[2007-08-27 09:26:23.709 'BlockList' 4284 error]

\[2007-08-27 09:26:24.005 'BlockList' 4284 info] Closing connection vcserv01.domain.inc:AdminUser

\[2007-08-27 09:26:24.005 'blocklist' 4284 verbose] Shutting down VMDB service...

\[2007-08-27 09:26:24.005 'blocklist' 4284 verbose] Unregistering callback...

\[2007-08-27 09:26:24.005 'blocklist' 4284 verbose] ...done

\[2007-08-27 09:26:24.005 'vcbMounter' 4284 error] Error: Failed to open the disk: One LUN could not be opened

\[2007-08-27 09:26:24.005 'vcbMounter' 4284 error] An error occurred, cleaning up...

\[2007-08-27 09:26:26.427 'vcbMounter' 4284 verbose] Shutting down VMDB service...

\[2007-08-27 09:26:26.427 'vcbMounter' 4284 verbose] Unregistering callback...

\[2007-08-27 09:26:26.427 'vcbMounter' 4284 verbose] ...done

[fng]

It seems that the volume is not seen exactly the same between your proxy and ESX.

Couldn't it be a multipath problem ?

As far as I remember VCB doesn't support it.

[RDArmstrong]

It could be a multipath problem, but I'm not sure where. I have a single interface in VLAN15 (IP 192.168.1.10) on my VCB proxy and I have a single interface on VLAN15 (IP 192.168.1.15) on my Virtual Center server. Each ESX Server does have two teamed ports on VLAN15 for redundancy. Do you think this could be the "multipath" that's causing the heartburn?

Thanks

Rob

[Raymond]

Hi there,

I pretty much got this working last week too. Using the almost the same setup.

I read ur pre and post jobs and I see an issue.

1. Are you trying to do a Full VMDK backup, i see u used -FULLVM so I guess that is a yes. Just want to confirm that.

If so here are the pre and post commands i used and know work.

The -FullVM option must use pre-backup.bat and post-backup.bat i.e.

Pre

cmd /c "C:\Program Files\VMware\VMware Consolidated Backup Framework\backupexec\pre-backup.bat"

Try to use these commands and let know if it works.

Thanks

Ray

[RDArmstrong]

Ray,

Thanks for the reply. I really appreciate your assistance. I'm not quite at that stage as I cannot get vcbMounter.exe functioning from a command prompt, so my BackupExec jobs fail because vcbmounter fails. Are you using an Equallogic array too? If so, do you have more than one array member? I have 3 PS300e's and I'm wondering if having more than one array and having fully redundant pathing between the arrays is causing my problems. It appears that the LUN ID that the ESX server does not match the LUN ID that the VCB proxy is told to mount.

I'm stuck! I'll keep fishing from several ponds until I find a solution

Regards,

Rob

[Raymond]

Hi there,

I am only using 1 EQL Array for my testing. I have not tired to use the vcbMounter.exe commands I have only used the GUI from Backup Exec. I can try out the commands tomorrow when I am back in the office.

Thanks

Ray

[alex2801]

Rob,

Can you test to backup a vm that is on a volume that is not extended? VMware does not recommend extending volumes (only temporary). Did you contact vmware support with this problem, afterall iscsi and equalogic is official supported i think?

Goodluck and keep us posted

Alex

[RDArmstrong]

Alex,

Yes, just yesterday afternoon, I created and added a new SAN volume to ESX. I moved one of the guest machines to this new storage volume and I encountered the same error message when attemting the vcbMount. So, it doesn't appear that the issue is related to the volume extents. Thanks for the advice on extents. I will proceed with moving the rest of my guests onto the new un-extended volume to avoid any potential problems.

Yes, I do have a case open with VMWare support (through HP). Our VMWare support is through HP since we purchased the licensing through HP along with the server hardware. The case has been open for a week and I have not had any response to my last email to them on last Friday afternoon.

I will continue to work on this and post my results for the group.

Thanks for your help!

Rob

[RDArmstrong]

Ray,

Thanks for your help. It's interesting that you've had no problems while using 1 array. My problems may have something to do with the multiple physical paths that server can use to access the SAN volumes. Not sure at this point as I'm trying to gain a better understanding of the entire architecture.

Regards,

Rob

[alex2801]

Rob,

I only contacted vmware support directly in the past and they responded immediately with an estimate timeframe for an answer. your HP guys should respond quicker!!

Alex

[RDArmstrong]

Yea, it's sort of like having a your Ford repaired at a GM dealership

[Raymond]

Hi there,

Well i was able to get the command line vcbMounter working, let me know if you want me to paste the command i used.

You maybe correct with multiple physical paths is causing your issue since i am only using 1 array.

Thanks

Ray

[RDArmstrong]

Okay folks, this is one of those "tail between my legs moments". I was successful in getting VCB working and the problem I was having was a result of my own stupidity. drum roll please....

The problem was that I had NOT logged onto the SAN volume (where the VMDKs reside with the MS iSCSI initiator. I guess I was expecting VCB to mount the volume as needed, but I was wrong. Once I logged onto the volume with the initiator, VCB worked perfectly.

So, thanks to all for your support. Hopefully, once I gain a bit more experience in this arena I'll be able to return the favor(s).

Regards,

Rob