VCB failed with cannot open disk file

[olano]

Hi,

My VCB backup frequently failed with error" cannot open disk file..."

Here is the error log:

Error: Cannot open disk blklst://snapshot-19176[VMFSDataStoreKIMProdEVA1] CFMSGPKEXT01/CFMSGPKEXT01.vmdk@TKIMINFOMC02?xxxx/xxxx. Cannot open disk file: Error : Disk library failed (Unspecified error). Unable to open 'blklst://snapshot-19176[VMFSDataStoreKIMProdEVA1] CFMSGPKEXT01/CFMSGPKEXT01.vmdk@TKIMINFOMC02'. Failed to configure disk scsi1:0.

An error occurred, cleaning up...

Deleted directory E:\VCBBackupRoot\CFMSGPKEXT01

Error VM :'CFMSGPKEXT01' snapshot failed

Are there anyone know what's wrong with it?

I search on internet and found stop vmount2 service will be a workaround. So I try to stop the vmount service on the VCB proxy. First it is fine but after a while the same problem will happen again and sometimes the service can't stopped normally that I have to reset the VCB proxy. And sometimes the backup was completed with minor errors:

From: VBDA@tkimxcvomc01.omcapj.adapps.hp.com "E:\VCBBackupRoot\CFMSGPKEXT02\letters\c" Time: 10/03/2010 3:14:17 AM

E:\VCBBackupRoot\CFMSGPKEXT02\letters\c\WINDOWS\assembly\temp\PMF1TF8MF1\System.Configuration.Install.dll

Cannot read 4400 bytes at offset 0(:1): ( A device attached to the system is not functioning. ).

Are there anyone meet such problems and what's the root cause?

Thanks!

Lan Lan

[AndreTheGiant]

Can you manually create the VM snapshots?

Do you you use SAN mode?

Andre

[olano]

yes, I can create snapshot manually and we use SAN mode.(LUN)

[AndreTheGiant]

Try to use net mode (nbd instead of san) to see if there is a problem in your SAN configuration (to the VCB):

http://www.yellow-bricks.com/2008/01/15/new-vcb-features/

Andre

[olano]

Hi,

I have one question. Normally a file-level VCB backup flow should be like this:

0. run pre-backup script

1. quiesce the VM file system

2. put the VM in snapshot mode and take a snapshot

3. mount the VM snapshot from SAN to the VCB proxy

4.backup the VM

5. run post-backup script

6. umount the VM snapshot from the VCB proxy

7. take the VM out of snapshot mode

8. End

So I just wondering where the error is located? Just at step 1: it is reading the virtual disk information? If the snapshot has already been created? or the error is located at step 3, it can't mount the VM snapshot?

2010-03-09 23:22:58.707 'vcbMounter' 2948 error Error: Cannot open disk blklst://snapshot-19176VMFSDataStoreKIMProdEVA1 CFMSGPKEXT01/CFMSGPKEXT01.vmdk@TKIMINFOMC02?xxxx/xxxx. Cannot open disk file: Error : Disk library failed (Unspecified error). Unable to open 'blklst://snapshot-19176VMFSDataStoreKIMProdEVA1 CFMSGPKEXT01/CFMSGPKEXT01.vmdk@TKIMINFOMC02'. Failed to configure disk scsi1:0.

Sometimes the Backup completed with minor errors like :

From: VBDA@tkimxcvomc01.omcapj.adapps.hp.com "E:\VCBBackupRoot\CFMSGPKEXT02\letters\c" Time: 10/03/2010 3:14:17 AM

E:\VCBBackupRoot\CFMSGPKEXT02\letters\c\WINDOWS\assembly\temp\PMF1TF8MF1\System.Configuration.Install.dll

Cannot read 4400 bytes at offset 0(:1): ( A device attached to the system is not functioning. ).

Thanks!

Lan Lan

[AndreTheGiant]

You can see on your vSphere/VI client if the snap has made.

The problem usual is in step 3, and if you make a test in net mode, you can confirm is the problem is really there.

Andre

[olano]

Hi,

All of my VCB backup is file-level. This time I tried to use "vcbmounter" command to manually backup. Both fullvm-level backups at SAN or NBD mode were successful, but the file-level backup failed with the below error log:

{2010-03-13 13:53:28.363 'vcbMounter' 2572 info] Mount operation obtained backup info.

Error: Cannot open disk blklst://snapshot-19275[VMFSDataStoreKIMProdEVA2] CFMSGPKEXT02/CFMSGPKEXT02.vmdk@tkiminfomc02?xxxx/xxxx. Cannot open disk file: Error : Failed to create temporary file "D:\TEMP\vmware-SYSTEM\vmware-temp\vmware-.TMPREDO_000080644": Access is denied.

Unable to check for the existence of an old redo log or failure to choose a filename for a new redo log for disk 'blklst://snapshot-19275[VMFSDataStoreKIMProdEVA2] CFMSGPKEXT02/CFMSGPKEXT02.vmdk@tkiminfomc02'.

Failed to configure disk scsi4:0.

An error occurred, cleaning up...

what does the error mean? It mean to need some kind of admin right for the backup account?

Lan

[olano]

And I frequently found the below error logs in windows event logs of this VCB proxyx. Are there any connection for the backup failure?

Lan Lan

Event Type: Warning

Event Source: vstor3Bus

Event Category: Disk

Event ID: 57

Date: 11/03/2010

Time: 8:37:48 AM

User: N/A

Computer: TKIMXCVOMC01

Description:

The system failed to flush data to the transaction log. Corruption may occur.

Event Type: Information

Event Source: Application Popup

Event Category: None

Event ID: 26

Date: 10/03/2010

Time: 4:15:38 PM

User: N/A

Computer: TKIMXCVOMC01

Description:

Application popup: Windows - Delayed Write Failed : Windows was unable to save all the data for the file E:\VCBBackupRoot\TSG1ADCOMC06\digits\1\$Mft. The data has been lost. This error may be caused by a failure of your computer hardware or network connection. Please try to save this file elsewhere.

Event Type: Warning

Event Source: Ntfs

Event Category: None

Event ID: 50

Date: 10/03/2010

Time: 4:15:38 PM

User: N/A

Computer: TKIMXCVOMC01

Description:

{Delayed Write Failed} Windows was unable to save all the data for the file . The data has been lost. This error may be caused by a failure of your computer hardware or network connection. Please try to save this file elsewhere

[AndreTheGiant]

Do you have a SAN FC or iSCSI?

Seems that you have a stage area (E:\) on the SAN and there is some problem on it (check driver version and configuration).

And check also that you can see (under disk management) also the VMFS LUNs (you must only see them, but not touch at all).

Andre

[olano]

Thanks Andre, I am using a SAN FC.

But I need to say the E:\ is not on SAN. It is on local disk of the VCB proxy.

what do you mean the driver and configuration? what kind of driver do I need to check?

I can see our six VMFS LUNs under disk management. Any thing else I need to check?

I also suspect something wrong with E:\, but I am not sure what kind of problem it is....

Thanks!

[AndreTheGiant]

From this error:

Delayed Write Failed : Windows was unable to save all the data for the file E:\VCBBackupRoot\TSG1ADCOMC06\digits\1\$Mft

I though that was also a SAN disk.

Check your controller, driver and your disks, cause it is a strange problem for a local disk.

Andre