VCB mount process, very slow

[arkturas]

We are currently testing VCB, when I run the VCB mount command to backup a VM, the mount process takes a very long time to complete.

Environment:

ESX3.5

VCB 1.5, Windows 2003 R2 staging area 200GB EVA5000

VM : Windows 2003 R2, 158GB vmdk total

The process takes 48min to complete the Disk Queue on the VCB server is constantly at 100!

I run the following command from the vcb server: vcbmounter -h esx.domain.com -a user -p password -r D:\VCBMount -t fullvm

is this what I should expect to see ? help would be appreciated

[athlon_crazy]

Remember mount process not only mount the volume as what you can see in nix environment. The process also include snapshot creation, transfer the image, remove the snapshot n etc. for hundred over gb images, i would consider this acceptable.

System Engineer

Zen Systems Sdn Bhd

Malaysia

[dconvery]

48 minutes is actually pretty good. There is a limitation in the way CMD.exe handles file copies and it only gets about 1GB/min on a good day.

Dave

************************

Accomplishing the impossible means only that the boss will add it to your regular duties.

Doug Larson

[arkturas]

Thanks for the response guys, in the production environment I will need to take nightly image backups of VM's totalling 2.3 TB (+/-80vms), the utilisation on the tape drives are pretty high as it is with other backups

The backups will work something like this:

1. automated job will start mounting the VM data to the staging area. using the VCBmounter command.

2. once job 1 has completed a second job will backup the VM data as a file level backup.

Its crucial that the 1st job mount all the VM's to the staging area with enough time to start the backup to tape operation.

If what you say is true (and I sincerely hope it isn't) I will need to mount the VM data concurrently to achieve a favourable outcome.

If have also asked vmware support guys to look into this/ will post the results when i get feedback from them. (and award points accordingly)

[athlon_crazy]

u're so lucky if u can mount more than 8x vm concurrently. my environment allow me to mount max 4x vm only. it's bottleneck, sometime caused my esx doesn't response at all.

System Engineer

Zen Systems Sdn Bhd

Malaysia

[arkturas]

it caused your ESX server to not respond! thats the reason I'm trying to move away from backup agents on the ESX server's and backups being performed over the LAN, why oh why does snapshotting at the SAN level have to be so expensive! I'm sure it would make my life easier...

[arkturas]

I cant seem to resolve the Disk Queue issue, is anyone else getting high disk Queue length when running vcbmounter ? see attached image

[iceman76]

Hi all,

here are some ideas and real world numbers from our experience and from VMWare consultants

Workload is generally i/o bound and depends on following factors

- FC thtoughput to VCB (Sppeed of FC-Switch, Speed of FC-Port)

- if you do full vm backup, throughput to holding space

- number of hba's

- how much i/o the backup software can handle

If you need much throughput it is highly recommended to use multiple hba's for reading and writing to speed up the thing.

But now for the numbers

The Guys from VMWare got a transfer-rate of 500GB/hour - and this was "only" on a 2gb fc link. And these are real word numbers from the lab, not from marketing

We get transfer-rates about 220Gb/hour but we only write to a scsi-attached storage, and the file is beeing compressed while beeing copied.

So it should be possible to backup the vm's faster.

have you checked how long the copying takes, and how long taking&releasing the snapshot ? What does your storage-system tellyou about i/o queues during the copy-process. i don't know if you can trust the perf-mon values

regards

carsten

[dconvery]

Here's how they managed that throughput....

Dave Convery - VMware vExpert 2009

************************

Accomplishing the impossible means only that the boss will add it to your regular duties.

Doug Larson

[arkturas]

Ok, here's what I'm using to test VCB, unfortunately before this thread, I have been unable to find any decent benchmarks on what we are supposed to be getting with VCB *So thanks to everyone (points will be awarded)

Our test/Setup:

SAN_INF

EVA5000 A/P LUN

1x 200GB (VMFS)

1x 200GB (VCB/NTFS)

EVA6000 A/A LUN 200GB

HP 2GB FC switches (switch port is set to auto)

VCB_ INF

Dell 1855 Blade /Qlogic 2GB/s (4GB Memory, Dual procs)

A 200GB LUN on the EVA 5000

Windows 2003 R2

VCB framework 1.5

MPIO/ EVA5000 & EVA6000

ESX_INF

ESX3.5.0

Dell 1955 Blade /Qlogic 2GBs (8GB memory, dual quad core procs)

The VCB server & the ESX server are located in the same blade chassis system.

I’ve installed SAN Surfer on the VCB server and noticed when mounting a 120GB VM of which 88GB used it takes about 40min to mount & compress the VM.

According to SAN surfer HBA stats, I'm getting about 72.1 megabytes/sec (the bandwidth obviously split between the reading the LUN from the ESX server and exporting the VM to the VCB LUN) perhaps I need to look into teaming two HBA's, I think I seen some config options on the SAN/Surfer application for this.