VMware Cloud Community
pauliew1978
Enthusiast
Enthusiast

backup strategy

Hi there,

I have made a bit of a slip up in terms of planning for my vm backups. I pretty much ran out of budget for my esx implementation so I didnt have much spare for the backup part of the strategy. I'll explain my set up first.

2xesx servers (hp dl385's)

2xsanmelody servers with 1tb of storage (replicated between the two)

the above set up uses multipathing in an active/passive san failover. This all (so far ) works nicely.

My problem are my backups. I bought a buffalo terastation and had planned to use vmbk.pl to ftp the files to the terastation. However I made a monumental slip up in thinking the terastation would have a fairly decent write speed. It turns out it only writes at about 7.5mb/s!!. So for my 4 vms whcih are about 400gb in combined size it will take an eternity to copy (15 hours or something). This is itself isnt so bad as It is my understanding that using the vmbk script you essentially take the load off the vm as it does a backup of a snapshot (please point me out here if I am wrong). The problem comes when restoring a backup. If I get a corrupted vm it is going to take me 4 to 6 hours to restore from backup. The only alternative I can think of is to use the sata terastation disks in my sanmelody server (as they are sata also) and create a new lun to use as a storage lun. Then set up the vmbk script to copy it across to the new lun (which hopefully will give me faster performance than 7.5mb/s!).

How does everyone else store their vm backups and what sort of restore perfromance do you get?

many thanks for any help/suggestions.

Paul

Message was edited by:

pauliew1978

Reply
0 Kudos
9 Replies
Mike_Fink
Enthusiast
Enthusiast

Paul,

To begin, there are SO many different ways to backup and restore VMs that it's almost impossible to say what "others" are doing. In an environment like yours, what you are proposing is a reasonable way to get the .VMDK files backed up.

And, yes, if you lose a VM, you will need to transfer the exported VMDK file back from the terastation to the VMFS that is currently stored on the SANMelody servers. And, yes, this is going to take quite awhile.

The bottleneck, in this case, is the disk drive(s) in the Terastation. Conventional hard drives can only write ~10MB/s, and that is assuming a real best case scenerio. Although the issue is more complex, spindle count really determines overall read/write speed; which is why enterprise class SANs allow striping of data across dozens/hundreds of hard drives.

Using a high speed SAN (for both storage and backup locations) you may be able to get 50-70MB/s doing an export. However, as you're currently config, the only thing that is going to drastically improve your performance is more disk drives.

Also, a product that does compression (like ESXRanger) may also increase your speed. The underlying problem, however, is spindle count.

Reply
0 Kudos
pauliew1978
Enthusiast
Enthusiast

Hi Mike,

Thanks for the reply. I checked out the western digital se disks that reside in the terastation and they should theoretically write at 60mb/s (acording to http://www23.tomshardware.com/storage.html?modelx=33&model1=136&model2=676&chart=36). However the raid 5 is software raid and it appears that the terastation just really struggles with it.

I have put them in to a raid 1 set up as apparently raid 1 is done at the hardware level on the teraststion rather than at the software level so we'll see if there is a difference. mmm if only i could get esx ranger Smiley Sad and a decent nas device. Untill then vm corruption will have to be resolved using 5 hour restores. I am just hoping that vm corruption will not happen this year (then I can get a decent nas device).

Reply
0 Kudos
Mike_Fink
Enthusiast
Enthusiast

Paul,

First off, you're welcome. Smiley Happy

Drive speed is one of those "black arts" for measurement. Yes, if the data is sequential, and is fed to the drive at the right speed (so that it can write a block as soon as it recieves the block; without the need for a further rotiation), and your segment size is large enough....

You probably are starting to get the idea. Here is a good page on how disk IO and disk transfer rate are related:

http://storageadvisors.adaptec.com/2006/11/16/performance-units-of-measure/

I have seen some people get very good numbers from some specific array configurations; however, in most situations, you can figure a hard drive is good for ~10MB/s max transfer rate (assuming no RAID overhead). There are IOps calculators out there on the web for lots of different situations; I would guess that the WD drives might be able to do 125-150 IOPS. If you figure 32KB segment size, that gives you ~3.2-5 MB/sec.

Remember, in the storage world, spindles are beginning (and sometimes the end) of the speed calculation.

Oh, and BTW, yes, a RAID1 will likely marginally improve your write performance (because no parity calculations). However, RAID0 would result in a 2X gain over a RAID1 configuration (for write intensive IO). RAID1 provides improved read IO, but not improved write (over a single drive). RAID0 provides improved write IO, and better read IO (however, as I am sure you are aware, provides no redundancy).

Reply
0 Kudos
kharbin
Commander
Commander

You say "use vmbk.pl to ftp the files". What speed is the network? If 100MB, than 7.5Mb/sec ftp speeds are average. If a 1GB network, should be around 20-25MB/sec.

So what is the network speed?

Reply
0 Kudos
RParker
Immortal
Immortal

100MB = 10Mb/s

1000MB = 100Mb/s

Where are you getting that you will only achieve 25-30Mb/s?

Plus it depends on the tool used to transfer. WinSCP is much slower than native FTP server, and I can get burst in excess of 15Mb/s using Veeam on 100MB network.

Reply
0 Kudos
RParker
Immortal
Immortal

SAS drives, I can get 70-80 Mb/s write speeds....

If that's all you can get on that configuration, I think it's time for an upgrade. Even a Server could be used for backup, and the speed should definately be faster than 10mb/s.

Reply
0 Kudos
pauliew1978
Enthusiast
Enthusiast

Hi there again,

My network is fully gigabit with pretty much a dedicated gigabit link between the terastation and the esx server.Both are connected with gigabit and have not auto negotiated to slower speed.

thanks for all your help

Paul

Reply
0 Kudos
kharbin
Commander
Commander

Actually wire speed is: 100Mb = 11Mb/sec, and theorecticaly 1000Mb = 110MB/sec. But I have not had a server capable of hadling a sustained data stream of 110MB/sec from NIC to hard drive. That's why I say theoretical since I can actually say I have seen it.

As for where do I get the 20-25MB/sec speed on 1000Mb? When running in the service console, you will never achieve wire speed on a 1000Mb NIC. The console is "time sliced", so it can't get enough sustained CPU resources to process data at this speed.

And yes, it does depend on the tool used. For an uncompressed datastream WinSCP is just slow, FTP is the fastest. If you are getting 15MB/sec on a 100Mb net, then the Veeam utility is compressing the data as it is sent, thus being able to exceed the 11MB/sec limitaion. To send 15MB/sec with out compression would defy the laws of physics and therefor not possible.

Reply
0 Kudos
kharbin
Commander
Commander

Paulie,

Gigabit dedicated network should be fine. So the problem must either be in the COS or the Terrastation.

Best way to verify is to configure another host on this segment to allow ftp connections. Then point your scripts there and do a test backup of a single VM. Still slow?

Also, is the Terrastation set on RAID5? (slowest of all RAID levels).

Another thing is to try swapping out the patch cables. Bad cables will cause slow transfers.

my 2 cents

Reply
0 Kudos