Copying 700 GB of VMs directly to an ESX 4.1 Datastore

[radamanthe]

Hello Community,

We have a bunch of VMs (~700 GB in total) stored on a  1TB USB drive and we'd like to copy them directly to an ESX 4.1  Datastore (located on an EMC Clariion CX4-120).

We don't want to transfer them over the LAN - not an acceptable solution.

However, the Service Console of ESX 4.1 does not seem  to allow USB drives to be mounted (I've searched the Internet, asked  some VMware contacts, does not seem to be implemented/supported any  more).

Possible workarounds:

1) We have installed a W2K8 R2 physical server that has access to the SAN, however, apparently, there is no read + write drivers for VMFS volumes (datastores) for Windows

2) Maybe reinstalling this physical box with a Linux that has read + write drivers for VMFS volumes ?

3) Reinstalling the physical box with VMware ESX 3.x whose Service Console  allows USB drive mounting ? But will it be able to see the Datastores  created by ESX 4.1 ?

Thanks for your valuable inputs.

Cyril

[Martin83]

Hi Cyril,

is it technical not possible to transfer it over lan ?

In my opinion you need a lot more time to find and execute a workaround then easily copy it over lan.

Best regards

Martin

[ProPenguin]

What kind of server do you have?  What type of hard drives can you put in it?  Also what was your approach to mounting the hard drive on the ESX 4.1?

[idle-jam]

mount to a VM via USB passthru and have it upload via browse datastore from vSphere client. The network would only be contained within the virtual switch only. i believe this would one of the easiest approach ..

[radamanthe]

Hi idle-jam,

That's a nice one.

We'll try the ESX 3.5 workaround first and if we encounter other difficulties, then we'll try the USB passthru + vSphere Client Datastore transfer contained in vSwitch.

I'll let you know.

Thanks for your answers,

Cyril

[idle-jam]

hope to hear from you soon. btw do you mind sharing why is there a restriction of transferring it over the LAN?

[Rumple]

I think you are going to find that USB passthrough is going to royally suck performance wise as I am sure I read somewhere is it USB 1.1 (12MB/sec) and personally I would expect you to purple screen your host because 700GB copying through a passthough will probably fall over at some point and in the past when a VM or host loses that connection it panics.

could be worn,g but I can transfer across the lan much faster then 12MB/sec...without risk of killing my host...

[radamanthe]

Hi idle-jam,

Restriction regarding LAN is due to the fact that this is a test environment and we prefer avoiding any impact on the LAN which is shared by real production applications (I don't think there's any kind of QoS to limit certain types of transfer).

And even if there is no impact, we don't have time to explain / justify it towards Network team, so we prefer the "isolated" way which could also be the quickest one.

@Rumple: thanks for that USB passthru risk analysis, we'll then focus on our ESX 3.5 workaround.

Cheers,

Cyril

[admin]

seriously? your concerned about the throughput impact? a 100mb lan would handle that without problems unless your using 10/100 non-switched hubs. heck use scp and throttle the connection then. sounds like a pretty poorly run network environment. no offense.

[idle-jam]

you can use the above advise and throttle it. worse case let it run when you're off work, it should be completed the next day when you have come back to work.  another way is to create a dedicated vswitch attached back to back to the NAS/Laptop. you can do a direct transfer then.

[bulletprooffool]

Grab a copy of Veeam's FastSCP, kick of the copy at end of play one evening and don't stress about.

700GB is not really that much data.

[radamanthe]

Sounds interesting. I downloaded it and will give it a try since our ESX 3.5 USB mounting option does not work (no sign of USB device in dmesg).

And we'll see what the Network Admins will say about the data transfer after it happens, if they only see it.

Thanks for that.

Cheers,

Cyril

[ProPenguin]

Try pulling the harddrive out of the enclosure hooking it up to a computer via IDE cable as the seconddary hard drive.  Then connect crossover cable from that computer and the ESX host.  Push the files through.

[Rumple]

Hell of a lot of work to copy data when any user in the company can easily just go the same thing without getting approval from the network team (nor impacting production).

[bulletprooffool · Accepted solution]

No probs Cyril - easy to use - you should be fine!

[radamanthe]

Hi guys,

Just to let you know, copy over the network with Veeam FastSCP has done a good job : 700 GB transferred in one night and a few hours, no news from Network team so far, I guess it was transparent.

Thanks for your answers.

Cheers,

Cyril

[idle-jam]

wondering how many hours is it? would be keen to know how good fastscp is in real life.

[Josh26]

radamanthe wrote:

Hi guys,

Just to let you know, copy over the network with Veeam FastSCP has done a good job : 700 GB transferred in one night and a few hours, no news from Network team so far, I guess it was transparent.

Thanks for your answers.

Cheers,

Cyril

I cannot imagine what a network team would be doing to actually notice someone copying a lot of data and feel they have to step in. It's not like you're sending it up a DSL service.

[radamanthe]

I had someone taking care of it and I didn't ask for exact stats, but I can remember the test I run had an average transfer rate of 12 MB/s, so (700*1024)/12/60² ~ 16,6 hours, which is closer to "2 nights", but maybe the transfer rate was better during the night.

But I guess it depends on the Network, especially if you have QoS implemented and if traffic on ports 2500-5000 (FastSCP port range, customizable) is not prioritized.

Cheers,

Cyril

[idle-jam]

okay, thanks for sharing.