Moving VMs from one ESXi server to another - what a pain!!!

[ShahidSheikh]

This is where I miss my setup with VMware Server. I do not have a iSCSI or NFS central storage. Before I started the move to ESXi, all my VMware Server (1.0.7) machines were Ubuntu 7.0.4 servers all of which had the directories where VMs were stored exported through NFS and cross mounted on each of the servers. So on each server I had the directory structure:

/vmfs/svm01

/vmfs/svm02

/vmfs/svm03

/vmfs/svm04

Where only one directory was local and the remaining three were NFS mounts from the three other servers. This made moving moving VMs around a breeze. I could even run vmware-vdiskmanager with input and output VMs being on different servers. The only thing that was slow was provisioning thick vmdks.

Now under ESXi moving VMs is a pain. Apparently scp in ESXi does not support the -C (compression) flag. Through scp, I can either scp thick vmdks from one ESXi to another or import them into thin vmdks and then scp them. Either way its a painfully slow process. And scp only gives a maximum thruput of about 2.5 MBytes/sec (20 Mbps) on my 1 GB connected NICs (VM nic is separate from management nic). All servers are either Dell (2850, 2950) or HP 2U (380 G4) servers, all with 4 or more U320 SCSI drives in a hardware RAID.

I was hoping in RCLI I would be able to run vmkfstools.pl script across machines but alas, it only runs vmkfstools operations on a single machine too.

Right now I am trying to move one of my VMs that has a 120 GB vmdk and it says its going to take 15 hours to move. I can reduce the server downtime significantly (I think) by taking a snapshot and moving that first, then downing the VM and moving the deltas but still all of this for doing something I could do fairly painlessly in VMware Server seems a lot of work.

My question is that does anyone have a better way of moving VMs from one ESXi to another? I will eventually have a centralized NFS store but that is not going to happen for another 2 months.

It would be very cool if there was a way to NFS export the local datastores in ESXi.

[TechFan]

Ah, so the tech support mode is actually a VM with limited resources. . .if that is what you are saying with ESXi.

Yes, that is what I intend to do. . .I was just trying to figure out a way to only transfer changes instead of the entire VM each time. . .any way I look at it, it seems the entire file will be transferred. Because when rsync runs on two local (NFS mounts included) it just does complete copies. . .and I can't have rsync actually running on the NetApp. We are using the NetApp for NFS. . .provides NAS/SAN functionality in many different ways.

[ShahidSheikh]

That is not unique to ESXi. ESX is the same way. ESX just has a more rich flavor of Linux installed in the console.

It might be worth a try to see how rsync performs the way I suggested. I think it will perform better than you expect because you are not rsync-ing to an NFS mount. Meaning that your target (across the WAN) is not NFS. Its the rsync server on the remote host.

Your source is an NFS mounted FS. And yes you will take a hit there because you have to read the entire file from the NFS volume to determine what is different between it and the one that is on the remote rsync machine. But at least you are not doing that across the WAN. Across the WAN you should only be sending the deltas.

Also I thought you mentioned that you are already using snapshots. Why not use the snapshots to ship the deltas off-site. It would be like a manual mechanism of doing incremental backups. Personally, I am not confident in the snapshot mechanism to the point that I use it as a means of backup for production systems.

[s1xth]

I know this was posted a few posts back but just in case some of the newer posters didnt see, but Veeam will be releasing FASTSCP for ESXi in early January...if you go to their forums you will see the thread, they are doing PRIVATE beta testing right now. I am sure this will prove to be a great tool for us ESXi guys. Just keep your eyes open for it, hopefully they release it soon!

[TechFan]

Thanks. Yes. I just hadn't assumed that the tech support mode console was implemented the same way, but it would make sense.

I am now planning to do "almost" what you suggested. . .but I will just have to do a full copy to the hosted VM. . .then use rsync to get the network efficient transfer to offsite. . .I am doing that now by doing rsync -e ssh. . .which should work for the offsite transfer.

As I said, there are two purposes. . .protect us from failure of our SAN and get offsite backups.

When I mentioned snapshots. . .I am talking about the snapshotting in the NetApp using it special WAFL instant snapshotting. We do not use VM snapshotting except during testing.

Also. . .yes I have been watching and waiting for the new FastSCP. I didn't make it into the beta, but will see how that works or fits into a solution. . .problem is we are going to want this scheduled. . .

Anyway, thanks again. . .btw is running rsync to a rsync daemon faster than using the ssh method?

[ShahidSheikh]

I have not done extensive tests but I do believe rsync to a remote rsync daemon is faster than rsync over ssh. It does not have the encryption overhead of ssh. SSH is just the more common way of doing it because its convenient and secure.

[CKitz]

The FastSCP 3.0beta is out. I registered here and was able to download it right away.

The performance isn't that much better than regular scp though. Usually SCP gets me 2.5 - 3.2MB/s where this gets 4MB/s. My client is on Gigabit and my server is on 100Base-T.

[ShahidSheikh]

The conslusion that I came to was with very limited reqources the best way to do this is to stick a 1 TB HD in a workstation and use it as an NFS export. If you are using Windows XP then you can use Services for Unix (SFU). performance is pretty reasonable with SFU.

Mount the NFS export as a datastore on both source and destination ESXi. Then you can either write a script to do the move for you or you can do it manually from SSH or VIC. Taking a snapshot and moving the snapshot first, then shutting down the VM and moving the delta can reduce downtime of the VM. Though you may have to do that from SSH.

Been able to utilize the gigabit link between ESXi and the XP machine to 200 mbps while copying that way.

[ShahidSheikh]

Should also mention that at another client I used a similar scheme to backup VMs. They are strictly a windows shop and all their ESXi datastores are local RAID arrays on the servers themselves. I took the Windows XP machine that was running BackupExec with a 24 tape LTO-2 library connected to it, stuck a 1 TB HD in there and installed SFU on it. The ESXi machines dump a snapshot of the VMs on the 1 TB HD and BackupExec backs it up to tape.

[s1xth]

Did you write a script for the snapshot part of the backup procedure? Just curious.

[RenaudL]

Here's a low-level explanation for the poor scp performance on ESXi:

When a ssh client & server start a file transfer, they negotiate behind-the-scene for the actual protocol to use for the copy: SCP or sFTP. SCP is the old legacy protocol (which the "scp" command name historically comes from), but sFTP is the protocol usually chosen if both peers support it (which is the case for most heavyweight ssh suites).

The ssh suite used in ESXi is dropbear which is perfect for embedded environments, but unfortunately doesn't support sFTP. Any scp transfer from/to ESXi will therefore rely on the SCP protocol, which has an extremely constraining window limitation of 2 I/O blocks (IIRC). This tiny window cripples the throughput and explains the bad numbers you all get.

We don't support file transfers over scp, but we do provide other efficient & supported tools to perform the same kind of operations. Don't ask me which, I'm not a GUI guy

[s1xth]

So what do you recommend for file transfers then? What other tools do you recommend using instead?

Just curious what does vmotion use? Or I should say, how does vmotion move the vm's faster?

[ShahidSheikh]

Did you write a script for the snapshot part of the backup procedure? Just curious.

Not yet but I plan to.

[ShahidSheikh]

Just curious what does vmotion use? Or I should say, how does vmotion move the vm's faster?

VMotion does not move the vmdk files. The vmdk files have to be on a centralized storage that is the datastore of both source and target ESX/ESXi.

[s1xth]

Damn. I cant believe I never knew that. So if you have a bunch of ESX/ESXi servers that are using local storage, and you have VMOTION it wont work correct? You need to have your hosts on a SAN?

[mcowger]

Correct - the option wont be available.

--Matt

[R-Sommer]

I did a lot of tests and realized scp and rsync are rather slow, even rsync between to local SATA disks (about 6 MB/s IIRC). But using cp for all none vmdk files and vmkfstools -i for all vmdk files to copy them locally is acceptable for me (cp about 18 MB/s, vmkfstools about 35 MB/s). Therefore I'm pretty sure, that the limitation is on cpu or s.th. like this and not on network interface.

The next time I will make some test using iSCSI and NAS in comparison with local speed.

[devzero]

2. Enable unsupported sshd to your ESXi system.

3. Place this 'rsync-static-stripped' binary into /bin or somewhere else in your path on your ESXi system.

what about running rsync in daemon mode instead ? (-> http://rsync.samba.org/ftp/rsync/rsyncd.conf.html )

that should speed up things a LOT as the data doesn´t need to go trough sshd.....

[kpc]

I'm on annual leave till the 31st March. For urgent requests please contact Tony Ghirardi or Gavin Purnell.

--

This email and any accompanying document(s) contain information from Kent Police, which is confidential or privileged.

The information is intended to be for the exclusive use of the individual(s) or bodies to whom it is addressed.

If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited.

If you have received this email in error, please notify us immediately by contacting the sender or telephoning 01622 690690.

The copyright in the contents of this email and any enclosure is the property of Kent Police and any unauthorised reproduction or disclosure is contrary to the provisions of the Copyright Designs and Patents Act 1998.

[TechFan]

How would you run rsync in daemon mode on an ESXi box? Rsync isn't

there. . .

[devzero]

sure - if you don`t copy it there, it won`t be there. as any other 3rd party tool.....