Vmotion architecture

[vok]

Hello,

I've been playing with Vmotion and have been trying to better undertand how it works. If I have a VM on ESX1 and I am migrating to ESX2, is there communitcations between ESX1 and ESX2? As in data/transtition states/etc...?

Thanks

Khanh

[kooltechies]

Hi ,

Yes there is a Data transfer between the two hosts but that data transfer is not your disk data as the disk data need not be transfered because they are already on the shared datastore. Only the data that gets transferred is the memory Bitmap of the VM being transferred from source to destination.

All this transmission happens through the vmotion vmkernel interface that you create on both the hosts. At around 95 % of transfer you will get some ping timeouts also as at that time it copies the dirty memory (writes in memory to the destination vmdk) then it deletes the entry of vm's .vmx file from the source and registers that .vmx with the destination.

Thanks,

Samir

P.S : if you think this is a helpful answer please consider rewarding points

[jeremypage]

Yes, basically ESX takes a snapshot of your working memory and copies it to the machine you are targeting to move your VM to. After the initial snapshot is moved over it does differential copies until it's close enough to "freeze" it and start running on the new location. So the speed of your connection between the two ESX hosts is important. Vmotion uses it's own physical network card as a best practice.

[vok]

One thing that I didn't mentioned is that I am running a network tool that captures all communications between ESX1,ESX2, VMs, and NAS. From what you guys are saying is that there are communications between the two ESX Servers and if that is the case I should see that in my logs?

Or is the communication between ESX1 ---> VM Infrastrucre --> ESX2? or ESX1 -

---

> ESX2 ?

Thanks

Khanh

[weinstein5]

The communication is between the two ESX servers - if you look at the configuration of you two esx servers you will see a vmkernel port enabled for vmotion and assigned an IP address on both esx servers - it is through these ports that communication occurs -

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[kooltechies]

Hi ,

Another place for looking the log files in esx will be /varlog/vmware/hostd.log on both source and destination. You can also look for vmotion related thing in /proc/vmware/migration directory structure.

Thanks,

Samir

[vok]

Thanks for the help everyone.

On a different subject, is there any tunning parameters for VMotion if I am on a slower network (100Mps)? Something like a long timeout setting?

Thanks

Khanh

[kooltechies]

Hi Khanh,

There are two things to consider 100mbps is not a very bad speed for VMotion.

1) Do not do multiple vmotions at the same time.

2) When you do the vmotion , you need to set priority as lower so that it can do it more smoothly.

Thanks,

Samir

[vok]

Thanks for the info.

I'm trying to see if there are any limitations of Vmotion. Does anybody

know how slow the network can be and still have Vmotion perform the

migration successfully?

I tried doing a migration at T1 speed but it dies at 90%. I was able to migrate at 100Mbps.

Thanks

Khanh

[Dave_Mishchenko]

You can go pretty slow and still have vmotion succeed. Typically if you fail at 90% it indicates a problem with the vmkernel setup on the hosts. Where you routing your traffic when you did the T1 speed test?

tried doing a migration at T1 speed but it dies at 90%.

[mudha]

while doing vmotion if its going on and let sat it is at 70% and the ESX host one that's ESX1 goes down and wht will be the status

[RParker]

There are two things to consider 100mbps is not a very bad speed for VMotion.

I am pretty sure it's in the documentation, that GIG network is required for VMotion to function... That's a minimum. Maybe if you fake the NIC somehow into thinking it's connected to a Gig port, and then it's actually 100 Mbs speed maybe, but otherwise vmkernel won't let you configure vmotion on a NIC that's not a Gig (1000) connection.

[weinstein5]

Gig is not required it is recommended - it provides the fastes transport to copy system state memory so less chance of someone noticing - I have been successful with 100 MB and 10 MB it just took longer with moredropped packets -

[admin]

GigE is required for production support. It often works at slower speeds, but (a) the memory mutation rate may be higher than achievable network bandwidth, and (b) the longer freeze time when the final state is transferred may cause timeouts within the guest os and its applications. In 2009 it's unusual to find 10Mb networking in environments that also meet VMotion's shared storage requirements. Some 100Mb remains, but we see it less and less, too.

[vok]

Correct,

I use IPNetSim to slow the network down for testing.

[vok]

Yes I was routing traffic. I was doing database inserts.

[Dave_Mishchenko]

You'd want to check your setup again and from the console you can run vmkping to check that the vmkernel port has IP connectivity to the other host.

[vok]

Base on what you said regarding the memory multation, have you ever faild on a migration when on a 1Giga bit line? I'm trying to test the multation theory by running a java app that does a lot of floating point calulations and arraylist assignments.

Looks like migration works all the time. By the way, I've set my VM to 4gigs, and my test pushes the CPU to 100% and100% Memory, but the migration seems work all the time.

Thanks

Khanh

[admin]

There are many reasons that a VMotion might not complete. In normal operation it has been very reliable according to VMware customers that have had it in production. Even if a VMotion doesn't complete for some reason, it just means that the VM remains running on the original host. If your Java tests simulate worst-case conditions for your environment and VMotion continues to work then you should feel confident that it will meet your needs.