VMware Cloud Community
Ken_Cline
Champion
Champion
Jump to solution

Is VMotion protocol routable?

Under ESX 2, the VMotion protocol used broadcasts to locate other hosts with which to communicate. This use of broadcasts required that all hosts within a given "VMotion domain" were in the same broadcast domain, essentially making VMotion a non-routable protocol (at least during session initiation).

Does the same hold true for a VI-3 environment?

Ken Cline VMware vExpert 2009 VMware Communities User Moderator Blogging at: http://KensVirtualReality.wordpress.com/
Reply
0 Kudos
1 Solution

Accepted Solutions
jasonboche
Immortal
Immortal
Jump to solution

VMotion across a router to a different subnet completed successfully. Although, it ran slowly at first because I forgot that I was packet shaping on the router to simulate a T1 WAN link. So basically when I started, I was VMotioning over a 1.544Mbps T1 :smileyshocked:

Jas

VCDX3 #34, VCDX4, VCDX5, VCAP4-DCA #14, VCAP4-DCD #35, VCAP5-DCD, VCPx4, vEXPERTx4, MCSEx3, MCSAx2, MCP, CCAx2, A+

View solution in original post

Reply
0 Kudos
17 Replies
Texiwill
Leadership
Leadership
Jump to solution

Hi Ken,

vMotion has not changed to allow it to be routable. I hope one day it will be... But I think the performance would be a major issue and the VMs would start to complain.

Best regards,

Edward Haletky

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
jasonboche
Immortal
Immortal
Jump to solution

The recommended default gateway for VMotion in ESX2 was the same as the VMotion IP address.

In ESX3 now, we are to use the actual default gateway (if applicable) for the VMotion subnet. Since we're configuring values for a VMKernel switch, I would bet VMotion could be routed through the gateway, much the way other VMKernel traffic can be routed such as ISCSI or NFS.

This would be pretty easy to set up and test if anyone wants to put the hour or so into it. My lab is a routed network with multiple subnets so if I can get away from the Citrix project I'm buried in, I'll give it a whirl and let you know what I find out Ken.

VCDX3 #34, VCDX4, VCDX5, VCAP4-DCA #14, VCAP4-DCD #35, VCAP5-DCD, VCPx4, vEXPERTx4, MCSEx3, MCSAx2, MCP, CCAx2, A+
Texiwill
Leadership
Leadership
Jump to solution

Hello,

Since vMotion uses its own protocol, the router would have to understand that protocol. I have not heard any do. When a vMotion occurs the iSCSI/NAS network is not really being accessed.... Still I agree, it seems like it could work, but would it still be fast enough? Please post the results of your test. I do not have the network capacity here to run my own yet.

Best regards,

Edward

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
Ken_Cline
Champion
Champion
Jump to solution

Hi Edward!

Long time no see. That's what I thought, but I've not had a reason to test it and the question came up on the internal PDL, so I thought I'd ask here.

Good to hear from you!

KLC

Ken Cline VMware vExpert 2009 VMware Communities User Moderator Blogging at: http://KensVirtualReality.wordpress.com/
Reply
0 Kudos
Ken_Cline
Champion
Champion
Jump to solution

Jason,

If you do get the chance to play with it, I would appreciate it. My "lab" is much more basic...

Thanks!

Ken Cline VMware vExpert 2009 VMware Communities User Moderator Blogging at: http://KensVirtualReality.wordpress.com/
Reply
0 Kudos
jasonboche
Immortal
Immortal
Jump to solution

Ok.. maybe tonight. Home lab is routable also.

VCDX3 #34, VCDX4, VCDX5, VCAP4-DCA #14, VCAP4-DCD #35, VCAP5-DCD, VCPx4, vEXPERTx4, MCSEx3, MCSAx2, MCP, CCAx2, A+
Reply
0 Kudos
jasonboche
Immortal
Immortal
Jump to solution

VMotion across a router to a different subnet completed successfully. Although, it ran slowly at first because I forgot that I was packet shaping on the router to simulate a T1 WAN link. So basically when I started, I was VMotioning over a 1.544Mbps T1 :smileyshocked:

Jas

VCDX3 #34, VCDX4, VCDX5, VCAP4-DCA #14, VCAP4-DCD #35, VCAP5-DCD, VCPx4, vEXPERTx4, MCSEx3, MCSAx2, MCP, CCAx2, A+
Reply
0 Kudos
VirtualKenneth
Virtuoso
Virtuoso
Jump to solution

Yep I've managed and tested this before, I didn't notice any speed differences however. Hmmmm

Reply
0 Kudos
stuart_ling
Contributor
Contributor
Jump to solution

One of the issues I can see is that you would be adding more complexity to the process which could make troubleshooting vMotion problems much more difficult. This is certainly not something to be put in without a fair bit of planning.

Reply
0 Kudos
Texiwill
Leadership
Leadership
Jump to solution

Hello,

That is a very nice change! Can you give us the specifics? What subnet to what subnet? Where were the netmasks involved? When you setup the default gateway for each side, how were they setup?

How far apart where the subnets?

Even so, I am not sure I would do this on anything that was not a dedicated link. I can just see people trying to vMotion over a non-dedicated link. Remember vMotion is unencrypted so the memory footprint is available to a man in the middle attack.

Best regards,

Edward

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
jasonboche
Immortal
Immortal
Jump to solution

Yep I've managed and tested this before, I didn't

notice any speed differences however. Hmmmm

Hi Vliegenmepper

I had the performance issue because my router bandwidth was capped at 1.544Mbps incoming and outgoing in order to simulate a WAN link in the lab.

Everything else being equal, I would expect there would be some decrease in speed due to the simple fact that an extra hop and piece of equipment has been added in between the path of the VMotion. When you boil it down, the router needs to process the flurry of frames that pass through it which will add some degree of overhead, whether measurable or not.

VCDX3 #34, VCDX4, VCDX5, VCAP4-DCA #14, VCAP4-DCD #35, VCAP5-DCD, VCPx4, vEXPERTx4, MCSEx3, MCSAx2, MCP, CCAx2, A+
Reply
0 Kudos
VirtualKenneth
Virtuoso
Virtuoso
Jump to solution

Hi Jason,

Clear to me now, I agree you'll loose some performance all depending on your network setup.

I just tested with an extra router in the core switch. VMotioned from 192.168.1.0 to 10.0.0.0 subnet. It was more to see if it worked or not.

Reply
0 Kudos
Ken_Cline
Champion
Champion
Jump to solution

Hey guys,

Thanks for taking the time to test this out for me. I really appreciate your efforts and your willingness to share the results!

Thanks again,

KLC

Ken Cline VMware vExpert 2009 VMware Communities User Moderator Blogging at: http://KensVirtualReality.wordpress.com/
Reply
0 Kudos
sbeaver
Leadership
Leadership
Jump to solution

Brings up some interesting possibilities. I wonder how creative people might get

Steve Beaver
VMware Communities User Moderator
VMware vExpert 2009 - 2020
VMware NSX vExpert - 2019 - 2020
====
Co-Author of "VMware ESX Essentials in the Virtual Data Center"
(ISBN:1420070274) from Auerbach
Come check out my blog: [www.virtualizationpractice.com/blog|http://www.virtualizationpractice.com/blog/]
Come follow me on twitter http://www.twitter.com/sbeaver

**The Cloud is a journey, not a project.**
Reply
0 Kudos
jasonboche
Immortal
Immortal
Jump to solution

Hello,

That is a very nice change! Can you give us the

specifics? What subnet to what subnet? Where were the

netmasks involved? When you setup the default gateway

for each side, how were they setup?

Host 1 vmotion subnet 192.168.110.0/24

Host 2 vmotion subnet 172.16.0.0/24

How far apart where the subnets?

About 3/4" of an inch.

To make things a little more interesting, my two IP routers are both VMs running on the ESX hosts. So the VMotion traffic was actually flowing through a VM running router software to forward the packets to the other ESX host.

Even so, I am not sure I would do this on anything

that was not a dedicated link. I can just see people

trying to vMotion over a non-dedicated link. Remember

vMotion is unencrypted so the memory footprint is

available to a man in the middle attack.

That makes two of us Smiley Happy

One of the goofy things I found out while changing the VMKernel IP, SM, and DG, you have to do it in two steps. First change the IP and SM. Apply those changes. Then it will allow you to change the DG. If you try to change the IP, SM, and DG all in one step, the DG config will fail because the IP and SM hasn't been applied and it will sqwak about the DG not being on the same subnet as the VMKernel IP so it won't allow the change to take place. Trademark VMware quirks. You gotta dig to find 'em, but they are there.

VCDX3 #34, VCDX4, VCDX5, VCAP4-DCA #14, VCAP4-DCD #35, VCAP5-DCD, VCPx4, vEXPERTx4, MCSEx3, MCSAx2, MCP, CCAx2, A+
Reply
0 Kudos
jasonboche
Immortal
Immortal
Jump to solution

Hey guys,

Thanks for taking the time to test this out for me. I

really appreciate your efforts and your willingness

to share the results!

Thanks again,

KLC

Thanks Ken. That's one of the harder 10 points I've worked for.

Word to the wise if anyone tries changing their VMKernel subnet and they are using swISCSI, you'll lose all access to your ISCSI storage immediately when you make the change since you're breaking the VMware rule that says the ISCSI (VMKernel) port must be on the same subnet as the COS. Doesn't matter if you're using CHAP authentication or not. During my tests, I had about 10 VMs lose their ISCSI storage which killed the VMs. Luckily when my testing was done after I put everything back in to place the VMs booted back up and I haven't seen any ill results. These weren't PROD or DEV VMs at work. These were PROD and DEV VMs at home.

VCDX3 #34, VCDX4, VCDX5, VCAP4-DCA #14, VCAP4-DCD #35, VCAP5-DCD, VCPx4, vEXPERTx4, MCSEx3, MCSAx2, MCP, CCAx2, A+
Reply
0 Kudos
RobMarson
Contributor
Contributor
Jump to solution

Our company provides a network appliance that enables server and storage extensions over any type of wide area network, including IP networks. The product enables a virtual wire over which any protocol can be extended over any network. We recently deployed a solution with a customer to enable live server migrations over a public IP WAN using VMotion. Our virtual also enables jumbo frame MTU transparency, bulk data encryption, lossless data compression (achieving 5x compression for VMs), and wide area packet loss immunity. Further information is available from our website.

http://www.aforesolutions.com/ase_platform.php

http://www.aforesolutions.com/ase_resources.php

Reply
0 Kudos