VMware Cloud Community
Luca82
Enthusiast
Enthusiast
Jump to solution

Which network uses svMotion?

Hi everyone, I have a question about svmotion network. In the guide I see that the vmk "provisioning network" is used for cold migration, so I'm trying to understand in wich scenario:

- cold migration between 2 hosts and between their local datastore (from local to local): does it use the "provisioning network"?

- hot migration (shared nothing) between 2 hosts and between their local datastore (from local to local): does it use the "vmotion network" for both ram and disk?

- if I cold/hot migrate a virtual disk between two storage (same host) there is no neet to use a network....am I right?

can someone confirm or clarify that?

1 Solution

Accepted Solutions
MattGoddard
Enthusiast
Enthusiast
Jump to solution

Thanks for the screenshot. I see what you mean now.

So remember when I said that a "cold" migration (i.e. svMotion) in your setup - that is, a setup that uses IP-based storage - will move the data over the host management network by default? Well, if you create a new vmkernel, assign it an IP address, and then click that "Provisioning traffic" box, it will use that network for svMotion and not the management network.

[Side note: It's a little silly for the management network settings to have a "Provisioning traffic" check box since, if no other vmkernel has "Provisioning traffic" checked, svMotion will use the management network regardless of whether you check that box or not. Pretty confusing!]

So in other words, all those check boxes in your screenshot are there to enable you to separate different kinds of traffic over different vmkernel networks. If you create a new vmkernel network and click the "Provisioning traffic" check box and nothing else then "cold" migrations - and only "cold" migrations - will use that network.

I'll answer your bullet point questions directly:

vMotion traffic: if enabled, is used to move RAM between hosts in a Hot Migration (vMotion); if disabled, vMotion doesn't work

Correct. Unlike with svMotion, the "vMotion traffic" box must be checked on at least one vmkernel network, otherwise vMotion won't function and therefore "hot" migrations won't work.

shared nothing vMotion: if the virtual machine disk is on local datastore, we need to vMotion+svMotion: RAM is moved on vMotion traffic network; while the disk is moved on Management Network

Correct by default. Create a separate vmkernel and click "Provisioning traffic" to move VM disk data on that network instead of the management network. (Or, if you already have a separate vMotion vmkernel, you could just check the "Provisioning traffic" box on that one, so that you're running both "hot" and "cold" migrations across that network.)

If the Virtual Machine is powered off (cold migration) the vMotion traffic network is never used

Strictly speaking, that's not accurate; it depends. Imo, it's easier to think of it this way: with IP-based storage, a "cold" migration will use the management vmkernel network by default, as noted above, or some other network if you've clicked the "Provisioning traffic" box on it. In either case, that network might also be the vMotion network. It depends where you've clicked those "vMotion traffic" and "Provisioning traffic" boxes. In very simple setups, it's common to have only a single vmkernel - the management network - and to check the "vMotion traffic" box on it. In that scenario, all traffic - "hot", "cold", management, VM - will run over that network.

if cold migrate between 2 hosts a poff vm and storage is shared between hosts: nothing to move between hosts: no network used

This is only correct if the hosts and shared storage are all VAAI-capable. If VAAI is not available and you're using IP-based shared storage (e.g. NFS, iSCSI), a "cold" migration is going to involve the host pulling all of the VM's disk data up from [source datastore] via the management network (by default) or whichever network has "Provisioning traffic" enabled, and then sending it back the same way to [destination datastore], even though both datastores live on the same storage device. (Yes, this is horribly inefficient!)

if cold migrate between 2 hosts a poff vm and vmdk is on local datastore:

---- if there is the Provisioning traffic enabled it is used to move vmdk between hosts

---- if the Provisioning traffic is not enabled, it is used the management network to move vmdk between hosts

Correct.

This is what I have understood.... but I don't find any official/clear documentation about it

Honestly, I sympathize. The official documentation isn't very clear on this stuff and, imo, VMware makes it more confusing by talking about "hot" and "cold" migrations. I find it much easier to focus on the technologies involved - i.e. vMotion and svMotion - what they're for and how they specifically work. It makes a lot more sense to me that way.

---------------------------------------------------------------------------------------------------------

Was it helpful? Let us know by completing this short survey here.

View solution in original post

11 Replies
daphnissov
Immortal
Immortal
Jump to solution

All of these questions I have answered in my book you can read for free in the second link in my signature.

Luca82
Enthusiast
Enthusiast
Jump to solution

Thank you, your guide is very helpful... but I cannot find any information about the "provisioning network" (it is not mentioned anywhere) so I can't understand when it is used.

0 Kudos
sjesse
Leadership
Leadership
Jump to solution

Look at VMkernel Networking Layer

Provisioning TCP/IP stack

Supports the traffic for virtual machine cold migration, cloning, and snapshot migration. You can use the provisioning TCP/IP to handle Network File Copy (NFC) traffic during long-distance vMotion. NFC provides a file-specific FTP service for vSphere. ESXi uses NFC for copying and moving data between datastores. VMkernel adapters configured with the provisioning TCP/IP stack handle the traffic from cloning the virtual disks of the migrated virtual machines in long-distance vMotion. By using the provisioning TCP/IP stack, you can isolate the traffic from the cloning operations on a separate gateway. After you configure a VMkernel adapter with the provisioning TCP/IP stack, all adapters on the default TCP/IP stack are disabled for the Provisioning traffic.

0 Kudos
daphnissov
Immortal
Immortal
Jump to solution

0 Kudos
Luca82
Enthusiast
Enthusiast
Jump to solution

Sorry, but I still not understand. If I svMotion or migrate a vm folder/disk (hot or cold migration) there is no need to use any network (no management, no vmotion, no provisioning) because the data is moved between storage through ESX or not (VAAI or Data Mover)...but without any need to use network.

The only scenario that comes to my mind in which there is the need to move data between hosts is vmotion for the virtual machine ram (ok, now it uses vmotion network) and virtual machine disk just only if it is hosted on ESX local datastore. Therefore if I move a virtual machine between hosts and its disk is on local datastore, I assume that is used the "vmotion network" for both ram and disk if vm is powered on,  whilest it uses "management network" for folder/disk in case of the vm is powered off. The "provisioning network" is used in opposite to "management network" only in the latter scenario.

0 Kudos
sjesse
Leadership
Leadership
Jump to solution

I was only referring to the provisioning network. Storage vmotion doesn't need any network, unless your doing storage vmotion across non shared storage.

0 Kudos
MattGoddard
Enthusiast
Enthusiast
Jump to solution

By default, an ESXi host uses its default TCP/IP stack for all TCP/IP communication, which includes vMotion of VM compute/memory data and may also (but does not always) include svMotion of VM disk data. The only reason you'd use the provisioning or vMotion TCP/IP stacks is if you need to completely isolate that traffic for some reason. This is relatively unusual; most people just use the default stack and segment traffic using different VLANs/subnets. Since your setup appears to be a pair of hosts using local storage, I can't see a reason for you to use the provisioning or vMotion stacks.

One key thing to remember: vMotion and svMotion are two different things that work in different ways. vMotion moves the VM's compute/memory data between two hosts over TCP/IP, typically using a dedicated vMotion vmkernel port with its own IP address. svMotion moves the VM's disk data between two datastores. In your case, since those two datastores are on two different local disks on two ESXi hosts, svMotion moves that data over TCP/IP using the host's management network. In other setups - e.g. mine, where my hosts are connected to a SAN over Fiber Channel - svMotion doesn't use TCP/IP at all. Instead, the host on which the VM is registered either pulls its disk data up the fiber network from [source datastore] and then transmits it via the same network to [destination datastore] or, ideally, the host and the SAN are VAAI capable and therefore the host simply tells the SAN to move the data from [source datastore] to [destination datastore] itself.

Another thing to remember is that in your setup, a "cold" migration only involves svMotion (since there's no compute/memory data to move) whereas a "hot" migration always involves both vMotion and svMotion. In my Fiber Channel setup above, I can "hot" migrate the compute/memory without moving the VM's disk data, in which case only vMotion is used. I can also do a "cold" migration of the VM without moving the disk data. But really that's just another way of saying "unregister the VM from host A and register it on host B". vMotion and svMotion aren't involved there at all.

Luca82
Enthusiast
Enthusiast
Jump to solution

Thank you a lot for your answers, but still not understand...I am not talking about tcpstack..... I have perfectly clear what are vmotion/svmotion/shared nothing vmotion ecc... I think you don't understand my question. I am talking about this option:

2019-08-30 12_08_00-VMkernel_ vMotion and Provisioning in dedicated TCP_IP stacks - IT Gaiden.png

As I read it is used ONLY for cold migration (and other thing....but I want to focus on migration)

As I know since now, just to recap:

- vMotion traffic: if enabled, is used to move RAM between hosts in a Hot Migration (vMotion); if disabled, vMotion doesn't work

- shared nothing vMotion: if the virtual machine disk is on local datastore, we need to vMotion+svMotion: RAM is moved on vMotion traffic network; while the disk is moved on Management Network (is it correct?)

- If the Virtual Machine is powered off (cold migration) the vMotion traffic network is never used

- if cold migrate between 2 hosts a poff vm and storage is shared between hosts: nothing to move between hosts: no network used

- if cold migrate between 2 hosts a poff vm and vmdk is on local datastore:

---- if there is the Provisioning traffic enabled it is used to move vmdk between hosts

---- if the Provisioning traffic is not enabled, it is used the management network to move vmdk between hosts

This is what I have understood.... but I don't find any official/clear documentation about it

0 Kudos
MattGoddard
Enthusiast
Enthusiast
Jump to solution

Thanks for the screenshot. I see what you mean now.

So remember when I said that a "cold" migration (i.e. svMotion) in your setup - that is, a setup that uses IP-based storage - will move the data over the host management network by default? Well, if you create a new vmkernel, assign it an IP address, and then click that "Provisioning traffic" box, it will use that network for svMotion and not the management network.

[Side note: It's a little silly for the management network settings to have a "Provisioning traffic" check box since, if no other vmkernel has "Provisioning traffic" checked, svMotion will use the management network regardless of whether you check that box or not. Pretty confusing!]

So in other words, all those check boxes in your screenshot are there to enable you to separate different kinds of traffic over different vmkernel networks. If you create a new vmkernel network and click the "Provisioning traffic" check box and nothing else then "cold" migrations - and only "cold" migrations - will use that network.

I'll answer your bullet point questions directly:

vMotion traffic: if enabled, is used to move RAM between hosts in a Hot Migration (vMotion); if disabled, vMotion doesn't work

Correct. Unlike with svMotion, the "vMotion traffic" box must be checked on at least one vmkernel network, otherwise vMotion won't function and therefore "hot" migrations won't work.

shared nothing vMotion: if the virtual machine disk is on local datastore, we need to vMotion+svMotion: RAM is moved on vMotion traffic network; while the disk is moved on Management Network

Correct by default. Create a separate vmkernel and click "Provisioning traffic" to move VM disk data on that network instead of the management network. (Or, if you already have a separate vMotion vmkernel, you could just check the "Provisioning traffic" box on that one, so that you're running both "hot" and "cold" migrations across that network.)

If the Virtual Machine is powered off (cold migration) the vMotion traffic network is never used

Strictly speaking, that's not accurate; it depends. Imo, it's easier to think of it this way: with IP-based storage, a "cold" migration will use the management vmkernel network by default, as noted above, or some other network if you've clicked the "Provisioning traffic" box on it. In either case, that network might also be the vMotion network. It depends where you've clicked those "vMotion traffic" and "Provisioning traffic" boxes. In very simple setups, it's common to have only a single vmkernel - the management network - and to check the "vMotion traffic" box on it. In that scenario, all traffic - "hot", "cold", management, VM - will run over that network.

if cold migrate between 2 hosts a poff vm and storage is shared between hosts: nothing to move between hosts: no network used

This is only correct if the hosts and shared storage are all VAAI-capable. If VAAI is not available and you're using IP-based shared storage (e.g. NFS, iSCSI), a "cold" migration is going to involve the host pulling all of the VM's disk data up from [source datastore] via the management network (by default) or whichever network has "Provisioning traffic" enabled, and then sending it back the same way to [destination datastore], even though both datastores live on the same storage device. (Yes, this is horribly inefficient!)

if cold migrate between 2 hosts a poff vm and vmdk is on local datastore:

---- if there is the Provisioning traffic enabled it is used to move vmdk between hosts

---- if the Provisioning traffic is not enabled, it is used the management network to move vmdk between hosts

Correct.

This is what I have understood.... but I don't find any official/clear documentation about it

Honestly, I sympathize. The official documentation isn't very clear on this stuff and, imo, VMware makes it more confusing by talking about "hot" and "cold" migrations. I find it much easier to focus on the technologies involved - i.e. vMotion and svMotion - what they're for and how they specifically work. It makes a lot more sense to me that way.

---------------------------------------------------------------------------------------------------------

Was it helpful? Let us know by completing this short survey here.

MattGoddard
Enthusiast
Enthusiast
Jump to solution

Slight correction, because I just did some digging and learned something that I didn't know before! VMware defines actively used disks and snapshots as "hot" data and will therefore use vMotion to migrate them, not svMotion. That changes some of what I said above.

  • In the case of a powered on VM with one .vmdk file and no snapshots on a host with local storage, when you migrate it to another host it will only use vMotion.
  • If the VM is running from an active snapshot then the base disk is considered to be "cold" data, so when you migrate the VM it will use vMotion to move the snapshot (and the compute/memory data) and then svMotion to move the base disk.
  • Finally, note that non-active snapshots are also "cold" data. If the VM has multiple snapshots, only the active one migrates via vMotion. All other disk data migrates via svMotion.
Vinothr90
Contributor
Contributor
Jump to solution

Can we configure a VMKernel adapter for both vMotion and Provisioning traffic?

0 Kudos