VMware Cloud Community
saintdle1
Contributor
Contributor

Select vMotion virtual NIC takes 60 seconds to complete

Hi, I have a script that sets up VMware hosts networking, but for some reason when it's configuring the VMkernel to be enabled for vMotion, the script sits there waiting, and looking in the vCenter task, it always takes 60 seconds for this action to complete.

Is this normal, or is it some kind of fault,

Here is my command line that's causing the wait

New-VMHostNetworkAdapter -VMhost $vmhost -virtualswitch vSwitch1 -portgroup vMotion_02 -ip $vmotionip1 -subnetmask 255.255.255.0 -Mtu 1500 | Set-VMHostNetworkAdapter -VMotionEnabled $true -Confirm:$False

vCenter Task details

Select vMotion virtual NIC

start time - 18/07/2016 14:22:27

Finish time - 18/07/2016 14:23:28

Cheers in advance

Personal technical ramblings; - www.vEducate.co.uk
Tags (1)
Reply
0 Kudos
11 Replies
LucD
Leadership
Leadership

You will need to have a look in the vpxd logs to see what is actually taking so long.

As a sidenote, why the separate Set-NetworkAdapter? You could do this all in the same cmdlet.

For example (btw I used splatting to make it a bit more readable)

$sNIC = @{

    VMHost = $vmhost

    VirtualSwitch = 'vSwitch1'

    PortGroup = 'vMotion_02'

    IP = $vmotionip1

    SubnetMask = '255.255.255.0'

    VMotionEnabled = $true

    Confirm = $False

}

New-VMHostNetworkAdapter @sNIC


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
saintdle1
Contributor
Contributor

I'm pretty new to powershell/powerCLI extra

so I'm getting there with it, cheers for the advice

Personal technical ramblings; - www.vEducate.co.uk
Reply
0 Kudos
LucD
Leadership
Leadership

No problem :smileycool:

Is it still relatively slow when you do it with the one cmdlet?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
saintdle1
Contributor
Contributor

yes,

So the commands are run as part of a wider range of commands for setting up the host, but essentially, this one is the command that takes the longest.

If I manually do each command once, the setting the vMotion as enabled on the VMkernel still takes 60 seconds, I'm just digging the VPXD logs now

Personal technical ramblings; - www.vEducate.co.uk
Reply
0 Kudos
saintdle1
Contributor
Contributor

So here is what I can see

2016-07-18T13:59:15.068Z info vpxd[7F12A1A5A700] [Originator@6876 sub=vpxLro opID=25df7774] [VpxLRO] -- BEGIN task-450 -- vmotionSystem-85 -- vim.host.VMotionSystem.selectVnic -- 528cbacf-8653-2a11-5a19-e11ca1677dd8(52c203eb-d209-034b-aae2-1ff5dd20f4ab)

2016-07-18T13:59:16.742Z info vpxd[7F12A1A5A700] [Originator@6876 sub=MoHostConfig opID=25df7774] [VpxdMoVMotionSystem::SelectVnic] Wait for another host sync to get the updated config.vmotion.

2016-07-18T13:59:20.787Z info vpxd[7F12A1A5A700] [Originator@6876 sub=MoHostConfig opID=25df7774] [VpxdMoVMotionSystem::SelectVnic] Wait for another host sync to get the updated config.vmotion.

2016-07-18T13:59:23.806Z info vpxd[7F129A37E700] [Originator@6876 sub=vpxLro opID=opId-4a7a6049-bb9e-4091-a288-cfc79dbcc093-0b-ba] [VpxLRO] -- BEGIN task-internal-37101 -- ServiceInstance -- vim.ServiceInstance.GetServerClock -- 52eec56b-d324-0482-9082-f321031163b9(52476de5-cefd-0c10-3358-c4372a62441a)

2016-07-18T13:59:23.806Z info vpxd[7F129A37E700] [Originator@6876 sub=vpxLro opID=opId-4a7a6049-bb9e-4091-a288-cfc79dbcc093-0b-ba] [VpxLRO] -- FINISH task-internal-37101

2016-07-18T13:59:28.824Z info vpxd[7F12A1A5A700] [Originator@6876 sub=MoHostConfig opID=25df7774] [VpxdMoVMotionSystem::SelectVnic] Wait for another host sync to get the updated config.vmotion.

2016-07-18T13:59:29.323Z info vpxd[7F129A37E700] [Originator@6876 sub=vpxLro opID=41e9774b] [VpxLRO] -- BEGIN task-internal-37105 -- ServiceInstance -- vim.ServiceInstance.GetContent -- 528fdae4-c7cd-a4a6-dc1d-afe447fd456b(522069fc-1b04-64ea-231c-19d4016386fb)

2016-07-18T13:59:29.323Z info vpxd[7F129A37E700] [Originator@6876 sub=vpxLro opID=41e9774b] [VpxLRO] -- FINISH task-internal-37105

2016-07-18T13:59:31.734Z info vpxd[7F129A37E700] [Originator@6876 sub=vpxLro opID=opId-7105e296-69ab-4f0d-a47a-6a6e8a55a452-eb-31] [VpxLRO] -- BEGIN task-internal-37106 -- ServiceInstance -- vim.ServiceInstance.GetServerClock -- 52148c8c-8030-f970-4851-fabee8a687b0(52098b41-f179-96ae-4b0d-be6d4a910baf)

2016-07-18T13:59:31.734Z info vpxd[7F129A37E700] [Originator@6876 sub=vpxLro opID=opId-7105e296-69ab-4f0d-a47a-6a6e8a55a452-eb-31] [VpxLRO] -- FINISH task-internal-37106

2016-07-18T13:59:35.831Z info vpxd[7F129A480700] [Originator@6876 sub=vpxLro opID=opId-0b605910-2141-497f-a7ee-cc6fcd95061d-ef-c2] [VpxLRO] -- BEGIN task-internal-37107 -- ServiceInstance -- vim.ServiceInstance.GetServerClock -- 521491ed-d696-cb3c-2d5d-3d7f894ff072(52fbe230-e55d-fc25-346e-deb3a644a76a)

2016-07-18T13:59:35.831Z info vpxd[7F129A480700] [Originator@6876 sub=vpxLro opID=opId-0b605910-2141-497f-a7ee-cc6fcd95061d-ef-c2] [VpxLRO] -- FINISH task-internal-37107

2016-07-18T13:59:40.871Z info vpxd[7F12A1A5A700] [Originator@6876 sub=MoHostConfig opID=25df7774] [VpxdMoVMotionSystem::SelectVnic] Wait for another host sync to get the updated config.vmotion.

2016-07-18T13:59:48.289Z info vpxd[7F1299E74700] [Originator@6876 sub=vpxLro opID=5acc69e6] [VpxLRO] -- BEGIN task-internal-37162 -- SessionManager -- vim.SessionManager.sessionIsActive -- 524c82a0-2ea9-292a-734c-712bbd3bd276(5202344d-52fa-94b4-6231-6b59aafcef2c)

2016-07-18T13:59:48.289Z info vpxd[7F1299E74700] [Originator@6876 sub=vpxLro opID=5acc69e6] [VpxLRO] -- FINISH task-internal-37162

2016-07-18T13:59:52.556Z info vpxd[7F1299E74700] [Originator@6876 sub=vpxLro opID=671e63f] [VpxLRO] -- BEGIN task-internal-37163 -- PerfMgr -- vim.PerformanceManager.queryStats -- 52af8b9e-72c1-a9af-521f-963d0a2f2023(52cfcca5-d355-a6fe-5ae7-97c659b72312)

2016-07-18T13:59:52.572Z info vpxd[7F1299E74700] [Originator@6876 sub=vpxLro opID=671e63f] [VpxLRO] -- FINISH task-internal-37163

2016-07-18T13:59:53.806Z info vpxd[7F1299E74700] [Originator@6876 sub=vpxLro opID=opId-4679ee64-5961-4f96-aae5-eb60fcce1da1-88-56] [VpxLRO] -- BEGIN task-internal-37164 -- ServiceInstance -- vim.ServiceInstance.GetServerClock -- 52eec56b-d324-0482-9082-f321031163b9(52476de5-cefd-0c10-3358-c4372a62441a)

2016-07-18T13:59:53.806Z info vpxd[7F1299E74700] [Originator@6876 sub=vpxLro opID=opId-4679ee64-5961-4f96-aae5-eb60fcce1da1-88-56] [VpxLRO] -- FINISH task-internal-37164

2016-07-18T13:59:56.912Z info vpxd[7F12A1A5A700] [Originator@6876 sub=MoHostConfig opID=25df7774] [VpxdMoVMotionSystem::SelectVnic] Wait for another host sync to get the updated config.vmotion.

2016-07-18T14:00:01.734Z info vpxd[7F129A603700] [Originator@6876 sub=vpxLro opID=opId-61dbc410-6e68-4890-a028-5913dbdd223e-52-d0] [VpxLRO] -- BEGIN task-internal-37166 -- ServiceInstance -- vim.ServiceInstance.GetServerClock -- 52148c8c-8030-f970-4851-fabee8a687b0(52098b41-f179-96ae-4b0d-be6d4a910baf)

2016-07-18T14:00:01.734Z info vpxd[7F129A603700] [Originator@6876 sub=vpxLro opID=opId-61dbc410-6e68-4890-a028-5913dbdd223e-52-d0] [VpxLRO] -- FINISH task-internal-37166

2016-07-18T14:00:02.584Z info vpxd[7F1299E74700] [Originator@6876 sub=vpxLro opID=2d915ede] [VpxLRO] -- BEGIN task-internal-37167 -- ServiceInstance -- vim.ServiceInstance.GetContent -- 52a1ca47-5f96-cdaf-3d5c-2aa8ffd935d2

2016-07-18T14:00:02.585Z info vpxd[7F1299E74700] [Originator@6876 sub=vpxLro opID=2d915ede] [VpxLRO] -- FINISH task-internal-37167

2016-07-18T14:00:02.605Z info vpxd[7F129ACFD700] [Originator@6876 sub=vpxLro opID=8eb0ae0] [VpxLRO] -- BEGIN task-internal-37168 -- SessionManager -- vim.SessionManager.GetCurrentSession -- 522140cc-051c-287c-0658-c08b9402afbf

2016-07-18T14:00:02.605Z info vpxd[7F129ACFD700] [Originator@6876 sub=vpxLro opID=8eb0ae0] [VpxLRO] -- FINISH task-internal-37168

2016-07-18T14:00:02.607Z info vpxd[7F1299E74700] [Originator@6876 sub=vpxLro opID=3e284b4d] [VpxLRO] -- BEGIN task-internal-37169 -- ServiceInstance -- vim.ServiceInstance.GetContent -- 527d1458-9eef-ad47-4684-f45c664e9fce

2016-07-18T14:00:02.607Z info vpxd[7F1299E74700] [Originator@6876 sub=vpxLro opID=3e284b4d] [VpxLRO] -- FINISH task-internal-37169

2016-07-18T14:00:02.615Z info vpxd[7F129ACFD700] [Originator@6876 sub=vpxLro opID=6f8966e3] [VpxLRO] -- BEGIN task-internal-37170 -- SessionManager -- vim.SessionManager.loginExtensionByCertificate -- 52e5c65b-0958-652b-119d-015b3029bf70

2016-07-18T14:00:02.616Z info vpxd[7F129ACFD700] [Originator@6876 sub=vpxLro opID=6f8966e3] [VpxLRO] -- FINISH task-internal-37170

2016-07-18T14:00:02.619Z info vpxd[7F1299E74700] [Originator@6876 sub=vpxLro opID=621723a3] [VpxLRO] -- BEGIN task-internal-37171 -- ServiceInstance -- vim.ServiceInstance.GetContent -- 52e5c65b-0958-652b-119d-015b3029bf70(52638661-c07f-71c1-3b82-bde745d6c7e6)

2016-07-18T14:00:02.619Z info vpxd[7F1299E74700] [Originator@6876 sub=vpxLro opID=621723a3] [VpxLRO] -- FINISH task-internal-37171

2016-07-18T14:00:02.631Z info vpxd[7F1299FF7700] [Originator@6876 sub=vpxLro opID=7ab3b6ee] [VpxLRO] -- BEGIN task-internal-37172 -- group-d1 -- vim.ManagedEntity.GetTag -- 52e5c65b-0958-652b-119d-015b3029bf70(52638661-c07f-71c1-3b82-bde745d6c7e6)

2016-07-18T14:00:02.631Z info vpxd[7F1299FF7700] [Originator@6876 sub=vpxLro opID=7ab3b6ee] [VpxLRO] -- FINISH task-internal-37172

2016-07-18T14:00:02.633Z info vpxd[7F129ACFD700] [Originator@6876 sub=vpxLro opID=15b8f5e2] [VpxLRO] -- BEGIN task-internal-37173 -- ServiceInstance -- vim.ServiceInstance.retrieveContent -- 52e5c65b-0958-652b-119d-015b3029bf70(52638661-c07f-71c1-3b82-bde745d6c7e6)

2016-07-18T14:00:02.633Z info vpxd[7F129ACFD700] [Originator@6876 sub=vpxLro opID=15b8f5e2] [VpxLRO] -- FINISH task-internal-37173

2016-07-18T14:00:02.640Z info vpxd[7F129A603700] [Originator@6876 sub=vpxLro opID=89c27ba] [VpxLRO] -- BEGIN task-internal-37174 -- SessionManager -- vim.SessionManager.logout -- 52e5c65b-0958-652b-119d-015b3029bf70(52638661-c07f-71c1-3b82-bde745d6c7e6)

2016-07-18T14:00:02.640Z info vpxd[7F129A603700] [Originator@6876 sub=vpxLro opID=89c27ba] [VpxLRO] -- FINISH task-internal-37174

2016-07-18T14:00:05.831Z info vpxd[7F129A786700] [Originator@6876 sub=vpxLro opID=opId-0b605910-2141-497f-a7ee-cc6fcd95061d-09-30] [VpxLRO] -- BEGIN task-internal-37175 -- ServiceInstance -- vim.ServiceInstance.GetServerClock -- 521491ed-d696-cb3c-2d5d-3d7f894ff072(52fbe230-e55d-fc25-346e-deb3a644a76a)

2016-07-18T14:00:05.831Z info vpxd[7F129A786700] [Originator@6876 sub=vpxLro opID=opId-0b605910-2141-497f-a7ee-cc6fcd95061d-09-30] [VpxLRO] -- FINISH task-internal-37175

2016-07-18T14:00:16.955Z error vpxd[7F12A1A5A700] [Originator@6876 sub=MoHostConfig opID=25df7774] [VpxdMoVMotionSystem::SelectVnic] Host sync doesn't send the updated config.vmotion after 60 seconds.

2016-07-18T14:00:16.955Z info vpxd[7F12A1A5A700] [Originator@6876 sub=vpxLro opID=25df7774] [VpxLRO] -- FINISH task-450

It looks as if the last line is the most interesting "Host sync doesnt send the updated config.vmotion after 60 seconds, so I presume that the host takes the vMotion config fine straight away, but doesnt tell vCenter

Personal technical ramblings; - www.vEducate.co.uk
Reply
0 Kudos
LucD
Leadership
Leadership

Looks more like there might be an issue with the sync of one or more of the vmkernels over which vMotion was defined.
Are you usiing VDS for all vmkernels, or are there VSS based ones?

Are all the nodes in the cluster fully configured?

Is HA showing warnings/errors?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
saintdle1
Contributor
Contributor

using standard switches for everything

all hosts are configured and working with HA and DRS with no warnings or errors

Personal technical ramblings; - www.vEducate.co.uk
Reply
0 Kudos
LucD
Leadership
Leadership

Did you check if the time on all ESXi nodes is synchronised?

Do you have multiple multiple vmnic on the switch? Are they in an active/standby config?

Try use only one vmnic on the switch (move the 2nd one to Unused Adapters), see if that makes a difference.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
pmwurst
Contributor
Contributor

Hi LucD,

great to read from you on this - that's raising hopes 😃
We are facing the same issue in our environment - no matter whether we try it on vDS's or vSS's, always on HA+DRS powered Clusters without warnings.

Indeedd, we have multiple vmnics on the vswitches but those 60 seconds are affecting us in active/active and active/standby configurations.

On vDS we are only able to trigger active/unused on portgroup basis - afaik there is no possibility to configure "teaming and failover" on vDS directly. But we get that delay even if we set the second adapter to unsused...

Did you tried that in your (test) environment and are you sure that this is not a general issue?

Thanks and regards!

Reply
0 Kudos
LucD
Leadership
Leadership

Could be a general issue.

But not a global one, otherwise I would have expected more complaints.

In which vSphere version (vCenter/VCSA, ESXi) do you experience the issue?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
pmwurst
Contributor
Contributor

Ah yes, we had similiar thoughts on this but didn't find a way to fix / work around this issue :smileyconfused:

We're on VCSA 6.0.0 3634749 / ESXi 6.0.0 3825889 and tested on vDS 5.5.0 and 6.0.0...

Reply
0 Kudos