VMware Cloud Community
TryllZ
Hot Shot
Hot Shot

Migrating vmnics to DVSwitch : Operation Timed Out

Hi,

This setup is in Workstation version 16.1.0 build-17198959

vCenter version VMware-vCenter-Server-Appliance-6.7.0.45000-16708996_OVF10 (6.7 U3 J)

ESXi version VMware-VMvisor-Installer-201912001-15160138.x86_64

I'm facing an issue with vCenter timing out when migrating vmnics to DVSwitch. The vmnics are not assigned to anything. I create VMKernel adapters in DVswitch, then attach the vmnics to DVSwitch. I did the same for Management ports, and they worked fine. The moment I start the migration for the remaining vmnics, after sometime has passed vCenter shows as reconnecting https://i.ibb.co/19cXJgV/v-Center-Reconnecting.png.

The vpxd.log shows the following.

2021-05-05T13:42:45.149Z info vpxd[05049] [Originator@6876 sub=vpxLro opID=q-298:01-b8] [VpxLRO] -- BEGIN lro-3207 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 52de6b05-ff57-5212-8e2f-2e5c30351224(520a6ec9-121f-c4c8-7aaa-cb0b0aa54c06)
2021-05-05T13:42:45.149Z info vpxd[05049] [Originator@6876 sub=vpxLro opID=q-298:01-b8] [VpxLRO] -- FINISH lro-3207
2021-05-05T13:42:45.150Z info vpxd[05049] [Originator@6876 sub=vpxLro opID=q-297:01-d0] [VpxLRO] -- BEGIN lro-3208 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 52de6b05-ff57-5212-8e2f-2e5c30351224(520a6ec9-121f-c4c8-7aaa-cb0b0aa54c06)
2021-05-05T13:42:45.150Z info vpxd[05049] [Originator@6876 sub=vpxLro opID=q-297:01-d0] [VpxLRO] -- FINISH lro-3208
2021-05-05T13:42:45.160Z info vpxd[05003] [Originator@6876 sub=vpxLro opID=q-299:01-a1] [VpxLRO] -- BEGIN lro-3209 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 52de6b05-ff57-5212-8e2f-2e5c30351224(520a6ec9-121f-c4c8-7aaa-cb0b0aa54c06)
2021-05-05T13:42:45.160Z info vpxd[05003] [Originator@6876 sub=vpxLro opID=q-299:01-a1] [VpxLRO] -- FINISH lro-3209
2021-05-05T13:42:45.594Z error vpxd[05738] [Originator@6876 sub=hostMethod] Commit call for method [updateNetworkConfig] transaction Id [17] failed on host [[vim.HostSystem:host-44,w-esx-vsn-01.vlab.lab]] with exception:[(vmodl.fault.SystemError) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> reason = "Transaction has rolled back on the host."
--> msg = "Received SOAP response fault from [<cs p:00007fb83c3d4020, TCP:w-esx-vsn-01.vlab.lab:443>]: commitTransaction
--> Received SOAP response fault from [<cs p:000000126b0b8950, TCP:localhost:8307>]: commitTransaction
--> A general system error occurred: Transaction has rolled back on the host."
--> }]
2021-05-05T13:42:45.595Z error vpxd[05734] [Originator@6876 sub=hostMethod] Commit call for method [updateNetworkConfig] transaction Id [15] failed on host [[vim.HostSystem:host-47,w-esx-vsn-02.vlab.lab]] with exception:[(vmodl.fault.SystemError) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> reason = "Transaction has rolled back on the host."
--> msg = "Received SOAP response fault from [<cs p:000055b41bcbab90, TCP:w-esx-vsn-02.vlab.lab:443>]: commitTransaction
--> Received SOAP response fault from [<cs p:0000002102452d90, TCP:localhost:8307>]: commitTransaction
--> A general system error occurred: Transaction has rolled back on the host."
--> }]
2021-05-05T13:42:45.605Z error vpxd[05077] [Originator@6876 sub=hostMethod] Commit call for method [updateNetworkConfig] transaction Id [16] failed on host [[vim.HostSystem:host-50,w-esx-vsn-03.vlab.lab]] with exception:[(vmodl.fault.SystemError) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> reason = "Transaction has rolled back on the host."
--> msg = "Received SOAP response fault from [<cs p:00007fb8344acf50, TCP:w-esx-vsn-03.vlab.lab:443>]: commitTransaction
--> Received SOAP response fault from [<cs p:0000003ff197bfb0, TCP:localhost:8307>]: commitTransaction
--> A general system error occurred: Transaction has rolled back on the host."
--> }]
2021-05-05T13:42:45.682Z info vpxd[05739] [Originator@6876 sub=vpxLro opID=q-301:h5ui-getProperties:urn:vmomi:DistributedVirtualPortgroup:dvportgroup-88:ddefdc78-bdea-43c9-8e4d-670bab2c3978:1364190193:01-4c] [VpxLRO] -- BEGIN lro-3210 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 52de6b05-ff57-5212-8e2f-2e5c30351224(520a6ec9-121f-c4c8-7aaa-cb0b0aa54c06)
2021-05-05T13:42:45.683Z info vpxd[05739] [Originator@6876 sub=vpxLro opID=q-301:h5ui-getProperties:urn:vmomi:DistributedVirtualPortgroup:dvportgroup-88:ddefdc78-bdea-43c9-8e4d-670bab2c3978:1364190193:01-4c] [VpxLRO] -- FINISH lro-3210

Firewall has nothing blocked, all ports and protocols are open. I don't know if this is related to the NTP time sync issue I'm facing https://communities.vmware.com/t5/VMware-vCenter-Discussions/NTP-Syncing-6-hours-ahead-ESXi-from-sam...

Complete vxpd.log can be found here : https://mega.nz/folder/hpwz2ISQ#lEuR4AbcECRxBkhqmK7HqA

Reply
0 Kudos
7 Replies
TryllZ
Hot Shot
Hot Shot

Anyone knows about this error

A general system error occurred: Connection reset by peer: The connection is terminated by the remote end with a reset packet. Usually, this is a sign of a network problem, timeout, or service overload.

Reply
0 Kudos
TryllZ
Hot Shot
Hot Shot

Found this, going through it..

https://kb.vmware.com/s/article/76259

Tags (1)
Reply
0 Kudos
virtualhelp
Enthusiast
Enthusiast

Hello,

Enter via ssh to vcsa, enter the following commands. and then retry the migration from the vmnic to the DVswitch.

Run the following commands:

service-control --stop -all
Service-control --start -all

and you share the results with us.

Reply
0 Kudos
TryllZ
Hot Shot
Hot Shot

Hi,

Thanks, I ran the command with the results below.

Command> service-control --stop --all
Operation not cancellable. Please wait for it to finish...
Performing stop operation on service vmware-pod...
Successfully stopped service vmware-pod
Performing stop operation on profile: ALL...
Successfully stopped service vmware-vmon
Successfully stopped profile: ALL.
Performing stop operation on service vmdnsd...
Successfully stopped service vmdnsd
Performing stop operation on service vmware-stsd...
Successfully stopped service vmware-stsd
Performing stop operation on service vmware-sts-idmd...
Successfully stopped service vmware-sts-idmd
Performing stop operation on service vmcad...
Successfully stopped service vmcad
Performing stop operation on service vmdird...
Successfully stopped service vmdird
Performing stop operation on service vmafdd...
Successfully stopped service vmafdd
Performing stop operation on service lwsmd...
Successfully stopped service lwsmd
Command> service-control --start --all
Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Successfully started service lwsmd
Performing start operation on service vmafdd...
Successfully started service vmafdd
Performing start operation on service vmdird...
Successfully started service vmdird
Performing start operation on service vmcad...
Successfully started service vmcad
Performing start operation on service vmware-sts-idmd...
Successfully started service vmware-sts-idmd
Performing start operation on service vmware-stsd...
Successfully started service vmware-stsd
Performing start operation on service vmdnsd...
Successfully started service vmdnsd
Performing start operation on profile: ALL...
Successfully started service vmware-vmon
Successfully started profile: ALL.
Performing start operation on service vmware-pod...
Successfully started service vmware-pod
Command>

Then I performed vmnics assignment, as usual the 1st set assigns successfully, no issues none whatsoever.

https://i.ibb.co/hYHyNPN/W-WSRV-ADDC-01-2021-05-08-20-13-48.png

The next assignment fails as usual, take note of the time between 1st and second assignment, merely 1 minute.

https://i.ibb.co/MVGkkZQ/W-WSRV-ADDC-01-2021-05-08-20-14-32.png

https://i.ibb.co/Krjs2fk/W-WSRV-ADDC-01-2021-05-08-20-18-20.png

Thank You

 

Reply
0 Kudos
TryllZ
Hot Shot
Hot Shot

Just something of note, I have observed vCenter keeps logging me out very very often, I had to log out just a few minutes after assigning the vmnics, and then again after a few minutes.

https://i.ibb.co/TRs1d8v/W-WSRV-ADDC-01-2021-05-08-20-12-06.png

I don't know if this is the reason I can't assign more vmnics because its times out and then logs out as well.

Reply
0 Kudos
virtualhelp
Enthusiast
Enthusiast

Hello,

Disable the firewall completely in the ESXi, I indicate the command:
esxcli network firewall set --enabled false.

Restart the management agents of the esxi, I indicate the command:
/etc/init.d/hostd restart
/etc/init.d/vpxa restart

Try again to migrate the vmnic to the distributed switch.

Another test option is to put a host in maintenance mode and then run the migration of that host's vmnic to rule out getting the same error.

I remain attentive to your comments.

Regards.

Reply
0 Kudos
TryllZ
Hot Shot
Hot Shot

Sorry the steps you mentioned did not resolve the issue.

Not able to find the source of the problem.

Tried removing and re-adding the hosts to vCenter again the same issue and no information in vCenter logs except Operation Timed Out. I have disabled RollBack still the issue exists.

Anything else I can do ?

Reply
0 Kudos