vMotion fails at 22% with time out error

[9990374530]

Hi,

vMotion keep on failing at 22% with below error. All firewall rules are in place but no luck. Any help would be appreciated.

vMotion failed. System Error. Source side error is : Source side relocate failed for the virtual machine. failed with error Connection closed by remote host, possibly due to timeout (195887167). msg.checkpoint.precopyfailure Migration to host failed with error Connection closed by remote host, possibly due to timeout (195887167). vob.vmotion.stream.keepalive.read.fail:vMotion migration Target side error is : A general system error occurred: vMotion failed: unknown error msg.migrate.waitdata.platform:Failed waiting for data. Error 195887137. Timeout. vob.vmotion.net.send.start.failed:vMotion migration [172423128:5033295202744408204] failed to send 

[ChrisFD2]

What does Service Mesh diagnostics show?

Usually this error is related to host to IX appliance data flow, check https://ports.esp.vmware.com/home/VMware-HCX for both source and destination sites, and ensure that the Network & Compute Profiles backing the Service Mesh are configured correctly.

[9990374530]

Hi,

Service Mesh Diag is all green without any error. All the ports are configured properley.

All the appliances are up in service mesh, that shows that it has been configured correct.

Source and destination ports 4500 and 500 are open but every time it keeps on failing at 22% with timeout error.

[ChrisFD2]

UDP 500 hasn't been a requirement for quite a while, only UDP 4500 (or 4500-4268 if APR is enabled).

What are the source and destination environments?

How is the source host vMotion vmk set up? Is it all on a single vmk with management, or does it have its own vmk/backing network? As you need to ensure the HCX profiles match this config. If for example the HCX vMotion network is set to a different L3 network to what is use on the hosts then you need a static route on the compute profile as per the HCX user guide.

Network traffic for a HCX vMotion would be source host vmk > IX appliance over vMotion network, and then IX to IX over the uplink network, then from destination IX to destination host via the vMotion network.

[9990374530]

Hi Chris,

Yes, vmotion network is in a different L3 network that we use in the hosts.

It seems that we need to define an static route like you say in the compute profile for vmotion. But it should be created in the Network profile where we will define the Vmotion network traffic type right?

Can you guide me where I can create static route for vMotion in compute profile? Do I need to enable MON and configure the static route?

Is there anything else that I need to validate?

thanks for the support 

[D_Sato]

Sorry, I interrupted you.

Please refer to the documentation for how to configure static routes in computing profiles.
There is Advanced Configurations on the screen where you specify the network profile for vMotion, so you can configure it from there.
When configuring static routes, be careful not to include other networks such as the management network.
The MON function for L2 extension does not require any configuration.

https://docs.vmware.com/en/VMware-HCX/4.9/hcx-user-guide/GUID-BBAC979E-8899-45AD-9E01-98A132CE146E.h...
10. Select the vMotion Network Profile

[ChrisFD2]

@D_Sato is correct, the steps are in the user guide, it is quite comprehensive so I would suggest you take an offline copy of the PDF. I believe I also pointed you toward the static route in another recent thread.

[9990374530]

Hi guys,

thanks a lot.

i believe it should be here

[9990374530]

Hello Guys,

I have createed the static route for vmotion in compute profile but it failed again. service mesh diagnostics is executed successfully without any errors.

From the IX machine the route sends trafic to cloud side.

[ChrisFD2]

Is this to VMC? Is it going over DX or public internet?

Are you able to share more of your configuration?

If it's VMC then the support team should be able to assist also.

[9990374530]

Hi,

Its an POC on VMC infrastructure with policy based VPN setup.

[ChrisFD2]

As in, HCX from source to the VMC SDDC destination is being routed over the VPN? In which case that is not supported.

https://kb.vmware.com/s/article/78021

[9990374530]

Hi Chris,

You are right, you cannot put an encryption tunnel on top of another.

As we don’t have a DX, we need to create a network profile for the HCX service mesh that goes through the Internet. I believe I can do the connection between the HCX components through my public IP. If there’s any constraint on this, or issue to configure some specific IP ranges as it’s a pilot, I can revert the private vCenter resolution and move to public just for my testing purposes.

Once we decides to move to production we can change this setup and config the required network on-premise and VMC.

Does this make sense?

[ChrisFD2]

Hi,

For site pairing, if vCenter is set to public then you need to pair the sites over the internet using the public HCX IP/URL. If it's private, you can pair over the VPN.

For the uplink, you can SNAT out from your source environment on the uplink profile network, the setup of this will be configured as long as the far end compute profile is using 'Internet' as its uplink and you select that when creating the Service Mesh.

https://docs.vmware.com/en/VMware-HCX/4.9/hcx-getting-started/GUID-70F9C40C-804C-4FC8-9FBD-77F9B2FA7...

[9990374530]

Hi Chris,

So, I have set vcenter and hcx to private in vmc. Is it correct?
Is there any option to configure SNAT in source uplink network profile?

I am unable to understand this line!!!

the setup of this will be configured as long as the far end compute profile is using 'Internet' as its uplink and you select that when creating the Service Mesh.

[ChrisFD2]

If resolution is private, then you can pair over the VPN, however the Service Mesh must go over either the internet or DX.

SNAT would be done by your on-prem router/firewall, provided the uplink interface can access the internet. 

[9990374530]

Hi Chris,

So I have set the resolution as private for vCenter and HCX under the settings tab in VMC. So, now I can pair over the VPN. Is that right?

However, when you say service mesh would go over the internet, I have selected this configuration in the service mesh, is that correct?

 

 I did the SNAT of HCX components to a single public IP at On-premise firewall.

[9990374530]

Hi Chris,

Or may be I will have to create a network profile for HCX uplink traffic so that HCX service mesh that goes through the Internet? If I do the connection between the HCX components through their public IP then do I need to request the public IP from the HCX console itself?

Then what will be the IP resource pool, prefix and gateway I will have to assign in case I will be configuring HCX uplink network profile with public IP?