Cross Communication testing between 2 VMs

[MrVmware9423]

Dear Team,

we have migrated the DFW rules from the old to the new environment using MC (Distributed firewall -Advanced Mode), now we are moving to the second stage, i.e We are planning to migrate a workload from the old to the new environment using HCX (Bulk Migration).

VM1 & VM2 both runs in an old environment, VM1 (Web) communicates with VM2 (App, DB)

I would like to confirm if we migrate VM1 (not extended the VM1 network) to the new environment will it be able to communicate with VM2 =>  Yes/No

OR 

I would like to confirm if we migrate VM1 (extended the VM1 network) to the new environment will it be able to communicate with VM2 =>  Yes/No

Please assist.

[grimsrue]

@MrVmware9423 

To answer your question it will depend on if your two environments are in two separate network domains or not.

If both environments are in the same domain then the destination side should have the VLAN/Subnet trunked down the the ESXi hosts. You will create a portgroup on the vDS for the VLAN/Subnet and attach your VM to it when you migrate it.

If both environments are in different Network domains or your network is not using 802.1q Trunking then you will have to extend/stretch your Portgroup over to the destination before you migrate your VMs

The only other way is to change the IP address of the Web VM after you migrate it then update the DNS so it can be resolved by the App/DB VMs. Might have to make some web, app, or DB changes as well depending how the Web, App or DB is setup on the VMs if you use this method.

[MrVmware9423]

I will follow below steps to validate cross communication between workloads which are running in old (VCF3.x )and new VCF4.x environment.

Could you please validate the below steps and add if something is missing. 

VMs Connected to the same logical Switch.
- Deploy 2 Test VMs and they should be part of the same logical segment
- Create a Test DFW rule in both environment which will allow connectivity between VM1, and VM2 on port 22.
- Test the connectivity between VM1 and VM2 on port 22 (vice-versa)
- Test the north south connectivity on VM1 (Try to resolve DNS query)
- Extend the VM1 logical Switch connectivity using HCX.
- Migrate VM1 to VCF4.x environment.
- Test the connectivity between VM1 and VM2 on port 22 (vice-versa)
- Test the north-south connectivity (Try to resolve DNS query)
- Migrate the VM2 on VCF4.x
- Test the connectivity between VM1 and VM2 on port 22 (vice-versa)
- Test the north-south connectivity on VM2 (Try to resolve DNS query).
- Disconnect the VM2 logical switch from DLR
- Unextend the Network using HCX
- Test the connectivity between VM1 and VM2 on port 22 (vice-versa)
- Test the north-south connectivity (Try to resolve DNS query)

VMs Connected to different logical Switches.
- Deploy 2 Test VMs and they should be connected to two different logical segments.
- Create a Test DFW rule in both environments which will allow connectivity between VM1, and VM2 on port 22.
- Test the connectivity between VM1 and VM2 on port 22 (vice-versa).
- Test the north-south connectivity (Try to resolve DNS query)
- Extend the VM1 logical Switch network using HCX.
- Migrate VM1 to VCF4.x environment using HCX.
- Test the connectivity between VM1 and VM2 on port 22 (vice-versa)
- Test the north-south connectivity on VM1 (Try to resolve DNS query).
- Disconnect VM1 logical switch from DLR
- Unextend the VM1 logical switch.
- Test the connectivity between VM1 and VM2 on port 22 (vice-versa)
- Test the north-south connectivity on VM1 (Try to resolve DNS query).
- Migrate the VM2 on VCF4.x
- Test the connectivity between VM1 and VM2 on port 22 (vice-versa)
- Test the north-south connectivity on VM2 (Try to resolve DNS query)
- Disconnect VM2 logical switch from DLR.
- Unextend the VM2 logical switch network.
- Test the connectivity between VM1 and VM2 on port 22 (vice-versa)
- Test the north-south connectivity (Try to resolve DNS query)

[grimsrue]

@MrVmware9423,

Your steps all look correct except for "Unextend the Network using HCX" in both your scenarios. I am not sure what you mean by "Disconnect VM1 logical switch from DLR"? I am also assuming your are using NSX-T DFW?

I am still not sure if your VCF3.x and VCF4.x environments are in the same network domain or if they are in separate network domains. Are you expecting to move the VMs from NSX Souce to NSX-T/Overlay on the destination side or are you going to move the VMs from physical source to the physical network on the destination side? HCX network extension is for allowing VMs to be migrated to different networking environment without changing the VMs network config (i.e. IPs, DNS, firewall, NAS, etc.). HCX network extension just "hairpins" the network traffic back to the source environment. It is essentially a VPN.

You have to do one of two things before you can unextend the network. Expectation is VCF3.x is a separate Source network 'domain' & VCF4.x is separate destination network 'domain')

1. You would need to reIP the VMs to use a new IP from a subnet on the destination side (physical or NSX-T Overlay), then you can remove the network extension.

OR

2. You will have to migrate ALL VMs sitting on the source side subnet to the destination side, then swing the source side Subnet to the destination side (Datacenter. External Cloud, NSX-T, etc).  You can then remove the network extension.
NOTE: This step is a bit more involved based on where the subnet is being swung over to.

If VCF3.x and VCF4.x environments are sitting in the same network domain and will NOT be using the NSX-T overlay then there is no reason to use HCX network extension services, because you can just make use of 802.1q VLAN trunking to make sure both environments can see the same subnet. You can then use HCX to migrate the VMs between the environments and your VMs will not lose network connectivity.

[MrVmware9423]

Thank you for the detailed explanation.

I did the testing as per the above plan of action, and it worked fine. Just wanted to understand one thing in vcf3.x NSX-V environment we have 100 logical switch overlay switches and In VCF4.x we have 4 overlay logical segments which we have extended and then migrated. When I try to see the routes in old NSX-V not able to see NSX-T logical segments are getting learned and In NSX-T not able to see NSX-V logical switches are learned by NSX-T still cross-communication work. ? Could someone please assist.how cross communication is working?/

[MrVmware9423 · Accepted solution]

Since NSX is a stub and won't transit any traffic most customer push a default route down to NSX that will pull all traffic up to the physical network. In this case two separate NSX instances wouldn't have routes in their routing tables for the others network, but would sent all traffic north to the physical network which does have routes to both in the routing table. Uplink router advertise a default route (0.0.0.0/0) to T0. So if VCF-A advertises a network into the physical BGP ASN it will never be populated into the routing table of VCF-B because the default route supersedes the individual routes. Since NSX is a stub ASN of the physical ASN in this use case and won't transit any traffic it isn't needed for the physical ASN to exchange full tables. We send all of our routes up, they just send a default down to us.

[grimsrue]

Your first statement throws off the rest of your post. I had to read your post multiple times to sort of figure out what you are saying. Again you left out a large amount of info in your original post. I did asked multiple times for the info you gave in this last post. If you had give a description of how VCF-A and VCF-B environments were connected on the network then we could have answered your question more specifically. Diagrams help as well.

In your first sentence you state "Since NSX is a stub and won't transit any traffic most customer push a default route down to NSX that will pull all traffic up to the physical network."??????

In your last sentence you state "We send all of our routes up, they just send a default down to us."

To me, your first sentence in the post does not make sense with the rest of what you said in your post. First of all I assume you are using the word "Stub" to refer to the NSX instances connecting to a single physical router and not multiple routers? When I read "Stub" I think of a network that does not route anywhere. Also, a physical router pushing its routes down to NSX will not "pull all traffic up to the physical network". Traffic Routes OUT or Routes IN. There is no pull, but there is push.

Now that I read through you last post multiple times I agree with you that your VCF-A and VCF-B NSX instance would not have each other routes in their routing tables. That is because there is usually a physical router between them and that physical router will have the two NSX-T instances routes stored in its routing table. In some cases you might be able to establish a OSPF router adjacency between NSX-V and NSX-T as long as they were both sitting on the same switch with no router between them. Then NSX-V and NSX-T will have routes for each other in their routing table.

All that being said.......

Your original post was asking if you could migrate your VMs with or without stretching the network through HCX. Now that I have more info I can more specifically answer your question.
You have to stretch/extend your NSX-V segments to NSX-T or the VMs will lose network connectivity once they fully land in NSX-T, unless you are planning on changing those VMs IP addresses once they get to NSX-T. NSX-V and NSX-T are separate network domains and can not attach each other subnets. HCX will create a tunnel between "V" and "T". Then HCX extension process will create a segment on NSX-T and then hair pin the network back to the NSX-V segment. Once all VMs (Sitting on the NSX-V network segment) are migrated to NSX-T then you can swing the Subnet off of NSX-V over to NSX-T (Swap the segment Gateways, shut down the NSX-V segment, advertise NSX-T segment to physical network). At that time you can remove the NSX-V segment extension in HCX.

The other way to do it is to extend the NSX-V network segment to NSX-T. Give the NSX-T segment its own gateway address. Migrate the VMs over to NSX-T and then change the VMs gateway to the NSX-T segment's gateway. You will still have to migrate all the VMs off the NSX-V segment first before you can take down the HCX extension. The NSX-V segment still has to be shut down before you can advertise the NSX-T segment to the physical network.