Issues between newly created T1 and T0

[Achmo]

Hello all,

I am having a weird issue and I would like your assistance to solve it.
So, we recently purchased a private cloud solution that comes with vSphere and NSX-T.
The provider has created a single T0 gateway that connects to their infrastructure for Internet connectivity, a T1 gateway(let’s call it T1-A), and a couple of segments, from which we can use one of them for internet connectivity on a VM(let’s call it segment-A).
T1-A also comes with a sNAT rule and all the firewalls are allowing everything.
We created a new VM(VM-A), used segment-A, placed the correct IPs and everything works straight away, we can ping, install updates, etc..

Now, we created our own segment(segment-B), a new T1(T1-B) and we linked segment-B to T1-B and T1-B to the T0. We placed the sNAT rules, checked the firewall and configuration to be the same as the working ones, and we can not ping outside our infrastructure.
We tried changing segment-B to connect to T1-A(and we also created the same sNAT rules we did on T1-B to T1-A) and everything works, so we deduced the fault lies in the T1-B.

We checked the NAT, the firewall, and even the configuration(since we can compare it with the working example) and everything seems the same.
Now here is where the issue is getting even weirder:
Between each T1 and the single T0, we use 100.64.0.0/16(which I believe is by default).
When we try to ping from VM-B the IP of T0 on that subnet(in our example the IP is 100.64.32.4) it does not work, whereas from VM-A we can ping its corresponding IP(100.64.32.0).
To make things even weirder, we used the Traceroute tool from within NSX, and it says that both VM-A and VM-A can ping google IP(8.8.8.8) but in reality, we can not ping it from VM-B.

Do you guys have any suggestions on what to check next?

Thank you.

[shank89]

The advertisements on the T1 are often forgotten, have you ensured SNAT / connected segments and whatever else you require to be advertised, is?

[Achmo]

Hello,

Thanks for the swift reply. Yeah, I followed the same configuration as the working T1(T1-A).
Here is a screenshot:

It's really confusing because T1-A works, and with the same configuration T1-B does not work, so I am a bit lost

[shank89]

Can you get east-west working? VM on Seg-B pinging VM on Seg-A ?

[Achmo]

Hello,

East-west is not working in the current setup. The only way that I can ping each other is if I connect both VMs to the same T1.
Otherwise, it fails.
However, if I use the Port Connectivity Tool it shows that there is(or there should be) port connectivity between the 2:

Traceflow tool also says that the package is delivered.
Just to know, do these tools assume if something is working based on the setup, or do they actually run the commands/simulate things?
Because I find it very weird that for example I can not ping each other, but traceflow says that the packet is delivered without any issues. Same when I try to ping Google, traceflow is saying that the packet is delivered on the uplink, but nothing happens.

 

[shank89]

So it looks like you have workload on different host transport nodes.  If that is the case, are your GENEVE tunnels up?
Have you tested pings from TEP to TEP using jumbo frames?  vmkping ++netstack=vxlan <destTEPIP> -s 8972 -d (from an ESXi host).

[Achmo]

I checked the tunnels and they are all showing UP through NSX-T monitor's page.
Unfortunately, I do not have access to any console/ssh session, so all the troubleshoot and testing I can do is via the GUI manager and the console of a VM itself.

[shank89]

It might be helpful to contact someone who can check things in the CLI for you as well.

[SrVMoussa]

Hi, 

What firewalls applied rules to the VM, and let me ask you may have configured Your T1s to advertise the routes to its respective T0s, correct? 

[Gregor2]

Did you have a fix on this issue? I have the same problem with it.

Paycheckrecords

[Achmo]

Hey @SrVMoussa  the firewall that is applied to every T1 and T0, is to allow all traffic from any to any because so far this is a staging environment.
And yes, we have enabled everything on the T1 route re-distribution.

@Gregor2 so far nothing, we are now waiting for further checks from the provider's side.

[SrVMoussa]

Hi, 

I am sorry I didn't follow this one - I think you resolved it now 

If you did please let us know, and if not we may continue updating this thread 

[zlq5001]

Was there a solution to this issue? I am facing the same problem.

[mynameisalex]

I am also facing the same problem.

[RMS_engineer]

HI All,

Have as anyone tried the following

https://kb.vmware.com/s/article/83743

Also saw a similar if not the same solution on another forum:

https://communities.vmware.com/t5/VMware-NSX-Discussions/NSX-T-N-S-routing-troubleshooting/td-p/5301...

[SrVMoussa]

Hey I guess I replied to this one, but surprisingly it seem I didn't .. the only thing is that could be a TEP VLAN not passing the physical

[mmoubarak]

Can you migrate VM-B to the same host where VM-A resides? to test the connectivity.