Flow report is showing 2 different Route for the same Next Hop, Source and Destination IPs

[amirimran]

Hi, 

I have a setup of hub and spoke where internet traffic will go via hub's internet link. One of the spoke (Spoke1) is showing the Next Hop as the hub (eg: Hub1) but the Route are 2 different types; 1 is Branch to Branch and the other is Branch to Backhaul. This is for the same source and destination IPs. When I checked the other spokes (Spoke2 and Spoke3) for the same destination IP (which is Microsoft public IP), the flow report is showing the Next Hop as the hub (Hub1) but there's only 1 type of Route which is Branch to Backhaul. Right now, Spoke1 is having issue connecting to that Microsoft public IP. I can also see the Fortigate firewall log that we put between Hub1 and ISP link, that the traffic to that Microsoft public IP is showing as "IP connection error" which seems to suggest asymmetric routing.

Spoke1, 2 and 3 are all using the same profiles so I am currently stuck as why Spoke1 is seeing 2 different type of Route for the Next Hop Hub1. I am inclined to think that the different type of Route is causing the issue in Spoke1. Appreciate any help

[Benjaman]

@amirimran 

Can you help share the outputs that you are observing? Like screenshots etc.

[amirimran]

Hi @Benjaman ,

Attached is the screenshot. Just noticed that for some flows, it is also not showing the link info. Is this normal?

[Benjaman]

Hi @amirimran 

Thank you for providing the screenshot. I think there are a few things to consider.

Firstly, if you see the route as "Branch to Branch" or "Branch to Backhaul" on the spoke side "Flows" tab, it is completely expected if

1. There is a business rule using just the "Application Name" and not the IP address in the destination IP field and

2. A business rule under the above indicating the destination as "Internet" and backhauling traffic to the Hub.

Reason is that VCE takes a few packets from the flows to read and identify the application name from the flow which involves a bit of time. During this process, as the application name for that flow is not yet determined, VCE sends the traffic out using the next best business rule.

So for a few flows, seeing "Branch to Backhaul" is expected since it's backhauling the traffic to the Hub, and for many more flows "Brach to Branch" should be seen since the subsequent flows start hitting the policy that has just the application name.

With respect to the Link Name not reflecting for a few flows, it's expected since for the flows that are being load-shared, it does not reflect the Interface or Link Name as it sends the traffic out of multiple links.

Coming to the original query on why you are unable to access the application - 

1. Please check if it's expected that the public route i.e., in the Flow dump that you shared, the route towards 52.123.128.14 is expected to be learnt from a remote branch. If not, please stop advertising it at the source VCE.

If it's expected to be learnt from the remote branch, then you can take captures on the LAN side of both VCEs to check if the packets that are actually sent are reaching the other side. 

2. Try configuring the business rule with IP address and play with it sending "Direct"/"Multipath"/"Backhaul" and observe the results. 

If you feel that VCE is dropping the traffic, please feel free to engage support and take it forward as this would have confidential data of your customer which they might not want to be discussed in this forum. 

[amirimran]

Thanks @Benjaman for the explanation. The same Business Policy is being used for other spokes. We don't have a specific rule for that application "Microsoft Office 365" but we do have the rule for "Internet" destination backhauling to the Hub.

It's just weird to me that only this spoke is having issue connecting to the destination IP address while the other spokes are working fine. For the other spokes, the Route is only showing "Branch to Backhaul" for that destination IP address. That is the only difference I can see.

Back to the Flow table of the problematic spoke, if the route to 52.123.128.14 is learned from other Branch, shouldn't the Next Hop showed the Branch Edge name instead of the Hub for flow that is showing "Branch to Branch"? This is another thing that confused me. 

I will try your suggestion playing around with the Business Policy and see what the results are.

Thanks!!

[Benjaman]

Hi @amirimran, 

Kindly check from "Remote Diagnostics" - "Flow Dump" to see what business rule this flow is actually taking. I suspect it might be hitting two types of business rules. One is the Application category(Rule defined just with Application) and the other could be the rule which is under the bottom pointing to "Internet" and might have been backhauled to the Hub.

Depending on the load of the edge, its model, traffic patterns, and the number of applications used, the edge also might be exhausting its application identification database due to which it is failing to hit the top rule.

If route 52.123.128.14 is learned from a specific remote branch, today there is no visibility to which branch it's going in the "Flows" tab. This should be an enhancement.

The Route types can either be "Branch to Branch", "Cloud via Gateway", "Branch to datacenter", or "Brach to backhaul" depending on the destination route. 

If this is the only spoke that shows Branch to Branch, then there should not be a route 52.123.128.14 learned from any other branches. If there is not already then we need to check further on why it displays "Branch to Branch" instead of "Cloud via Gateway" as I expect it should hit the gateway route if the business rule with just application-defined matches.

Route table dump in "Remote Diagnostics" would help and also "Overlay Flow Control" would help from which branch this route is being initiated.

Probably this might need more eyes from a support stand point if you see any discrepancies. Please feel free to create a support ticket after you research a bit more.