We have two sites, our main site, which Ill refer to as SiteA, and a remote site, which Ill refer to as SiteB.
SiteA has a VCenter appliance running with the IP 10.20.4.20. SiteA and SiteB are connected via a MetroE connection. The route from SiteA to SiteB is reachable via IP 10.20.4.250. In addition to this there is also a VPN connection on their firewall for redundancy, which we manually control at this point.
SiteB's VSphere server IP is 10.30.4.10. SiteB’s to SiteA’s MetroE connection routes via IP 10.30.4.250 .
During normal operation SiteB 10.30.4.10 gets to 10.20.4.20 via the MetroE connection on 10.30.4.250 . When we need to schedule maintenance on our MetroE connection we failover to VPN. The VPN route is reachable via SiteB default gateway which is on the firewall, 10.30.4.1. The VSphere server at SiteB is also configured with a default gateway of 10.30.4.1 . My problems begin with VSphere when we failover to VPN. Vsphere is hardcoding a manual route of 10.20.4.20 255.255.255.255 10.30.4.250 vmk1 Manual which can be seen running the command esxcli network ip route ipv4 list.
The MetroE route might still be reachable but is not the way we want the traffic to take. All other machines when we failover to VPN take the VPN route.
I reached out to VmWare who says the route is always added to their route table and does not disappear until it is unreachable for an HR. As no other devices do this, I am wondering what is going on with VSphere. How can we get VSphere to behave like all other devices that always go to their default gateway to get the route? I can simulate this issue with other sites as well.
The following route is coming from ICMP redirects from the firewall / router. You can disable ICMP redirects on the router and check.
10.20.4.20 255.255.255.255 10.30.4.250 vmk0 MANUAL
=> Reference KB:
Can you please get the output for the following two commands on ESXi host.
# esxcfg-vmknic -l
# esxcli network ip route ipv4 list
FROM SITEB
Interface Port Group/DVPort/Opaque Network IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type NetStack
vmk0 Management Network IPv4 10.30.4.10 255.255.255.0 10.30.4.255 00:1f:fe:ab:50:ef 1500 65535 true STATIC defaultTcpipStack
Network Netmask Gateway Interface Source
--------------- --------------- --------------- --------- ------
default 0.0.0.0 10.30.4.1 vmk0 MANUAL
10.20.4.20 255.255.255.255 10.30.4.250 vmk0 MANUAL
10.30.4.0 255.255.255.0 0.0.0.0 vmk0 MANUAL
The following route is coming from ICMP redirects from the firewall / router. You can disable ICMP redirects on the router and check.
10.20.4.20 255.255.255.255 10.30.4.250 vmk0 MANUAL
=> Reference KB:
I agree as we see that in the wireshark.
Why is VSphere adding these as a manual route and keeping it there? Other network devices are switching their routes based on the route change on the firewall.
To manually trigger the clearing of the ICMP Redirect generated routes, restart the network using these commands:
# esxcli network ip interface set -e false -i vmk0
# esxcli network ip interface set -e true -i vmk0
is there a way to deny icmp redirects to vsphere hosts? Found the below
These commands would work on vCenter Appliance.
But in this case the routes are added on ESXi host. So either disable / enable vmk0 from DCUI or disable ICMP redirect on router / firewall.
appreciate your help!