Internet Traffic not routed for Dual-Homed SHD

[raGlenn]

I have a dual-homed Photon/v3.0 instance as Skyline-HD where one link (NIC) is configured for an Internet facing subnet, however, this Photon instance cannot access Internet targets.

<<Configuration Specifics>>
Host: ESXi200/v6.7U3, Guest: SkylineHD v2.5.2, dual homed as per Photon OS 3.0 documentation

Of the SHD VM ...

NIC1 Configuration File: 99-dhcp-en.network
[Match]
Name=eth0

[Network]
DHCP=yes

[DHCP]
UseDNS=true

NIC2 Configuration File: 99-static-en.network
[Match]
Name=eth1

[Network]
Address=172.20.0.24/24
Gateway=172.20.0.2
DNS=172.20.0.11

[DHCP]
UseDNS=false

<<CLI Queries>>
# ip address
1: lo
…
2: eth0: …
Inet 10.208.122.154/24 …
…
3: eth1: …
Inet: 172.20.0.24/24 …
…

# ip route
default via 172.20.0.2 dev eth1 proto static
default via 10.208.122.I dev eth0 proto dhcp src 10.208 .122.154 metric 1024
10.208.122.0/24 dev eth0 proto kernel scope link src 10.208.122.154
10.208.122.I dev eth0 proto dhcp scope link src 10.208.122.154 metric 1024
172.20.0.0/24 dev eth1 proto kernel scope link src 172.20.0.24

# networkctl status
State: routable
Address: 10.208.122.154 on eth0
172.20.0.24 on eth1
…
Gateway: 172.20.0.2 on eth1
10.208.122.I on eth0
DNS: 172.20.0.11
10.208.1.12
10.208.1.14

<<Problem Specifics>>
Of this dual homed configuration, routing non-172.20.0 queries to the DNS servers of the internet facing NIC (DHCP client) are not resolved by the respective DNS Servers. Upon IP route query (above), I find no route for such servers; I can’t ping them. In adding a dynamic route for the internet facing DNS Servers …

# route add -net 10.208.1.0/24 gw 10.208.122.154
# ip route
default via 172.20.0.2 dev eth1 proto static
default via 10.208.122.I dev eth0 proto dhcp src 10.208 .122.154 metric 1024
10.208.1.0/24 via 10.208.122.154 dev eth0 scope link
10.208.122.0/24 dev eth0 proto kernel scope link src 10.208.122.154
10.208.122.I dev eth0 proto dhcp scope link src 10.208.122.154 metric 1024
172.20.0.0/24 dev eth1 proto kernel scope link src 172.20.0.24

… I am now able to ping those DNS servers; e.g. 10.208.1.12, however, I cannot ping 8.8.8.8 or vmware.com (known Internet targets). However, I can ping ESXi200.InfraMgmt.PAx; an intranet zone member. The failed pings are resolved by (the wrong) NIC/gw172.20.0.2 as Destination Host Unreachable. Other hosts on this subnet can access the internet.  I'm now exhausted and hoping somebody out there has some insight to this problem; ... help ... 😉 

Thanks,

[raGlenn · Accepted solution]

I finally got this resolved (without moderation help; -no one ever responded 😉 ).   The solution is to follow the Photon OS/v3.0 doc and those other docs referenced thereby; allow 4 or 5 days to sift through everything and try stuff.  However, the prescriptions thereof yield ambiguous (at best) results while applying such to a vCenter instance.  For resolution of my SHD instance (this problem) i added 2 route sections to the Internet facing NIC (eth0) configuration; i didn't need to do anything to the 2nd/static NIC (eth1). This new eth0 configuration follows:

[Match]
Name=eth0

[Network]
DHCP=yes

[DHCP]
UseDNS=true

[Route]
GatewayOnLink=true
Destination=10.208.1.0/24

[Route]
GatewayOnLink=true
Destination=0.0.0.0/0

For this link, my DHCP hands out 10.208.122.0 IPs, however, the DNS Servers are on 10.208.1.0; that was one issue. The other issue was to get defaulted traffic out the Internet facing NIC as the default deployment method i used did yield a default route out the other NIC; hence, the added destination=0.0.0.0/0 route (above).

For those pursuing resolve for multihomed vCenter instances, you may want to LinkLocalAddressing=true on your Intranet side NIC. All such arguments are (somewhat) defined on https://www.freedesktop.org/software/systemd/man/systemd.network.html. -May the force be with u!