I have two installations with vCenter installed on them that are both doing something that seems peculiar. They send DNS queries from the vCenter IP address to the assigned DNS servers. Roughly 15% of the time, they will generate a query the same second with the same source port to both the primary and secondary DNS servers. When this happens, the firewall logs that the vCenter server responds to the secondary DNS server's answer with an ICMP type 3 code 3 (port unreachable). This clearly shouldn't be happening (the query to the secondary DNS server probably shouldn't even be happening). Any ideas on why this would occur?
vCenter Server with an embedded Platform Services Controller
Example firewall log entries:
access-list vm_interface_access_in permitted udp vm-interface/10.1.1.5(33201) -> dc_interface/10.1.3.1(53)
access-list vm_interface_access_in permitted udp vm-interface/10.1.1.5(33201) -> dc_interface/10.1.3.2(53)
No matching connection for ICMP error message: icmp src vm_interface:10.1.1.5 dst dc_interface:10.1.3.2 (type 3, code 3) on vm_interface. Original IP payload: udp src 10.1.3.2/53 dst 10.1.1.5/33201.
Same issue here. Surely vCenter shouldn't be sending out two different DNS requests from the same ephemeral port before the first request has been replied to.
My assumption is that it is closing the port on vCenter once it gets the first response (from which ever of the two responds quickest), so that the second response hits a closed port and then vCentre sends the ICMP port unreachable.
Packet Capture and related log of an example instance from me:
|3168: 13:41:37.191045||802.1Q vlan#111 P0 10.216.20.28.36228 > 10.200.223.6.53: udp 58|
|3169: 13:41:37.191152||802.1Q vlan#111 P0 10.216.20.28.36228 > 10.200.224.6.53: udp 58|
|3170: 13:41:37.192739||802.1Q vlan#111 P0 10.200.224.6.53 > 10.216.20.28.36228: udp 128|
|3171: 13:41:37.193639||802.1Q vlan#111 P0 10.200.223.6.53 > 10.216.20.28.36228: udp 128|
Jun 13 2020 13:41:37 %ASA-4-313005: No matching connection for ICMP error message: icmp src oob-lzb:10.216.20.28 dst oob-met:10.200.223.6 (type 3, code 3) on oob-lzb interface. Original IP payload: udp src 10.200.223.6/53 dst 10.216.20.28/36228.
Looks like a bug to me.
We are running vCentre Server Appliance 220.127.116.11000 (with embedded PSC)