VMware Cloud Community
rsmith42
Contributor
Contributor
‎12-17-2019 12:55 PM

vCenter 6.7 generating ICMP port unreachable for DNS query responses

Hello folks,

I have two installations with vCenter installed on them that are both doing something that seems peculiar.  They send DNS queries from the vCenter IP address to the assigned DNS servers.  Roughly 15% of the time, they will generate a query the same second with the same source port to both the primary and secondary DNS servers.  When this happens, the firewall logs that the vCenter server responds to the secondary DNS server's answer with an ICMP type 3 code 3 (port unreachable).  This clearly shouldn't be happening (the query to the secondary DNS server probably shouldn't even be happening).  Any ideas on why this would occur?

vCenter Server with an embedded Platform Services Controller

v6.7.0.30000

Build 13010631

Example firewall log entries:

access-list vm_interface_access_in permitted udp vm-interface/10.1.1.5(33201) -> dc_interface/10.1.3.1(53)

access-list vm_interface_access_in permitted udp vm-interface/10.1.1.5(33201) -> dc_interface/10.1.3.2(53)

No matching connection for ICMP error message:  icmp src vm_interface:10.1.1.5 dst dc_interface:10.1.3.2 (type 3, code 3) on vm_interface.  Original IP payload:  udp src 10.1.3.2/53 dst 10.1.1.5/33201.

Thank you!

0 Kudos
1 Reply
relmes
Contributor
Contributor
‎06-13-2020 07:00 AM

Same issue here.  Surely vCenter shouldn't be sending out two different DNS requests from the same ephemeral port before the first request has been replied to.

My assumption is that it is closing the port on vCenter once it gets the first response (from which ever of the two responds quickest), so that the second response hits a closed port and then vCentre sends the ICMP port unreachable.

Packet Capture and related log of an example instance from me:

3168: 13:41:37.191045   802.1Q vlan#111 P0 10.216.20.28.36228 > 10.200.223.6.53:  udp 58
3169: 13:41:37.191152   802.1Q vlan#111 P0 10.216.20.28.36228 > 10.200.224.6.53:  udp 58
3170: 13:41:37.192739   802.1Q vlan#111 P0 10.200.224.6.53 > 10.216.20.28.36228:  udp 128
3171: 13:41:37.193639   802.1Q vlan#111 P0 10.200.223.6.53 > 10.216.20.28.36228:  udp 128

Jun 13 2020 13:41:37 %ASA-4-313005: No matching connection for ICMP error message: icmp src oob-lzb:10.216.20.28 dst oob-met:10.200.223.6 (type 3, code 3) on oob-lzb interface.  Original IP payload: udp src 10.200.223.6/53 dst 10.216.20.28/36228.

Looks like a bug to me.

We are running vCentre Server Appliance 6.7.0.44000 (with embedded PSC)

0 Kudos