P2V VM's or newly created VM's can not ping outside of local network. Other's on the same ESXi host can ping without issues.

[mharvey92069]

I have a ESXi 5.5 host that has 3 NICs connected in different networks. These networks are untagged and switchport does not have any security enabled on it. I see the arp changes on my firewall and the ACLs allow all traffic. The VMs are Windows 2003 R2 servers. Others that I have converted from physical seem to work fine without issues however there are some that just don't want to seem to behave correctly. Looked at routes on VMs and they look identical. This happens on both newly created VMs from scratch, aka DVD ISO build, and P2V. Any ideas?

[SG1234]

are the vms able to ping with in themselves?

can you share the vswitch layout ?

what is the vm hardware version , host OS version , and the NIC types you are using for the VMs.

[mharvey92069]

The VM's that are in that same subnet can ping each other yes. For this example NS1 is not able to reach outside networks but both WEB131 and WEB141 can. WEB141 can ping NS1 but NS1 cannot ping WEB131.

VM version 8.

ESXi Version is ESXi 5.5.0, 1623387

OS's tried both Windows 2003 R2 x32 and Windows 2008 R2 x64.

NIC on VM is Flexible which shows as VMware Accelerated AMD PCNet Adapter.

[SG1234]

so this is a layer3 networking problem  -- what is the default gateway on these VMs ....

[mharvey92069]

The only IP address associated with the ESXI host is in 192.168.5.0/24 space. Do I need to give it an IP in each network?

‌

[mharvey92069]

It's strange how others work just fine though without it. I also tried to set an ip in the 192.168.5.0/24 space but it didn't ping outside too.

[SG1234]

since the VMS are on seperate networks you will need a router to route between the networks....

[mharvey92069]

I  do have a router that is routing traffic to these networks outside of the esxi environment. It is a Cisco ASA 5520. It has 3 dedicated ports which have the 3 seperate networks. I have listed them below. The ARP changes for the VMs are making it to the ASA, but traffic isn't flowing for some of the VMs. The ACL rules on the FW allow the traffic and we're working fine on the physicals. Keep in mind 2 VMs in same subnet work just fine, but the 3rd can't talk outside the subnet.

Ports:

if0=192.168.5.0/24

if1=69.43.192.0/23

if2=69.43.194.0/24

‌

Topology for VM:

ASA -> Switch -> VM Host

Topology for real:

ASA -> Switch -> Teamed NIC on Physical Server

[mharvey92069]

[SG1234]

even though there are all connected to the same external switch -- you need to use a gateway so that packet crosses the vswitch on the ESX - otherwise the packet is stuck in the vswitch ....

helps?

[mharvey92069]

The ASA is the gateway and has an IP address of .1 in all networks so all traffic should route correctly. Like I said earlier, the arp changes are making it to the ASA but doesn't seem to make it out of that network. I'll look at adding IPs to each subnet on the vswitches but it should be working in theory. There are multiple instances that this is not affecting. Will adding a gateway on the vswitch nat the traffic to another ip, as this would messup ACLs and applications. Am I missing anything?

[SG1234]

ok - got it  -- are the VMs able to hit the default gateway of the other network ?

[mharvey92069]

‌tthe ones that are working correctly yes. The ones that are not able to get out no.

[SG1234]

can you have the VM ping the management network on ESX and then use tcpdump-uw to watch the traffic.?

also try clearing the arp table on the switch

[mharvey92069]

Arp clear on switch didnt do anything. I tried pinging various GW and then pinged mgmt if of the private vkernel.

[mharvey92069]

Here is another capture done from all NICs. The machine having the issues is 69.43.194.15. Tried pinging various hosts.

[sanals]

After reading all the above posts, I was wondering about the ASA part. Did you try checking the logs on the firewall? You mentioned about the ACLs for the not-working VMs. Please check the ASA buffer logs, filtering for one of the VM IP addresses that are not working, and then try to ping the gateway IP address for that subnet (or any other external IP address) from that VM. Also please post a screenshot of the same.

[mharvey92069]

I checked the logs and no errors are reported. You have to remember this is a physical to virtual migration. If I power on the old physical machine t can reach networks just fine. I see arp changes on switch and ASA without errors. I am able to ping other devices on vswitch just not internet. I temp put in any/any rules just to be sure but same response.

[sanals]

You mentioned about the smooth arp changes otherwise I would have requested to try changing the MAC address of the VM to the respective physical one. Still, something tells me its either the ASA or the vswitch. Can you try moving one problematic VM to another subnet to see if it makes a difference ?

[mharvey92069]

IP/MAC change did not change the behavior. I'm wondering if it has something to do with ESXI 5.5...

‌