Strange IP issue with vLANs and subnets

[Canicula]

Hi, I'm setting up an internal VI3.5 system that will have about a dozen vLANs to separate various internal demo and test environments. I've got a problem were I can't ping to VMs or physical servers from one vLAN and IP subnet to the other. I can ping the default gateway for each vLAN. These are set-up on the Cisco switches. Here is the simplified set-up I currently have configured to try to troubleshoot this problem.

4 x Dell 2950 servers with 2 onboard NICS (0 and 1) and 4 NICs on a PCI Quad card (NICs 2 to 5)

EMC SAN connected vi Fibre. No iSCSI using any of the NIC ports.

1 x HP DL360 G5 running Windows 2003, SQL2005 and VirtualCentre. Also VI CLient

3 x Cisco switches. All ports that are in use are Trunk ports to allow all vLAN traffic.

Management vLAN is 10.102.193.0 with 255.255.255.0 mask. 193 network is vLAN 10

VMotion vLAN is 10.102.194.0 with 255.255.255.0 mask. 194 network is vLAN 20.

Demo vLAN 10.102.195.0 with mask 255.255.255.0. 195 network is vLAN 30.

The four Dell servers are running ESX 3.5 and have Service Console IPs on the 193 network and vLAN 10 (10,102.192.21, 22, 23 and 24)

The HP Server is also on the management vLAN and has an IP of 10.102.193.15

VIrtualCentre works fine and I can manage the 4 ESX servers etc as expected. I have 2 vSwitches setup on each server. 0 for Service Console (vLAN ID 10) and a port group for Virtual Machines (vLAN ID 30), 1 for VMotion VMkernal (vLAN ID 20). Service console access from VirtualCentre is fine. VMotion works. All licensing is fine.

I have a Windows 2003 VM with an IP of 10.102.195.101. I can VMotion this VM to any of the 4 ESX hosts. From the command prompt in Windows in this VM I can ping 10.102.195.1 (i.e the gateway of the 195 network). This gateway is configured in the vLAN settings on the Cisco switches. i Can also ping 10.102.193.1 (i.e. the gateway of the 193 management LAN). However I can't ping any devices on the 193 LAN. Not the management server on 10.102.193.15, nor any of the ESX Service console IP addresses.

The converse is try as well. From the command prompt on the Management server on the 193 address I can ping the 10.102.195.1 address but not the actual VM running with an address of 10.102.195.101. On the 193 network I can ping the other devices on the same network. When I had a 2nd VM running on the 195 network on IP address 10.102.195.102 I was able to ping between the 2 VMs on that that network.

So I can ping between devices within each IP subnet, and can ping the gateway address of other subnets from them all, but I can't ping any devices running on the two subnets from devices on the other one. The Cisco guys assure me the vLANs etc and ports on the switches are configured okay. They can telnet into the switches and ping the various gateways etc.

I'm baffled. Anyone any ideas or pointers (assuming you follow my ramblings above)?

Cheers,

Ian

[Texiwill]

Hello,

Perhaps a drawing would help? However, where in all this is the gateway? Is it a router that is routing properly between VLANs? That is where I would look first as unless there is a gateway/router device routing between VLANs, then there is no way for them to cross.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[Canicula]

Hi Edward. Yes the Cisco switches are routing between the VLANS. So I'm assured by by our Cisco admins anyway.

Ian

[Texiwill]

Hello,

If you can not ping between VLANs then it is not routing as far as I can tell. If you can 'ping' the gateway from the VMs but not between VMs on different VLANs then it is the gateway that could be the issue. Or there is a route issue within the VMs. I would check everything from one VM to another. At this point ESX is not really involved too much other than to provide a Layer 2 switch but once you are out of it and can ping the gateway it is not an ESX issue anymore but a routing problem of some sort.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[Cameron2007]

I had a similar problem previously and on the Vswitches you could try adding the VLAN ID as 4095 which should allow all networks. This may allow you to ping across.

[Canicula]

Thanks Guys. I'll try the 4095 vLAN ID tomorrow. I've gone home in disgust for today

Ian

[Texiwill]

Hello,

VLAN 4095 is used within the vSwitch for IDS/IPS, it allows all VMs on that VLAN ID to see all traffic for every other VLAN. THis is not really what you want... It looks like you have a routing issue is all.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[Canicula]

This is now fixed. Our network team logged it with Cisco who said it was a known issue with the IOS. They told them to make some changes to the IOS in use. The pinging now works across vLANS. I'll try and get more detailed info for the archive just in case anyone else gets a similar issue in future.

Thanks for the help to all. Time to get some work done now I can communicate between the vLANs...

Ian

[ngarira]

hi canicula,

i am in the same probllem as you were

kindly send to me the solution to my address james.ngarira @ke.dimensiondata.com

i will highly appreciate