Problem Networking with Crossover Cable from ESXi 5.1 to NAS Server

[brandonpoc]

Hello everyone! My first post, but I've been lurking and gaining knowledge and practical advice from the forum. The past few days and nights have been dedicated to becoming intimate with VMware ESXi and it's internals and playing with different setups. I'm definitely moving from other virtualization solutions (OpenVZ, Xen) to VMW. That is, as long as we can figure this out! 😃

Anyway --

In an attempt to speed-up and dedicate my SAN access, I have created a dedicated "SAN" vSwitch on my ESXi 5.1 server and assigned it a dedicated gigabit NIC with which I have connected a SAN (FreeNAS) server via crossover ("patch") cable. Both ESXi and the FreeNAS server recognize that the link is up. I have assigned this SAN network to several guest VMs running on the ESXi server and all have been assigned IP addresses on the same IP network and subnet as the interface on the FreeNAS server. The ESXi Server and FreeNAS Server are two separate physical machines and FreeNAS server has no virtualizer.

I am unable to transfer network traffic between the VMWare ESXi guests and the FreeNAS server via that crossover connected gigabit ethernet link. I can ping and initiate transfers intra-host from guest to guest that share the SAN vSwitch interface, but NOT to the external machine. I have no tried connecting it through a hub, but in my experience the crossover cable should be just fine and provide the functionality needed for computer-to-computer ethernet connectivity.

I tried adding a VMKernel interface to the SAN network and assigned VMWare an IP on the same subnet but this did not work either.

For reference, I have assigned the following to the "SAN" virtual machine network (vSwitch1, assigned vmnic1). The three guests are able to communicate. They also have connectivity through eth0 to the LAN and the Internet via a local gateway at 10.200.99.1.

• guest1 at IP 10.200.100.5 (eth1, 255.255.255.0 no gw/routing assigned)
• guest2 at IP 10.200.100.6 (eth1, 255.255.255.0 no gw/routing assigned)
• guest3 at IP 10.200.100.7 (eth1, 255.255.255.0 no gw/routing assigned)

I have the following assigned to my FreeNAS:

• IP 10.200.100.1 (interface re1; 255.255.255.0 no gw/routing assigned), with re0 being the LAN interface with a  gateway and LAN IP.

As evidenced here, all IPs and subnets are correct, but they do not communcate! Does anyone have any idea how to make this work?

Thanks in advance for your consideration on this matter :smileygrin:.

Brandon Edward

[MannyS]

AFAIK, this will not work. Transfer between the guests and storage doesnt work that way. You create datastores on the host and the guests use those datastores. How are you trying to access the guests? Why would you want to transfer between the FreeNAS and guests the way your trying to. Just curious.

[brandonpoc]

In this sense, it is just network connectivity. Ignore the storage aspect of it. I cannot do anything, let alone store data, from the NAS server to the VMWare server when connected via crossover cable NIC-to-NIC. Each is assigned its address properly yet it does not work. Guests access the LAN network just fine and can access outside of their vSwitch to the Internet via the LAN's router or other machines on the LAN, for instance, yet the second interface (and vSwitch) cannot access the machine connected directly to it.

And, to respond to your statement, I do have the guests using storage on the network via NFS and iSCSI (both datastore types) to the same server; besides, the guests can mount whatever they want as they are full-fledged operating systems with network access, so they can mount NFS, Samba/CIFS, AFS, WebDAV, whatever over the network to themselves. The reason I do it like this is because I have special containers, stored in files, that require a program on the guests to access the data held within the NAS accessed by NFS. I cannot provide this directly with VMWare. Instead of plugging all of the disks into VMWare I decided on building out a storage server so that I could access the disks directly for speed so there is no VMware overhead, needed to access more than 2.2TB of data as there are 5x4TB drives in the main ZFS storage volume; etc.

So, to re-iterate, the problem has nothing to do with storage-- instead it has to do with the inability of the guests to communicate outside of the VMWare ESXi host itself on vSwitch1, which is assigned to a physical NIC and connected via crossover cable to another machine. The guests communicate amongst themselves just fine on the IP network (netmask 255.255.255.0, aka CIDR /24) but NOT the machine connected to it!

Thanks!

Brandon

[MannyS]

Thanks for the detailed response. Yep, guests wont speak with the host like that. Dunno how familiar you are with vSphere, but what you are doing is called Raw Device Mapping, so that presents the storage straight to the guest bypassing the virtualization layer and is good for the reasons you mentioned.

Again, dont know how much you want to lab up, but you can nest ESXi inside of ESXi, that way you can go create shared datastores, play with vMotion, DRS/vCenter/Fault Tolerance/SDRS, in other words all the good stuff. Just download the evals and away ya go!

Have fun labbing!

[brandonpoc]

No, no -- this is again nothing to do with storage at it's root. I just brought it up because that's what it's plugged into, but that's not relavant here. Just forget the storage, honestly, it's about guests being able to access the network one of their vmnic devices are on.

Here's the scenario:

Imagine you have an extra NIC, aside from vmnic0, plugged in to the ESXi server, called vmnic1, and you added it to a new vSwitch called vSwitch1. On vSwitch0 you have the primary network card vmnic0, but on this one you have vmnci1. You then assigned that NIC to two virtual machines, and plugged the NIC physically into another machine with a cross-over cable. You assign Guest 1 and Guest 2's second network card with 192.168.100.1 and 1921.68.100.2, respectively, and the other machine (or, located on a physical switch, whatever) you assign 192.168.100.3. You would expect to be able to communicate with 192.168.100.3 from .1 and .2, and vice-versa. But in my case, I am only able to communicate between .1 and .2 -- .3 is isolated for some reason.

After all, the exact same thing happens with vmnic0 when communicating on the network -- everything plugged in, the guests are able to see.

Make sense now? :smileygrin:.

So, I am wondering WHY the guests aren't able to get traffic to the other machine connected via cross-over cable on the second NIC (vmnic1). I may try plugging it into a physical switch here in a moment, should this not be resolved, but it shouldn't make a difference -- shouldn't. Because the crossover cable is supposed to be doing what it's supposed to do!

And no time for labbing, I have to get this production setup done, first, the labbing was the past 4 nights! :smileygrin:

Thanks,

Brandon

[brandonpoc]

I still haven't had any success with this -- anyone know what be preventing these from communicating?

[MannyS]

I think I see light now!

What's your load balancing set too on the vSwitches? Are the vmnic's on standby by chance?

[a_p_]

Before going into details about the vSwitch and VM network setup, disconnect the ESXi host from the storage, then use e.g. a notebook with an IP address in the storage subnet and connect it to the storage to see whether you can communicate with it. If this works connect the notebook to the ESXi network port to see what happens. (always using the Cross-Over cable). With this done you know which part you can rule out from trouble-shooting.

André

[brandonpoc]

I tested just as you suggested, and indeed I am able to communicate between my laptop and the NAS server via cross-over cable by ICMP, TCP, etc and it works just fine. However when I plugged my laptop into the ESXi server, I was unable to communicate with any of the virtual machines that are assigned to the second vSwitch to which the second NIC is assigned.

Odd.

Any ideas?

[a_p_]

Please post a screen shot of the ESXi host's current virtual network configuration (vSwitches and Port Groups). This might make it easier to troubleshoot.

André

[brandonpoc]

See the following screen-shot:

That's my networking setup; I have that second network, "SAN", connected via cross-over cable to my other machine.

Thanks,

Brandon

[a_p_]

From the ESXi side, the setup seems to be ok (assuming you are using the default policies for the vSwitch and port group). What I could think of is a wrong routing within the "Susie" VM, i.e. not sending the storage traffic through the correct virtual network adapter. Did you already check this?

André

[rickardnobel]

From inside the VM, when you do for example a ping, do you get any ARP reply from the NAS device?

Does the VM has two IP addresses?

Could you try to attach a vmkernel interface to the vSwitch1 and set a unused IP address in the 10.200.100.x scope and then try to ping from the ESXi shell?

[brandonpoc]

The VM indeed has two IP addresses -- the main interface, which also has a gateway set, is on a LAN and has an address in 10.100.0.x/255.255.255.0 while the second interface connected cross-over is on 10.100.1.0/255.255.255.0 with no gateway. I am not seeing any traffic whatsoever in tcpdump or anything (and yes, I enabled promiscuous mode in VMWare for the vSwitch1 :smileygrin:). When I attempt to ping from the other machine connected on the cross-over, it should receive it and send it back out that interface as it is coming in that interface and has a source and destination IP on that interface. I tried setting up a VMKernel for management traffic with an IP of 10.100.1.2 and it didn't work, but I will give it another go right now.

OOPS: Forgot to mention, the IPs are different because I changed my network scheme around. They all match from host to host. No typos or anything, and the netmasks are consistent across each interface.

[rickardnobel]

Good that you have doublechecked the netmasks, since that could in some situations create the effect you see. You have also verified the NAS stations mask as well?

When you run tcpdump from the VM on the interface attached to vSwitch1, do you see any traffic at all? If having a tcpdump filter like "icmp or arp", (and clear the guest OS arp cache first), do you see the ARP request broadcast leave the VM?