Questions about Networking Best Practices (Relative to VM)

[TheRealJason]

I am looking for some answers and suggestions related to networking my ESX servers. We currently have 5 ESX boxes in 2 clusters. Each machine is a Dell 1950 with 2 onboard Nics, and a 2-port Intel card. Currently, one of the onboard NICs is dedicated to the service console, one for vmkernel, and the 2 Intels are for VM traffic. We have had a couple of instances of networking issues/reconfiguration during business hours that have caused host isolation, and in turn the VMs shutting down and being brought up on another machine. This made me think about it a little more, and question my current setup. What I think would make sense for me to do, is to instead have 2 vSwitches. Each one made up of one of the onboard NICs, and one of the Intel NICs, connected to different core switches. One vSwitch would have the service console and the vmkernel, and the other for virtual server traffic. So if we lost both onboards, or lost the Intel card, or lost a switch, VM traffic, vmKernel traffic, and service console traffic would still work. This raised a couple of other questions.

Is it possible for the vmKernel and the Service Console to share the same switch?

Will vMotion use BOTH adapters when it is transferring the memory of a running machine?

Is my plan reasonable? Is it what other people are doing? Is there better ways to build in networking redundancy into my VMWare environment?

[Dave_Mishchenko]

What you've suggested is a good idea - especially the part about the vswitch mixing the onboard and Intel NIC ports. It will work for you to have both the vmkernel and service console port on the same vswitch (there's no iSCSI / NFS data stores correct?) . When you do vmotion, it will only use one of the NICs in the vswitch, it wont aggregate the bandwidth of both.

Some folks will setup the vswitch such that the SC port is active on one NIC (and has the other as standby), and then do the opposite for the vmkernel port so that by default the traffic will be on set NICs. You might also consider using VLANs to isolate you vmotion traffic if you have this all on one LAN.

[ITThies]

We´ve bounded all NICs to one vSwitch.

All services, virtual machine ports, vmkernel (different ip subnet), and service console are connected to that vSwitch.

We had no problems with that on many disaster scenarios.

[ITThies]

apendix...

To isolate our VMkernel traffic from the Service Console / Virtual Machine traffic, we´ve used a different, not routed ip subnet for VMkernel.

[TheRealJason]

Ok, I guess to take another step back, we have two core switches, and we have a multitude of VLANs. The ports that handle the virtual machine traffic are trunked, while the service console and vmkernel ports are not. I am assuming that I CAN trunk those ports to seperate the VLAN traffic, at least from a logical standpoint? I had a VLAN set up initially for virtual machine traffic, but since using the trunking, I decided that the VLAN would be my vmkernel traffic vlan, and am working on moving the handful of servers I created there off to other VLANs.

There is currently no iSCSI / NFS data stores. I am running a eval iSCSI target virtual appliance, but only connecting to that from virtual machines and other physical boxes. No current plans for VMFS on it.

Without creating 2 vSwitches, how do you dictate which is the primary and standby adapters for specific traffic from the service console and from the vmkernel?

[Dave_Mishchenko]

You'll be able to trunk the port and then set specific VLANs for both the SC port and vmkernel port. To do so, you would select either port in the single vswitch, click on edit and then set the VLAN ID on the general tab. On the NIC Teaming tab, you'll have the option to specify which vmnic is Active and which is Standby.

[TheRealJason]

So how would I choose a seperate active and standby adapter for the vmkernel and service console? 2 vswitches with the same 2 NICs assigned to each?

[Dave_Mishchenko · Accepted solution]

You'll just have one vSwitch with the 2 NICs and the vSwitch will have both your SC and vmkernel ports. You'll edit each port seperately and on the NIC Teaming tab, you'll be able to set different active and standby NICs. See the attached doc for where to find this.

[TheRealJason]

Ahh, I see now. Thanks, the document really spelled it out.

I want to get deeper into the benefits/downsides to using multiple port groups inside of a virtual switch. Right now I don't really see the difference it makes, but I'll leave that for another post I guess.

Thanks for the help guys!