ESX Networking Setup

[khenry567]

Got a networking question for all the experts out there about a certain networking setup. This is a network setup that I have not seen proposed on any docs before. The way that each host is networked is that there is only one vswitch.

Attached to this is the service console, vmotion, and production network. We have six pnics in each host. They are all teamed. We

need to attach to two different networks, one is our DMZ, the other is our Production Network. This is done using vlan ids. The pnics are attached to a Cisco 6500.

On the 6500 side there is a port channel for all 6 ports.

The idea behind this was to utilize as much band width as possible of every pnic. I thought it sounded great. Still do. We tested it

the day we installed it and it worked fine but of course now it doesn't. It worked then not now, we are not getting to the DMZ network thru the 6500's. I am

pretty sure it has to do with the way our Cisco's are configured and so is our networking guy. Something obviously changed between one day and the next.

Anyway my question is simply this: Is networking an ESX 3.0.2 host this way using only one vswitch present any obvious problems that you all are aware of? Comments please.

[Texiwill]

Hello,

You have 6 pNICS all attached to on vSwitch. This could be the issue from the start but I am not 100% sure. So please provide for us the configuration of your vswitch and pnics.

esxcfg-vswitch -l

esxcfg-vmknic -l

esxcfg-nics -l

This will help see how you are currently configured. A few issues that could be the case....

VMware nor anyone else recommends more than 2 pNICs per NIC Team. You do not get aggregation but either load balancing or failover. Load balancing has some pretty major issues with over 2 pNICs per link.

So in effect you may have 1 active link and 5 failover? Not a great use of resources.

THe suggestion has been the following without VLANs.

2 for SC (failover)

2 for vMotion (failover)

2 for VM Network

Or with VLANs, and 6 ports

2 for SC/vMotion VLANS

2 for Production VLANS

2 for DMZ VLANs

I would run everything in failover more, never Load Balancing. There is no gain in some cases.

Best regards,

Edward L. Haletky, author of the forthcoming 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', publishing January 2008, (c) 2008 Pearson Education.

[khenry567]

Hey Texiwill, thanks for the prompt reply. Here is the requested info.

# esxcfg-vswitch -l

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch0 32 11 32 vmnic5,vmnic4,vmnic3,vmnic2,vmnic0,vmnic1

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

Production Network portgroup1 10 1 vmnic1,vmnic0,vmnic2,vmnic3,vmnic4,vmnic5

DMZ portgroup3 101 1 vmnic1,vmnic0,vmnic2,vmnic3,vmnic4,vmnic5

VMotion portgroup4 0 1 vmnic1,vmnic0,vmnic2,vmnic3,vmnic4,vmnic5

Service Console portgroup0 10 1 vmnic1,vmnic0,vmnic2,vmnic3,vmnic4,vmnic5

# esxcfg-vmknic -l

Port Group IP Address Netmask Broadcast MAC Address MTU Enabled

VMotion 192.168.0.3 255.255.255.0 192.168.0.255 00:50:56:6e:d5:2c 1514 true

# esxcfg-nics -l

Name PCI Driver Link Speed Duplex Description

vmnic0 05:00.00 bnx2 Up 1000Mbps Full Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T

vmnic1 07:00.00 bnx2 Up 1000Mbps Full Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T

vmnic2 24:00.00 e1000 Up 1000Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic3 24:00.01 e1000 Up 1000Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic4 25:00.00 e1000 Up 1000Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic5 25:00.01 e1000 Up 1000Mbps Full Intel Corporation 82571EB Gigabit Ethernet Controller

[etieseler]

Hello,

I have to fully agree with Texiwill on spliting up your pnics.

Can you get to all the other networks? Can you ping devices on your production network, can you ping or access your service console, how about the VMotion addresses? Verify that routing is setup properly.

In our environment we set our Service Console to be VLAN ID 1, but this caused issues as we have a production network on the same VLAN so we were unable to access them. Maybe you are experiencing the same thing. When your configuring your vSwitch properties, see if the VLAN ID for Service Console is set to None, if its not then set it to None. What settings do you have for NIC Teaming?

But again, split up your pnics! Having 1 active and 5 fail over will not really help you. Especially if you want to use vmotion.

-Ed

[khenry567]

Thanks etieseler for your help here. To answer your questions:

I can get to the production network, I can vmotion, I can move vm's between clusters, but I can not get to the DMZ network.

I can ping everything except DMZ addresses.

Why set the vlan for the service console to 0? Do you have backup docs for this? Now it is set as 10.

All 6 nics are set as Active. None of the six physical nics are set as"Standby", all are listed in "Nic Teaming" as "Active".

From what both you and Texiwill have said even though I have all six physical nics teamed as "Active" I am not getting the aggregate

bandwidth of all six physical nics. Can anyone explain this to me please? Or point me to some doc that I can read and get some understanding.

Thanks all. I really appreciate this forum and the feedback. Hopefully I will be a contributor someday instead of just a taker.

[psharpley · Accepted solution]

Have a look at http://www.vmware.com/resources/techresources/997 and http://www.vmware.com/resources/techresources/412. Be aware that some configurations require external networking co-ordination to work. Also, you should consider keeping the VMotion network isolated for security reasons, see page 13 of http://www.vmware.com/resources/techresources/726.

Hope that helps.

[khenry567]

Thanks psharpley, appreciate your help. I will look at these docs.

[psharpley]

No problem, look into routing based on originating port id and based on

source-destination IP hash. You will be able to aggregate the combined

throughput of the bonded nics used by the VMs, but not all of the nics will

be used by a VM at the same time. Like I said, much depends on what you can

do with the physical switches and how many you have connected to the hosts.

[Texiwill]

Hello,

Re: Bandwidth

ESX does not aggregate, it does Load Balancing and Failover mode. Hence why using more than 2 pNICs per vSwitch is often a waste of pNIC. Also, when there are greater than 2 pNICs in load balance mode you could get up to 10 minutes of down time as the Switching fabric decides where to place MAC Address in the CAM and Arp Caches of the switches when a pSwitch/Cable fails. This is due to the way vSwitches work. Hence using more than 2 is not recommended. Actually VMware no longer even recommends Load Balancing. They suggest just using Failover modes.

The best thing is to split your pNICs across at least 3 vSwitches and use VLANs as well. This will aid in security as well as performance.

As for accessing your DMZ, that will require some form of router/gateway. Make sure you have the proper settings within the IP setup of each VM.

Best regards,

Edward L. Haletky, author of the forthcoming 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', publishing January 2008, (c) 2008 Pearson Education

[etieseler]

I do not have any documentation on why it does or does not work. In our situation, one of our production VLAN's is the native VLAN and the service console is a member of this VLAN. On the ESX Server we have two vSwitches, vSwitch0 contains VMotion and the Service Console, and vSwitch1 contains 3 production VLAN's. Perhaps it was giving us problems because native VLAN was spread to both vSwitches.

As you had suggested, it sounds like this may be in the switch. Have your network admin verify the routing table has a route to the DMZ and the ports assigned to the port groups allow the DMZ's VLAN ID. Have them do a 'show vlan' and make sure they are all listed and have the proper ports assigned to them.

Can VM's in the DMZ ping other VM's in the DMZ?

[khenry567]

Ok, guys, to of you who helped with this, thanks. We finally found the problem. It was with the vlan

setup of our 3500's and 6500's. The setup of the ESX's was correct. All vm's on the DMZ now work.