Virtual Switch & NIC configuration

[Apex1616]

Hi All,

I'm new to VMWare and I am in the process of designing my Virtual Switches with the Physical NIC's. We currently have 2 HP BL480c blades with one four port mezzanine card(NC325M) for each blade. That gives me 4 onboard Gig NIC's along with 4 more Gig NIC's on the mezzanine card for a total of 8 Gig NIC's. We will be buying more blades later, probably 2-4 more but for now we only have the two. Here is how I currently have it setup, let me know your thoughts and suggestions. I would post a picture to make it easier to see but I cannot figure out how to upload a picture. Thanks

VSwitch 1(DMZ, on subnet 100)

Onboard NIC 1-Active

Mez NIC 1-Standby

VSwitch 2(Virtual Machines and Service Console, on subnet 101)

OnBoard NIC 2-Active

Mez NIC2-Standby

VSwitch 3(VMotion on subnet 101, might VLAN this, thoughts?)

OnBoard NIC 3-Standby

Mez NIC 3-Active

VSwitch 4(iSCSI storage, on subnet 102)

OnBoard NIC 4-standby

Mez NIC 4-Active

Each NIC is connected to a separate Cisco 3750.

Each Cisco 3750 connected to the Equallogic PS400e SAN

Questions:

1. First off how does the configuration look and would you set it up any differently?

2. We are worried that we will have a bandwidth issue on the iSCSI NIC, is there any way to increase the bandwidth? I cannot use an HBA since they dont make one for the blades yet. Any other ideas? Anyone using iSCSI thru one gigabit NIC? How is the utilization of your NIC?

3. Should I configure a VLAN for my VMotion traffic since its on the same subnet as my Virtual Machine and Service Console traffic?

Thanks for your thoughts and time.

[Apex1616]

Forgot to add one more VSwitch

VSwitch 5 (Internal VM only)

[esiebert7625]

Two gigabit NIC's should be plenty for iSCSI, just team them in a vSwitch. Eight NIC's are plenty for one ESX server. I'm not a big fan of standby NIC's, if you got em, use em, make them both active so they can load balance and provide failover. Keep your Vmotion traffic off the same VLAN as your VM's. Isolate Vmotion and the SC on it's own VLAN.

VSwitch 1(DMZ, on subnet 100)

Onboard NIC 1- Active

Mez NIC 1- Active

VSwitch 2(Virtual Machines, use 802.1Q VLAN Tagging)

OnBoard NIC 2-Active

Mez NIC2-Active

VSwitch 3(VMotion, Service Console on subnet 101)

OnBoard NIC 3-Active

Mez NIC 3-Active

VSwitch 4(iSCSI storage, on subnet 102)

OnBoard NIC 4-Active

Mez NIC 4-Active

VSwitch 5 (Internal)

No pNIC's

Here's some good networking docs...

VMware ESX Server 3 802.1Q VLAN Solutions - http://www.vmware.com/pdf/esx3_vlan_wp.pdf

VMware Virtual Networking Concepts - http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf

Networking Virtual Machines - http://download3.vmware.com/vmworld/2006/TAC9689-A.pdf

Networking Scenarios & Troubleshooting - http://download3.vmware.com/vmworld/2006/tac9689-b.pdf

ESX3 Networking Internals - http://www.vmware-tsx.com/download.php?asset_id=41

High Performance ESX Networking - http://www.vmware-tsx.com/download.php?asset_id=43

Network Throughput in a Virtual Infrastructure - http://www.vmware.com/pdf/esx_network_planning.pdf

ESX Server, NIC Teaming and VLAN Trunking - http://blog.scottlowe.org/2006/12/04/esx-server-nic-teaming-and-vlan-trunking/

Integrating Virtual Machines into the Cisco Data Center Architecture - http://www.cisco.com/univercd/cc/td/doc/solution/vmware.pdf

Configuring iSCSI in a Vmware 3 environment - http://www.vmware.com/pdf/vi3_iscsi_cfg.pdf

Ethernet-based Storage Configuration - http://www.vmware.com/pdf/iscsi_storage_esx.pdf

iSCSI , NAS and IP Storage Configuration for Vmware ESX Server - http://download3.vmware.com/vmworld/2006/tac9722.pdf

Fyi…if you find this post helpful, please award points using the Helpful/Correct buttons.

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

Thanks, Eric

Visit my website: http://vmware-land.com

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

[VirtualNoitall]

"I'm not a big fan of standby NIC's, if you got em, use em, make them both active so they can load balance and provide failover"

Unless you have each nic going to a separate physical switch for increased redundancy and resiliency. Then Active/Standby is the appropriate configuration.

[VirtualNoitall]

VSwitch 3(VMotion, Service Console on subnet 101)

OnBoard NIC 3-Active

Mez NIC 3-Active

I would take this a step farther and setup the service console as active on one and vmotion active on the other. Then assign the other nic as standby for each. That way vmotion and sc traffic are isolated from each other except for when there is a failure.

[esiebert7625]

Sure thing, it does have it uses. Most shops using ESX tend to have more then one physical switch to connect there NIC's to.

[Apex1616]

To answer your question, we are a smaller shop and only have one Core Switch, a Cisco 4510. I was going to place the active NIC on one blade of the 4510 and place the standby NIC on another blade of the same 4510. So knowing we only have one core switch(besides the SAN switches), is Active/Standby the way to go?

Thanks!

[Apex1616]

I like this suggestion, thank you.

[Ken_Cline]

I would take this a step farther and setup the

service console as active on one and vmotion active

on the other. Then assign the other nic as standby

for each. That way vmotion and sc traffic are

isolated from each other except for when there is a

failure.

No need to go through the hassle of setting active/standby for this config. You have only two pNICs bound to the vSwitch, and you have only two consumers connecting to the vSwitch. The default load-balancing algorithm will put the first one to get powered up on one pNIC and the next one to get powered up on the other - no need to muck with advanced configuration options (KISS it!)

[VirtualNoitall]

Hopefully you don't really think that is a hassle or advanced.

[Ken_Cline]

Hopefully you don't really think that is a hassle or advanced.

Not really...but I do have a strong liking for a Really Simple Configuration

[esiebert7625]

With age comes wisdom (and the desire to keep things simple) </br> </br>

(sorry Ken, couldn't resist....)

[Ken_Cline]

With age comes wisdom (and the desire to keep things simple) </br>

</br>

(sorry Ken, couldn't resist....)</br></br>

NP Eric. Some of us are[/b]old enough to have a few gray hairs!

I kinda like the self-portrait you tagged on the end there...it adds character to the post

[Bryan70]

esiebert7625,

Nice writeup and set of reference URL's. Thanks!