ESX network planning help

[spieg]

Im hoping that someone can assist me in planning my esx network by answering a few questions. I currently am looking to set up 4 esx hosts. Each host has 10 vmnics available (2 onboard, 2x quad port intel). I have 2 cisco 3750 switches available(currently stacked). Ive read several docs and tried to establish all best practices with performance and resiliancy in mind. Based on that, so far my thoughts are:

Create vswitch0, create port group management, add vmnic2 and vmnic6 to this port group, connect vmnic2 to 3750-1, connect vmnic6 to 3750-2

Create vswitch1, create port group nfs, add vmnic3 and vmnic7 to this port group, connect vmnic3 to 3750-1, connect vmnic7 to 3750-2

Create vswitch2, create port group vmotion, add vmnic4 and vmnic8 to this port group, connect vmnic4 to 3750-1, connect vmnic8 to 3750-2

Create vswitch3, create port group virtual machines, add vmnic5 and vmnic9 to this port group, connect vmnic5 to 3750-1, connect vmnic9 to 3750-2

My questions are, can/should each of these exist in their own subnets? My virtual machines will be tied to my production network, 172.16.0.0/16. Can I set up each of the others (management, nfs, and vmotion) on their own /24's or should they all be on the same subnet? If they are each on their own, how does the gateway configuration tie in? Which port groups need to communicate in order for drs, ha, etc to work? Is there any reason to have multiple service consoles? Any other tips suggestions would be greatly appreciated!

[azn2kew]

For the networking configurations with 10 NICs I would use:

1. vNIC0->Service Console

2. vNIC1->VMotion

3. vNIC3->Service Console

4. vNIC4->VMotion

5. vNIC5-6>VM Network

6. vNIC7-8->DMZ Network

7. vNIC9-10->Backup/iSCSI/NFS/ or Spare Networks

VMkernel port groups is what you use for VMotion, iSCSI, NFS traffic. And you should secure these port groups seperatedly using VLAN. The Management Network (SC/VMotion) should be secure as well because VMotion traffic is clear text and can be vulnerable. Using multiple Service Consoles port provided redundancy for heartbeat failover. If one of the SC NIC is dead, you still have other port to function and ESX will not be isolated and restarted your virtual machines and it should have 2 SC ports. The gateway is configure to whatever your production/test/dev network configuration set to and you can consult with networking for those details. Here is a great blogs about networking you can check out its pretty long and good readings. I should also suggest you read basic networking from and good luck to your designs.

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

VMware vExpert 2009

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant

[Texiwill]

Hello,

My questions are, can/should each of these exist in their own subnets?

In general yes and in reality the SC, vMotion, NFS networks MUST be on different subnets.

My virtual machines will be tied to my production network, 172.16.0.0/16. Can I set up each of the others (management, nfs, and vmotion) on their own /24's or should they all be on the same subnet?

For security reasons you would do the following:

Management on its on network

VMOtion on its OWN Network

NFS on its OWN Network

If they are each on their own, how does the gateway configuration tie in?

Management/SC uses the gateway defined within the Service Console

VMotion uses the default Gateway entered for the the vmkernel ports

NFS uses a route based on the network address and mask setup for the vmkernel port.

Which port groups need to communicate in order for drs, ha, etc to work?

HA -> Management

DRS uses Management for Communication but VMotion Network for actually DRS MIgrations.

Is there any reason to have multiple service consoles?

No reason at all unless you want more redundancy for HA.

Any other tips suggestions would be greatly appreciated!

I would review these Specific Blogs as well as these ESX/ESXi white papers.

Best regards,

Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009, Virtualization Practice Analyst[/url]
Now Available: 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing the Virtual Environment'[/url]
Also available 'VMWare ESX Server in the Enterprise'[/url]
[url=http://www.astroarch.com/wiki/index.php/Blog_Roll]SearchVMware Pro[/url]|Blue Gears[/url]|Top Virtualization Security Links[/url]|Virtualization Security Round Table Podcast[/url]

[MHAV]

If you run ESX 4 and you want to use the Fault Tolerance (FT) Feature don't forget to create a seperate VLAN and NIC for FT.

Regards

Michael Haverbeck

If you find this information useful, please award points for "correct" or "helpful".

[jbogardus]

Not much has been discussed her yet about the use of multiple vSwitches. Using multiple vSwitches can be a simpler way of understanding the setup, but for greater flexibility and understanding you may want to consider using the full functionality of port groups on a single vSwitch. If you look into this way of setup you'll gain a better understanding of VMware networking.

You can have all 10 pNICs connected to a single vSwitch with VLAN trunking on all 10 physical switch ports to support all required VLANs with each NIC.

When setting up each port group or VMkernel connection use the VLAN setting to define the VLAN used by the port group, and use the Load Balancing tab to set up the specific NICs you want the port group to use.

So for example you could use the first 2 NICs for Service Console and VMotion. For SC make vmnic0 active and vmnic1 standby. For VMotion make vmnic1 active and vmnic0 standby.

For nfs make vmnic2 - 5 acitve, vmnic6-9 standy

For VMs make vmnic6 - 9 active, vmnics 2-5 standby

When considering this type of layout consider which physical adapter and physical switch each vmnic connection is using in you setup. Try to create each of the port groups to have connections on different adapters and different physical switch for best redundancy.

[Texiwill]

Hello,

Not much has been discussed her yet about the use of multiple vSwitches. Using multiple vSwitches can be a simpler way of understanding the setup, but for greater flexibility and understanding you may want to consider using the full functionality of port groups on a single vSwitch. If you look into this way of setup you'll gain a better understanding of VMware networking.

Actually quite a bit has been written about using multiple vSwitches. The problem is the # of pNICs often dictates the number of vSwitches.

You can have all 10 pNICs connected to a single vSwitch with VLAN trunking on all 10 physical switch ports to support all required VLANs with each NIC.

You can, from a security perspective it is not recommended.

Some people do this however. Not sure I would.

Best regards,

Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009, Virtualization Practice Analyst[/url]
Now Available: 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing the Virtual Environment'[/url]
Also available 'VMWare ESX Server in the Enterprise'[/url]
[url=http://www.astroarch.com/wiki/index.php/Blog_Roll]SearchVMware Pro[/url]|Blue Gears[/url]|Top Virtualization Security Links[/url]|Virtualization Security Round Table Podcast[/url]