help with network planning

[voodoo77]

Hello everyone,

I'm new to Vmware, and I'm still in the documenting process, but I'm trying to accomplish a vmware infrastructure network.

I have the following hardware :

Dell Equallogic PS6000XV - SAN - 4 NICs

2x Powerconnect 6224 24 GBE ports, managed switch

1x PowerEdge 1950 1U Quad Core Xeon with 4GB of RAM - Vcenter

3x PowerEdge 2950 2U 2x Quad Core Xeon with 64GB of RAM - 2 x quad NICs + 2 onboard - for ESX

Vmware Enterprise Edition (Mid-size acceleration kit)

Based on this, I'm planning to use 3 vlans on the physical switches, as follows :

pSwitch - vlan purpose - vSwitch

vlan 1 - production network (VMs) - vSwitch0

vlan 2 - vMotion and iSCSI traffic - vSwitch1

vlan 3 - vCenter and ESX management - vSwitch2

In the idea of having a redundant network, both switches will have fiber optic uplinks, and I'm planning on using the following NIC allocation

SAN - will have 2 NICs connected in each switch

ESX - 2 onboard NICs - vlan3 - SC and management network

- 2 NICs - vlan2 - iSCSI and vMotion network

- 2 NICs - vlan1 - production network (VMs) ( I know I can use more than 2 NICs here)

Please correct me if I'm doing something wrong.

Also, I suppose the vCenter system should have 2 NICs, one connected in vlan3, for managing the ESX hosts, and another one, connected in vlan1, in the production network, having a public IP where I can connect remotely. Is this right ?

Thank you.

[AndreTheGiant]

The network design can work.

Iscsi network must not be used for Vmotion o VM networking.

Move Vmotion in vlan3.

For remote management the best solution is a VPN, or NAT specific port.

Remeber also that the single VM console is going directly on the single ESX.

Andrea

[Texiwill]

Hello,

Welcome to the forums.

pSwitch - vlan purpose - vSwitch

vlan 1 - production network (VMs) - vSwitch0

vlan 2 - vMotion and iSCSI traffic - vSwitch1

vlan 3 - vCenter and ESX management - vSwitch2

In the idea of having a redundant network, both switches will have fiber optic uplinks, and I'm planning on using the following NIC allocation

SAN - will have 2 NICs connected in each switch

ESX - 2 onboard NICs - vlan3 - SC and management network

- 2 NICs - vlan2 - iSCSI and vMotion network

- 2 NICs - vlan1 - production network (VMs) ( I know I can use more than 2 NICs here)

In general people do not combine iSCSI and vMotion, both can be big hitters when they are in use.... SC/vMotion is often combined using multiple VLANs.

Also, I suppose the vCenter system should have 2 NICs, one connected in vlan3, for managing the ESX hosts, and another one, connected in vlan1, in the production network, having a public IP where I can connect remotely. Is this right ?

No you only need one for vCenter.... You would have a firewall between the Management network and your production network. The most secure thing would be to create virtual machines for the administrators to use. They would RDP into those VMs to run the VIC, etc. That way you only need to open up AD and RDP ports on the Management network firewall. Using the VIC from your production network is not recommended but often done.

Check out the following:

http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf

http://www.vmware.com/pdf/esx3_vlan_wp.pdf

Topology Blogs

The Great vSwitch Debate

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009, DABCC Analyst
====
Now Available on Rough-Cuts: 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing ESX and the Virtual Environment'
Also available 'VMWare ESX Server in the Enterprise'
SearchVMware Pro|Blue Gears|Top Virtualization Security Links|Virtualization Security Round Table Podcast

[proden20]

The design is fine. Maybe consider 2 vmkernel virtual switches (one for vmotion, one for iscsi), as you probably have the nics and ports available.

Your vcenter will most likely need two nics (or routing setup), so that Windows domain communication can take place (as your service console vlan probably won't route to it.) Use LACP at your switch for teamed nics.

I saw a post yesterday with a warning about using VLAN1 in Cisco environments...you might want to take a look at it:

http://communities.vmware.com/thread/208468

[habibalby]

Hello,

It's a best pracitce to keep the Service Console behind a firewall in seprated network. Create a VM portgroup in vSwitch2, and place a firewall with 2 vNICs 1 from vlan1 "External" and1 from vlan3 "Internal" vCenter and SC network. Beside that, you can put the vCenter in the same ESX Network, open the required ports between the Production and ESX "Internal DMZ" for your administrative purpose. Put don't forget, the vLAN Tagging in the PortGroup.

Best Regards,

Hussain Al Sayed

If you find this information useful, please award points for "correct" or "helpful".

[voodoo77]

Hello proden20,

It's true, I will not be using vlan1 for my setup.

My vlans will be vlan101, 102 and 103. My initial setup was considering having a separate vlan for iSCSI trafic, and another vlan for vMotion, but someone who knows better than my how Vmware Infrasture works, recommended me to use them in the same vlan.

I am planning to use externam vlan tagging, instead of virtual vlan tagging, I've heard that using virtual vlan tagging can cause a major speed drop.

I'm still having doubs configuring the network, I will not be physicaly there, I need to do my setup remotely. I need to know exactly how to setup the vlans on the switches and what NICs goes connected in which switch port, I will send this information to the ISP I will be using. I also don't know how to get access to the windows box containing the vCenter, after ISP completes the initial setup, so I can get it from there and complete the Infrastucture setup.

This is why I turned to the forums, in idea that someone with more knowledge than me, can give me a hand with the setup. This doesn't mean I dropped dead searching the forums and reading the documentation/manuals.

Thank you all for your time.

[proden20]

this is interesting. can someone give me more info as to why virtual vlan tagging is not recommended? It is a very nice feature that allows you to be very flexible with your network, particularly in deployment test and new server subnets...

As far as your config goes, that will get interesting when identifying nics after esx is installed. sometimes its not so obvious when creating virtual switches.

[Texiwill]

Hello,

There are a few problems with what you suggest. Your vCenter server should not be a gateway device, leave that to a proper firewall. The best solution is to have ESX service console + vCenter <-> FW <-> Other networks. Makign vCenter your gateway will lead to multiple attack points and a non-firewall device living in both security zones which is exactly what an attacker would seek out as a juicy target.

With 6 pNICS if you have IP Storage, it is best NOT to mix vMotion and iSCSI on the same vSwitch/cable but you can mix SC and vMotion using VLANs. That is the best practice.

VLAN are not 'not recommended' for use, they are however not recommended to achieve best security, as there are too may Layer-2 attacks against the physical network. In addition the 802.1q standard does not include security as part of VLANs, so use of VLANs is a TRUST issue. I.e. do you TRUST you will never experience a Layer 2 attack and that your physical switches protect you.....

This is a very large question to answer and in effect is your level of TRUST and may not be real security.


Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009, DABCC Analyst
====
Now Available on Rough-Cuts: 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing ESX and the Virtual Environment'
Also available 'VMWare ESX Server in the Enterprise'
SearchVMware Pro|Blue Gears|Top Virtualization Security Links|Virtualization Security Round Table Podcast

[voodoo77]

Hello Edward,

And thank you for your kind reply.

I want to clarify some things. I do not intend to use the vServer as a gateway for my network. I am just looking for a temporary solution to gain remote access to the vCenter until I manage to complete the Infrastructure setup. I am in a remote location and I need to build a plan for my ISP to properly setup the switches (I have 2 for redundancy), the ESX hosts (3) and the SAN.

Remote access to the vCenter will be for a short period of time, after the VI setup is complete I will hide it behind the production network.

So basically, I want a reliable network plan to have a strong network architecture, using redundant switches, and somehow, enable temporary remote access to the windows box holding the vCenter.

Using vlan it's the delicate matter. I'm not sure what's the best option, use external switch vlan tagging or use the virtual switch vlan tagging. This is why I came to you, I need opinions, and I hope I will get them.

Since I'm under pressure at this moment, using virtual switch tagging, may seem like a good option, and definitely ease my work.

However, I knew it's better to separate vMotion from iSCSI traffic, but this was not my decission, I trusted someone better than me, but since my plan it's not completed, I can make the change ongoing.

I am determined to read and search the forums until I find my answers, because I like the whole Infrastructure concept and its powers of manipulating such huge amount of processing power and data.

Thank you all for having patience with me.

[habibalby · Accepted solution]

Hello,

As suggested earlier, it's better to make a Back-to-Back DMZ to place your Hosts & vCenter behind it, after that you can enble TCP 3389 RDP from your production to vCenter server which is located behind the firewall.

I have done the same setup excactly:

• 4 Hosts, each with 20 gig of ram, 6 pNICs, 2 Dual-Process 3.0 GHz & single FC HBA Dual Ports.

• 2 pNICs for SC, vMotion & vCenter

• 2 pNICs for Production

• 2 pNICs for DMZ

• I have created a Trunk VLANs on my Clustred pSwitches and dedicate vmnic0 & vmnic1 -&gt; vSwitch0 and I have enabled physical Tagging on the ports.

• vmnic0 connected to pSwitch1 & vmnic1 connected to pSwitch2

• on vSwitch0, I have created 3 PortGroups

• Service Conole

• vMotion

• vCenter

• After connecting the hosts to the SAN, I have created 2 VMs, 1 for vCenter and 1 as an MS ISA Firewall, both mapped to vCenter PortGroups

• I have assigned MS ISA Firewall 2 vNICs, 1 from vCenter and the other from Production PortGroup, so I can get proper routing and Natting to reach to both networks via limited required ports that controlled by MS ISA Firewall, this assuming you know how to configure a Back-to-Back firewall

• I have allowed only RDP to vCenter Server from production to ESX Network.

• To get rid of DNS and other stuff issue, I have configured the vCenter Server to be as a DNS server for all the network that's behind the production. So, ESX Hosts can get proper name resolution.

This setup works perfectly without any issue, but one think I would like to suggest, not to use Virtual Firewall why?? because you will have to open other ports such as for your Backup server if it's setting in your production network, NTP Server, ect ect. As soon as load comes on your ESX Network, you will see the Virtual Firewall eat lots of CPU and memory resources. That's why i'm planning to change this Firewall with a physical Firewall, to gain this resources for something else.

Don't build your setup and later change your network archticture, once you change your setup, you will have to disconnect your hosts from the network which results = down your VMs loosing services.

Best Regards,

Hussain Al Sayed

If you find this information useful, please award points for "correct" or "helpful".

[Texiwill]

Hello,

I want to clarify some things. I do not intend to use the vServer as a gateway for my network. I am just looking for a temporary solution to gain remote access to the vCenter until I manage to complete the Infrastructure setup. I am in a remote location and I need to build a plan for my ISP to properly setup the switches (I have 2 for redundancy), the ESX hosts (3) and the SAN.

I see what you mean, I would not even use it as a temporary solution. I would put up a virtual firewall you can use temporarily or permanently.. I use a virtual firewall and have it setup to handle this scenario. I also pass my NTP, DNS, and other traffic through the firewall. I really do not like maintaining multiple DNS/NTP servers. A physical firewall is definitely better for larger environments but virtual ones work quite well.

Remote access to the vCenter will be for a short period of time, after the VI setup is complete I will hide it behind the production network.

You will still need that remote access to manage the system. There are times that I go to the vCenter server to do some things, and other times I use the VIC. However, all management tools should be used behind the management network. I would not place the VIC within any other environment. Instead create an administrative virtual machine on which the VIC resides. Then RDP to that VM through the firewall. Routing VIC traffic through your firewall is not really the way to go.

So basically, I want a reliable network plan to have a strong network architecture, using redundant switches, and somehow, enable temporary remote access to the windows box holding the vCenter.

As stated you will need more permanent access. So

2 pNICs 1 for SC, 1 for VMotion each backup for the other using VLANs or even subnets.

2 pNICs for IP Storage

2 pNICs for VMs

Using vlan it's the delicate matter. I'm not sure what's the best option, use external switch vlan tagging or use the virtual switch vlan tagging. This is why I came to you, I need opinions, and I hope I will get them.

I would not use VLANs except for SC/VMotion and whatever is within the VM network. For IP Storage use its own set of physical switches. SC/Vmotion should use its own set of physical switches and such for the VMs. If you can not do this then segregate everything as best as possible.

Since I'm under pressure at this moment, using virtual switch tagging, may seem like a good option, and definitely ease my work.

However, I knew it's better to separate vMotion from iSCSI traffic, but this was not my decission, I trusted someone better than me, but since my plan it's not completed, I can make the change ongoing.

Well iSCSI and VMotion is a real bad idea. Not sure what they were thinking, but alas you have the chance to change it now.

I am determined to read and search the forums until I find my answers, because I like the whole Infrastructure concept and its powers of manipulating such huge amount of processing power and data.

Your network design is very important moving forward. I personally use EST whenever I have to do VLAN tagging. It makes it easier for me to label the physical switch and using appropriately color coded cables. I use VST only for the VM network as needed. For the Admin/SC/Vmotion, EST is best, for iSCSI no VLAN and for VM Network use whichever works best.

Keep an eye on your security zones and segregate appropriately.

Best regards, Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009, DABCC Analyst[/url]
Now Available on Rough-Cuts: 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing ESX and the Virtual Environment'[/url]
Also available 'VMWare ESX Server in the Enterprise'[/url]
[url=http://www.astroarch.com/wiki/index.php/Blog_Roll]SearchVMware Pro[/url]|Blue Gears[/url]|Top Virtualization Security Links[/url]|Virtualization Security Round Table Podcast[/url]

[voodoo77]

Hello,

Based or your scenario, I will remain with 4 unused NICs on each ESX host, and also I'm not able to use NIC teaming (or am I wrong ?).

Feel free to correct me.

NIC scenario

2 pNICs for SC and VMotion

2 pNICs for IP Storage

2 pNICs for VMs

Network setup based on the NIC scenario above will look like this.

Virtual Networks

Usage - Port Group - VLAN ID

VM Network - Production - 0

VMotion - Vmkernel - 0

SC - Service Console - 0

ISCSI - IP Storage - 0

Note : PortGroups are using VLAN ID 0 (zero) or NONE inside vSwitch in EST mode.

Virtual Switch Layout

Virtual Switch - Usage

vSwitch0 - Production

vSwitch1 - SC and VMotion

vSwitch2 - IP Storage

Virtual Switch to Physical NIC to Physical Switch Mapping

pNIC - vSwitch - pSwitch - Type

vmnic0 - vSwitch1 - pSwitch1 - VLAN 20

vmnic1 - vSwitch1 - pSwitch2 - VLAN 20

vmnic2 - vSwitch0 - pSwitch1 - VLAN 10

vmnic3 - vSwitch0 - pSwitch2 - VLAN 10

vmnic4 - vSwitch2 - pSwitch1 - VLAN 30

vmnic5 - vSwitch2 - pSwitch2 - VLAN 30

Commands applied on Cisco IOS to configure switch port for VLAN access

( I need to find how this is accomplished on Dell PowerConnect 6224)

interface GigabitEthernet1/15

switchport (Configures the LAN port for Layer 2 switching)

switchport access vlan vlan_ID (The value can be 1 through 4094, except reserved VLANs)

switchport mode access (Configures the port to be an access port to prevent trunk negotiation delays)

spanning-tree portfast (Configure port-fast for initial STP delay)

IP Storage

The Equallogic PS6000XV SAN, which has 4 NICs, will be configured with MPIO, and it will have 2 NICs connected in each pSwitch on VLAN 30 configured ports, and inside VI will be on vSwitch2.

Jumbo Frames

Will be used inside IP Storage Network, I know this have to be configured all the equipments : SAN, pSwith and vSwitch 2.

1. on the pSwitch (using Cisco commands)

command to be runned in global configuration mode :system mtu jumbo 9000

Note. Switch reboot is required. MTU can be cheched with the following command : show system mtu

2. on the SAN (FilerView)

Go to Network &gt; Manage Interfaces and then clicking the Modify

link for the interface to be changed. Set the “MTU size” to 9000 (from

the default of 1500), click Apply.

You can verify the settings using Network -> Manage Interfaces -> Show All Interface Details, or by using ifconfig -a command from a Data ONTAP command prompt.

3. on the vSwitch (my case)

a. Add a vSwitch :

esxcfg-vswitch -a vSwitch2

b. Set the MTU value for the new vSwitch

esxcfg-vswitch -m 9000 vSwitch2

c. Add PortGroup

esxcfg-vswitch -A “IPStorage” vSwitch2

d. Add VMKernel Int.

esxcfg-vmknic -a -i 172.18.30.1 -n 255.255.255.0 -m 9000 IPStorage

e. Attach vmnics

esxcfg-vswitch -L vmnic4 vSwitch3

esxcfg-vswitch -L vmnic5 vSwitch3

f. Check if all is configured ok

esxcfg-vswitch -l should show the vSwitch’s MTU is now 9000.

esxcfg-nics -l should show the MTU for the NICs linked to that vSwitch are now set to 9000 as well.

g. Ping SAN with a package size of 9000 to see if everything Works

vmkping –s 9000 172.18.30.1

Conclusions

After reading Ken "The Great vSwitch debate" I understood that I need to have separate networks for IPStorage and VMotion, therefore I will proceed accordingly.

Also I will be using EST like you said, sounds better than VST (this was my first option). I just have to be careful to connect the right pNIC into the right pSwitch port.

Questions

Q1 : If I'm using 2 switches for having a redundant network, do I need to have one or two ports direct connected between the pSwitches and have them setup with STP ?

Q2 : Being this case scenario, can I still use NIC teaming and how ?

First thing that came to my mind is to add unused pNICs to the VLANS I want to have them with NIC teaming enabled.

Is there any other way beside using unalocated pNICs ?

Q3. (and probably the major one) Is this design reliable and can be used properly inside a production environment ?

Thanks again a million times.

[AndreTheGiant]

Q1 : If I'm using 2 switches for having a redundant network, do I need to have one or two ports direct connected between the pSwitches and have them setup with STP ?

Yes, if you have more then 1 link use LAG (or similar) in trunk mode for all your VLANs

Q2 : Being this case scenario, can I still use NIC teaming and how ?

With normal ESX teaming.

Is this design reliable and can be used properly inside a production environment ?

With 2 switches yes.

Andre

**if you found this or any other answer useful please consider allocating points for helpful or correct answers

[voodoo77]

bq. Q1 : If I'm using 2 switches for having a redundant network, do I need to have one or two ports direct connected between the pSwitches and have them setup with STP ?

Yes, if you have more then 1 link use LAG (or similar) in trunk mode for all your VLANs

This chapter is clear now : 2 ports on each pSwitch, directly connected, setup with STP, Trunk and LAG enabled.

bq. Q2 : Being this case scenario, can I still use NIC teaming and how ?

With normal ESX teaming.

Can I team the NICs in the same VLAN, even if they are physically connected into different pSwitches ? This must be it then.

bq. Is this design reliable and can be used properly inside a production environment ?

With 2 switches yes.

>

[Texiwill]

Hello,

If you have the spare pNIC put 2 for SC and 2 for VMotion. Put IP Storage on its OWN pSwitches, SC/VMotion on its own pSwitches and VM traffic on a third set of pSwitches. That is if you can. It is much more secure this way. They should have private and segregated networks.

Best regards, Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009, DABCC Analyst[/url]
Now Available on Rough-Cuts: 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing ESX and the Virtual Environment'[/url]
Also available 'VMWare ESX Server in the Enterprise'[/url]
[url=http://www.astroarch.com/wiki/index.php/Blog_Roll]SearchVMware Pro[/url]|Blue Gears[/url]|Top Virtualization Security Links[/url]|Virtualization Security Round Table Podcast[/url]

[voodoo77]

Hello,

If you have the spare pNIC put 2 for SC and 2 for VMotion. Put IP Storage on its OWN pSwitches, SC/VMotion on its own pSwitches and VM traffic on a third set of pSwitches. That is if you can. It is much more secure this way. They should have private and segregated networks.

Unfortunately, I only have 2 pSwitches I can rely on.

[AndreTheGiant]

&gt;Can I team the NICs in the same VLAN, even if they are physically connected into different pSwitches ? This must be it then.

Yes, 1 vSwith, 2 pNICs to 2 switches connected with LAG (and trunking into LAG for multiple VLANs)

Andre

**if you found this or any other answer useful please consider allocating points for helpful or correct answers

[voodoo77]

bq. &gt;Can I team the NICs in the same VLAN, even if they are physically connected into different pSwitches ? This must be it then.

Yes, 1 vSwith, 2 pNICs to 2 switches connected with LAG (and trunking into LAG for multiple VLANs)

Hello Andre,

I got the whole picture now, in my case there it will be only LAG (not going to mix the VLANs there).

Thank you for your kind explanations.

[Ken_Cline]

You've gotten lots of good advice. I'd like to point you to my series of blog articles on virtual networking. You can find it here: http://kensvirtualreality.wordpress.com/tag/vswitch/ - there are eight parts to the series and it addresses most of the questions you've raised here.

Ken Cline

VMware vExpert 2009

VMware Communities User Moderator

Blogging at: http://KensVirtualReality.wordpress.com/

[voodoo77]

You've gotten lots of good advice. I'd like to point you to my series of blog articles on virtual networking. You can find it here: http://kensvirtualreality.wordpress.com/tag/vswitch/ - there are eight parts to the series and it addresses most of the questions you've raised here.

Hi Ken,

This 8th part is fresh, I read all the 7 parts of the vSwitch debate yesterday, and indeed, your posts guided me achieving the setup plan (among lots others). You are a true gold mine to all this community.

Thank you.