Applying PNIC strategies in ESX to one's physical switchgear.

[sketchy00]

Ken Cline and Ed Haletky (among others) have put out some great information on standard practices for PNICs and vswitches in an ESX environment. I've read and re-read them in an attempt to build up my new ESX environment, and I still seem to be missing something. Especially when it comes to applying the physical NICs to one's switchgear, and if they should be on separate VLAN, separate switches all together, and how. Here's the setup of what I have.

* Qty 3, stacked GigE switches dedicated for my LAN network. (192.168.0.0 /24)

• Qty 2, trunked GigE switches dedicated for my SAN (10.10.0.0 /24)

• Qty 1, Dell/Equallogic PS5500 ISCSI SAN unit, connected up to the dedicated SAN switches.

• Qty 1, Dell M1000e blade enclosure, with Ethernet pass thoughts on all 3 fabrics; A, B, and C.

• Qty 4, Dell M600 blades in enclosure with 6 NIC's per blade

• Each blade has 6 NICs. According to Ed's article, they should consist of the following:

• 1 NIC per blade for the service console (vswitch0)

• 1 NIC per blade for vmotion (vswitch0)

• 2 NICs per blade dedicated for my SAN. (vswitch1) connected to my SAN (fabric A)

• 2 NICs per blade dedicated to my VM Network (vswitch2) connected to my LAN

• Qty 1, ESXi box that will run a VM stored on the SAN to run Vcenter. Virtualized, but not part of my blade cluster.

My uncertainty comes in when figuring out how the isolated networks will work.

1. If I have my LAN on a 192.168.0.0 /24 network, should I isolate the ports that the service console and vmotion into a VLAN on my LAN switchgear, and assign it a netblock of say, 172.16.32.0 /24, and make this routable to the primary LAN? (e.g. 192.168.0.0 /24)

2. Is the NIC dedicated for vmotion (but on vswitch0, the same as the service console) going to be on a separate VLAN itself, or will it live in that same theoretical netblock of say, 172.16.32.0 /24?

3. If one is having difficulty establishing a vlan for these mgmt purposes, can one have thise PNICs go to a separate switch that sits on a different leg (extremely well protected) of one's firewall?

I know once I put the missing pieces of the puzzle together, I will truly get it. Right now though, I'm not quite there yet. Any info would be greatly appreciated.

[Texiwill]

Hello,

• Each blade has 6 NICs. According to Ed's article, they should consist of the following:

• 1 NIC per blade for the service console (vswitch0)

• 1 NIC per blade for vmotion (vswitch0)

• 2 NICs per blade dedicated for my SAN. (vswitch1) connected to my SAN (fabric A)

• 2 NICs per blade dedicated to my VM Network (vswitch2) connected to my LAN

• Qty 1, ESXi box that will run a VM stored on the SAN to run Vcenter. Virtualized, but not part of my blade cluster.

1. If I have my LAN on a 192.168.0.0 /24 network, should I isolate the ports that the service console and vmotion into a VLAN on my LAN switchgear, and assign it a netblock of say, 172.16.32.0 /24, and make this routable to the primary LAN? (e.g. 192.168.0.0 /24)

Your VMotion and Service console networks must be on the wires using different VLANs. Your SC should have a separate subnet than your VMotion network.

2. Is the NIC dedicated for vmotion (but on vswitch0, the same as the service console) going to be on a separate VLAN itself, or will it live in that same theoretical netblock of say, 172.16.32.0 /24?

Actually, now, when you combine pNICs for redundancy you are using 1 pNIC for SC and 1 pNIC for VMotion but they are on different VLANs and different subnets. THe reason for this is if you have a failed pNIC you then have redundancy and both VLANs can run through 1 pNIC. So both pNICs involved need to have each VLAN trunked to them.

3. If one is having difficulty establishing a vlan for these mgmt purposes, can one have thise PNICs go to a separate switch that sits on a different leg (extremely well protected) of one's firewall?

Yes.

I know once I put the missing pieces of the puzzle together, I will truly get it. Right now though, I'm not quite there yet. Any info would be greatly appreciated.

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009, DABCC Analyst
====
Author of the books 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing ESX and the Virtual Environment' available for pre-order now
'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
SearchVMware Pro|Blue Gears|Top Virtualization Security Links|Virtualization Security Round Table Podcast

[sketchy00]

Thanks for the great feedback. So, for a point of clarity, let me attempt to summarize what would occur, and you can tell me if I'm interpreting things correctly or not.

If my primary LAN is on 192.168.0.0 /24, and I create a vlan for the service console of say, 172.16.32.0 /24, could my vmotion network be perhaps on another vlan under the 172.16.33.0 /24 network? Then both of these VLANs be routable to the LAN (192.168.0.0 /24) so they can use DNS, NTP, etc. These VLANs would consume ports on the same stack of switches that the primary LAN resides on.

Does that sound right?

If that is basically correct, then that is good to hear. I'm having a bugger of a time getting a static route of these vlans up to my primary lan, but that is outside of the scope of the forum, so that is why I asked if the sc and vmotion vlans could be put on switches that come into a different leg on my firewall; where it could contact DNS and NTP, etc. in that fasion.

- Pete

[habibalby]

Hello,

As long as the vLANs are Trunked with eachother and all the VLAN ID's set properly, then in case of failure, still you can access the S.C through 172.16.32.0 /24 and still you will be vMotioning via 172.16.33.0 /24. But make sure the trunking part and you will be done. I'm doing the same with 6 pNIC's and everything works perfectly. Beside the Default Gateways of each vLAN. In my case, I have two pSwitches cascaded as one logical unit. Each vLAN in both pSwitches i:e "vmnic0 goes to -> pSwitch0 port 8" && "vmnic1 goes to -> pSwitch1 port8". In pSwitch0 port 8 is vLAN 2 and in pSwitch1 port 8 is vLAN 3. vLAN 2 is 172.16. IP Schema and vLAN 3 is 10.1. IP Schema. For my S.C Default Gateway, I'm using the vFirewall internal network IP Address and for my vMotion I'm using the vLAN 3 gateway, so, my entire ESX Hosts are behind a Firewall as well as the vMotion network.

In additional to the DNS and NTP Service. For best securityt and better managability, it will be better to keep the vCenter beside the ESX "Service Console" Network, and configure the vCenter to act as a DNS Server for your ESX Farm.

Configure a virtual firewall, to jumb for your Internal network over your ESX Network for your administration. Then, in your Internal Network, configure an NTP Server to pull the correct time from the internet, and configure the ESX Server to pull the time from the physical host setting in the Internal Network. You will require only port TCP 123 to be opened between your Internal Network and the ESX Network.

Best Regards,

Hussain Al Sayed

If you find this information useful, please award points for "correct" or "helpful".

[Texiwill]

Hello,

If my primary LAN is on 192.168.0.0 /24, and I create a vlan for the service console of say, 172.16.32.0 /24, could my vmotion network be perhaps on another vlan under the 172.16.33.0 /24 network? Then both of these VLANs be routable to the LAN (192.168.0.0 /24) so they can use DNS, NTP, etc. These VLANs would consume ports on the same stack of switches that the primary LAN resides on.

Not really. Only 172.16.32.0/24 will be routable to 192.168.0.0/24. VMotion would never be routable to ANY other network. You just need to make sure that you trunk both VLANs through the appropriate switch ports.

As a matter of fact, for better security you would place a firewall between 172.16.32.0/24 and 192.169.0.0/24 so that your routing is through a firewall and not direct attached between the networks. This will protect your service consoles and vCenter which should also be on the 172.16.32.0 side of the firewall.

172.16.33.0/24 should not be routed to anywhere and should remain private to only ESX hosts.

Does that sound right?

If that is basically correct, then that is good to hear. I'm having a bugger of a time getting a static route of these vlans up to my primary lan, but that is outside of the scope of the forum, so that is why I asked if the sc and vmotion vlans could be put on switches that come into a different leg on my firewall; where it could contact DNS and NTP, etc. in that fasion.

Always use a firewall.

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009, DABCC Analyst
====
Author of the books 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing ESX and the Virtual Environment' available for pre-order now
'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
SearchVMware Pro|Blue Gears|Top Virtualization Security Links|Virtualization Security Round Table Podcast

[sketchy00]

Thanks fellas for the great feedback. I think that cleared up some of the confusion I had on the matter. Nice to know I don't have to working about routing on the vmotion network.

After hearing your feedback, and thinking aloud, it sounds as if I had these two networks (the service console and vmotion) where their wires were feeding into their own dedicated redundant switches, then have these switches sitting on their own dedicated inferface off of my ISA firewall, then I can make sure only my service console traffic (172.16.32.0 /24) traffic is routable. I would free up those ports on my LAN switches, make the configuration a bit cleaner, tighten up the security, and avoid some of the problems I've had with vlanning on my internal LAN switches. Hmm.... Interesting.

Thanks,

Pete

[Texiwill · Accepted solution]

Hello,

Definitely the way I would do it. GLad we could help. Please remember to add points by marking answers as helpful or correct.

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009, DABCC Analyst
====
Author of the books 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing ESX and the Virtual Environment' available for pre-order now
'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
SearchVMware Pro|Blue Gears|Top Virtualization Security Links|Virtualization Security Round Table Podcast