NIC Best Practice: ESX 3.5U3, 5 NICs, iSCSI

[Powl]

Hello.

I am bringing up a new VMware environment (DRS, HA, vMotion, two ESX hosts, iSCSI storage) and I have a couple of questions regarding best practices for physical NIC configuration, specifically regarding Service Console and vMotion teaming/redundancy. I've read many threads as well as referenced the VI3 ATDG/AOG regarding this same question, but am still a little confused.

Physical NIC Configuration:

NIC 1 - Service Console

NIC 2 - vMotion

NIC 3 - iSCSI

NIC 4 and 5 - Teamed and trunked for Virtual Machines

From what I have been reading, it looks like the Service Console and vMotion should both be redundant. It would appear that I could team NIC 1 and 2 for this purpose, setting NIC 1 as the primary for the Service Console with NIC 2 as the failover and NIC 2 as primary for vMotion with NIC 1 as the failover. However, I also seem to remember reading that vMotion should be on a seperate, unrouted network segment. So, say my Service Console has an IP address of 10.21.100.100 (VLAN 100) and vMotion has an IP of 192.168.200.100 (VLAN 200), how can I team two physical NICs on seperate VLANs? And, if I try to trunk each of the two physical NICs on the switch, I lose connectivity with the Service Console.

Is it not that important for vMotion to be on a seperate, unrouted network? If that is the case, I can just put both NICs on the same VLAN and be done with it. Am I missing something else? Any help is greatly appreciated. Just trying to configure the infrastructure with best practices in mind. Thanks.

Best Regards,

Justin

[Texiwill]

Hello,

Check out http://www.networkworld.com/community/node/36174 for a blog on using 5 pNICS with VMware ESX.

how can I team two physical NICs on seperate VLANs?

You use multiple portgroups on the same vSwitch. The trunking I leave to the pSwitch specialists.

Is it not that important for vMotion to be on a seperate, unrouted network? If that is the case, I can just put both NICs on the same VLAN and be done with it. Am I missing something else? Any help is greatly appreciated. Just trying to configure the infrastructure with best practices in mind. Thanks.

THey should be separate and can not be on the same network else vMOtion will not work. THey at least need to be parts of different subnets.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

Blue Gears and SearchVMware Pro Blogs: http://www.astroarch.com/wiki/index.php/Blog_Roll

Top Virtualization Security Links: http://www.astroarch.com/wiki/index.php/Top_Virtualization_Security_Links

[Powl]

Texiwill - Thank you for your prompt reply. I have read the article you mentioned and that makes sense to me and is in line with what I am trying to do. The only question is, if we use VLANs, how can this work without all NICs being trunked on the switch?

For instance:

pNIC0 -> vSwitch0 -> Portgroup0 (service console) (vlan 100)

pNIC1 -> vSwitch0 -> Portgroup1 (VMotion) (vlan 240)

pNIC2 -> vSwitch0 -> Portgroup2 (Storage Network) (vlan 200)

pNIC3 -> vSwitch1 -> Portgroup3 (VM Network) (trunked)

pNIC4 -> vSwitch1 -> Portgroup4 (VM Network) (trunked)

If pNIC0 fails over to pNIC1 or pNIC2 and those NICs are all configured for a specific VLAN on the switch, I don't see how it would work. It would seem to me that they would all need to be trunked to handle the different VLANs. The second I trunk the NIC that the Service Console is on, I lose all connectivity from VirtualCenter, can't ping, etc. (We're using a Cisco Catalyst 6506 running CatOS, if that helps in any way.)

On an aside,

THey should be separate and can not be on the same network else vMOtion will not work. THey at least need to be parts of different subnets.

Does that mean vMotion will not work on a flat network?

[Texiwill]

Hello,

Texiwill - Thank you for your prompt reply. I have read the article you mentioned and that makes sense to me and is in line with what I am trying to do. The only question is, if we use VLANs, how can this work without all NICs being trunked on the switch?

pNIC0 -> vSwitch0 -> Portgroup0 (service console) (vlan 100)

pNIC1 -> vSwitch0 -> Portgroup1 (VMotion) (vlan 240)

pNIC2 -> vSwitch0 -> Portgroup2 (Storage Network) (vlan 200)

pNIC3 -> vSwitch1 -> Portgroup3 (VM Network) (trunked)

pNIC4 -> vSwitch1 -> Portgroup4 (VM Network) (trunked)

If pNIC0 fails over to pNIC1 or pNIC2 and those NICs are all configured for a specific VLAN on the switch, I don't see how it would work. It would seem to me that they would all need to be trunked to handle the different VLANs. The second I trunk the NIC that the Service Console is on, I lose all connectivity from VirtualCenter, can't ping, etc. (We're using a Cisco Catalyst 6506 running CatOS, if that helps in any way.)

You must trunk vlan 100, 240, and 200 through all three pNICs (pNIC0, pNIC1, and pNIC2). You can not just trunk to them, but all three VLANs need to be accessible through each of these ports on the switch to whcih pNIC0, pNIC1, and pNIC2 are connected.

Does that mean vMotion will not work on a flat network?

It can work without VLANs, you just need VMotion to use a separate IP range.... I.e. your flat network could be 192.168.0.x and your VMotion could be 10.168.0.x where your flat network is really running two sets of IP addresses. It does work, but if just reuse of subnets, etc. Not sure if it would be flat anymore however.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

Blue Gears and SearchVMware Pro Blogs: http://www.astroarch.com/wiki/index.php/Blog_Roll

Top Virtualization Security Links: http://www.astroarch.com/wiki/index.php/Top_Virtualization_Security_Links

[mcowger]

It will work just fine on a flat network - its not a best security or performance practice.

--Matt

[Powl]

Figured it out. I didn't have a VLAN ID set for the Service Console, so when I trunked the port on the switch, I lost connection. Once I set a VLAN ID, I could trunk the port successfully. Thanks for your help.