VI3 network configuration (HP DL380G5, EqualLogic, Cisco)

[cmsJustin]

Hey everyone. We purchased our first ESX box 2 months ago and have been running about 15 VMs on it for a while now, however it's not configured the right way yet, so...

Our existing HP DL380 G5 has 10 NICs (2 onboard, 4-port PCI-Express, 4-port PCI-X). We just tried to buy the same exact thing, but the PCI-X card has been discontinued. So we either decide to get 10 ports again (2 x 4-port PCI-Express) and just make sure the vmnic's all match up, or maybe try to get by with 6 pNICs. Or maybe buy another PCI-Express card for the first server and rip the PCI-X card out (only ~$550 exra, my favorite option). How important is it for these to match?

We just purchased an EqualLogic PS3700X, along with a Cisco 3560G-24TS for iSCSI traffic, and a Cisco 4510R with some gigabit line cards for all the non-SAN/VMotion traffic.

My VLANs look like this (not in production yet, can be changed):

1 default (LAN)

2 voip

3 san

4 vmware (vmotion)

5 public (dmz, wireless guest access)

Here is how I plan to cable/configure the network (along with nifty pictures):

ESX Network diagram for 10 pNICs on Flickr: http://www.flickr.com/photos/cmsjustin/2103385029/

6 pNICs?: http://www.flickr.com/photos/cmsjustin/2104164196/

Red is iSCSI, Green is VMotion, and the Black ones are 802.1q trunks carrying traffic for VLAN 1, 2, and 5

The 4510 and 3560 have a VTP Etherchannel trunk between them (2 or 4 gig, haven't decided).

All VMs will use the MS iSCSI initiator if they need LUN access.

I was planning to set these across all 3 NIC cards for redundancy:

vmnic0: 802.1q trunk (default, voip, public)

vmnic1: 802.1q trunk (default, voip, public)

vmnic2: 802.1q trunk (default, voip, public)

vmnic3: 802.1q trunk (default, voip, public) (backup path on the 3560)

vmnic4: iSCSI

vmnic5: iSCSI

vmnic6: iSCSI

vmnic7: iSCSI (backup path on the 4510)

vmnic8: VMotion

vmnic9: VMotion (backup path on the 4510)

So here are my questions:

Does the service console belong on that main vSwitch for VM traffic?

Should I have VMotion redundant, or would it not matter? (STP or some other convergence time will kill the migration anyway I imagine...)

Should I rip out the PCI-X card and get the PCI-Express, or live with it, or just use 6 pNICs?

iSCSI multipathing (vmnic7)?

I also read somewhere that you only need 2 VLANs for ESX, something about VMotion should sit with iSCSI, or was it with VM traffic. Thoughts?

Does the 10-NIC diagram look ok? Any corrections/suggestions?

I tried to ask VMware for a suggested config, and they told me to call their PSO consulting services...

Hopefully I didn't forget anything, I was multitasking while writing this.

cmsJustin.blogspot.com

Twitter.com/cmsJustin

[cmsJustin]

bump

Anyone?

cmsJustin.blogspot.com

Twitter.com/cmsJustin

[Texiwill]

Hello,

Given your requirements you should at minimally do the following:

2 pNICs for SC (Administrative Network) vSwitch0 participates in storage network.

2 pNICs for vMotion (vMotion Network) vSwitch1

2 pNICs for iSCSI (Storage Network) vSwitch2

2 pNICs for VOIP vSwitch (voip network) vSwitch3

2 pNICs for VM Network vSwitch4

That would use all 10 of your pNICs.... Having more than 2 pNICs per vSwitch can cause all sorts of problems if you are using load balancing with up to 10 minutes delays if pSwitches go away. If you have more than 2 pNICs per vSwitch and you are not using load balancing but failover the extra assigned pNIC is really a waste of pNIC.

This would give the most redundancy, security, and performance.

Best regards,

Edward L. Haletky, author of the forthcoming 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', publishing January 2008, (c) 2008 Pearson Education. Available on Rough Cuts at http://safari.informit.com/9780132302074

[cmsJustin]

We currently don't have an administrative network. We only have 100 users and a few switches, and I'm not sure if the hassle of a separate network for switch management and the service console in ESX would be worth it...

Can't I just trunk all of the VoIP, VM (LAN), and DMZ traffic together?

Where should the service console go if not on it's own pNICs?

cmsJustin.blogspot.com

Twitter.com/cmsJustin

[Texiwill]

Hello,

From a security perspective, I find the hassle very much worth it.... But you have to choose between performance, security, and usability. I would never advocate having the ESX Service Console directly accessible by every user on your network. At the very least implement the pam_access modules to disallow certain hosts/users from accessing the system.

I would not trunk VoIP with anything as it is a heavy network hitter.

I would not trunk DMZ and anything together as it is the DMZ and should have physical separation for security reasons.

You want to consider security (physical separation of networks) vs redundancy (multiple pNICs) vs Performance or each network. Your DMZ must remain secure, else why have one. VoIP is network intensive. vMotion should absolutely remain separate for performance and security reasons. Since you have no admin network you could combine SC with the regular VM Network.... But that could also lead to performance issues.

So:

2 pNIC for VoIP

2 pNIC for DMZ (combining this with anything could have serious security concerns)

2 pNIC for vMotion (THis is the highest risk network)

2 pNIC for SC/Internal VM Network (this could have security concerns)

Remember 70% of all security breaches happen from inside the company not outside.

Best regards,

Edward L. Haletky, author of the forthcoming 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', publishing January 2008, (c) 2008 Pearson Education. Available on Rough Cuts at http://safari.informit.com/9780132302074

[cmsJustin]

But this still limits the number of VLANS I can have on the hosts. I was hoping to have separate SAN and VMotion networks, but have some sort of VLAN trunk interface to the Cisco switches.

I read these:

http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1280449,00.html

http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1283036,00.html

I've also reconsidered our VLAN plans with regards to the management VLAN you suggested, and we may go with that. We aren't very security conscious (small company), but really need to be.

cmsJustin.blogspot.com

Twitter.com/cmsJustin

[cmsJustin]

Also, I should point out that performance and uptime are our top priorities, but with reasonable security.

cmsJustin.blogspot.com

Twitter.com/cmsJustin

[jhanekom]

Texiwill: I'm curious about your claims of outages in excess of ten minutes if the following conditions are true (at least, this is what you seem to be saying):

- more than two uplinks

- a physical switch goes belly-up

Are you talking about the case where the switch will come up and link status will be restored, but the switch will be in blocking mode while it finishes booting up?

To the best of my knowledge, if you configure the rolling failover and beacon probing settings correctly (or use FEC/static 802.3ad with a multi-switch stack that supports channels accross pSwitches), this will not be an issue at all. Beacon probing was broken, but it was fixed in 3.0.2.

Justin: since you're using iSCSI, want to purchase additional NICs and are concerned about performance: consider investing in proper iSCSI HBA's. The software initiator is not able to load balance (hence using more than one NIC only gained you redundancy) and isn't the fastest way of shunting storage data around.

A discussion on the Software vs. Hardware initiators: http://communities.vmware.com/thread/61061

List of compatible adapters: http://www.vmware.com/pdf/vi3_io_guide.pdf

[Texiwill]

Hello,

With uptime and performance being your goals, you definitely want to have the VoIP network on their own set of pNICs, this will give that the performance necessary as well as the security. You really do not want people listening on other peoples conversations? As for a SAN and vMotion, SAN is over its own adapters, unless you are talking about iSCSI, if you are talking about iSCSI you need more pNIC to handle the iSCSI throughput.

The only real difference you are making is spliting out the DMZ into physical NICs and VoIP the same, they can use VLANs in the switch, this would considered EST (External Switch Tagging) in ESX supported VLAN parlance. You gain performance and better security in this case. The DMZ split is for security and performance, while the VoIP change is purely for performance with minor security considerations. YOu can still use VST (Virtual Switch Tagging) VLANs on the normal VM Network, and within the DMZ if you have multiple VLANs in the DMZ.

Best regards,

Edward L. Haletky, author of the forthcoming 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', publishing January 2008, Copyright 2008 Pearson Education. Available on Rough Cuts at http://safari.informit.com/9780132302074

[Texiwill]

Hello,

Texiwill: I'm curious about your claims of outages in excess of ten minutes if the following conditions are true (at least, this is what you seem to be saying):

- more than two uplinks

- a physical switch goes belly-up

Are you talking about the case where the switch will come up and link status will be restored, but the switch will be in blocking mode while it finishes booting up?

No, the physical switch goes belly up or the cables are pulled and the network was literally down for about 10 minutes as the switching network tried to determine where to settle everything. This was with or without beacon monitoring and Load Balancing.... I have not run the test on the latest patched version 3.0 or 3.5 however. It is on my list to do however. PM me if you want the details and are interested in running the test. Unfortunately, I do not have access to enough hardware at the moment to duplicate the environment. Working on it however.

Best regards,

Edward L. Haletky, author of the forthcoming 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', publishing January 2008, Copyright 2008 Pearson Education. Available on Rough Cuts at http://safari.informit.com/9780132302074

[cmsJustin]

jhanekom, I know that an iSCSI HBA will improve VMFS performance and redundancy, but our Windows VMs will be using the MS iSCSI initiator for data drives instead of setting up drives in VirtualCenter. I only plan on having 3 VMFS volumes at this time: 1 for all of our server VMs (512gb), 1 for desktops (512gb), and 1 for our domain controllers (64gb) so I can snapshot the whole volume and rollback our whole domain instead of one server if needed.

Texiwill, Our VoIP servers are not virtualized, they just might need occasional access to a server in VMware, like a tftp server or mail server for presence management. I thought the whole point of VLANs were to logically separate traffic without having separate switches and links. I doubt we will ever hit a bottleneck on the backplane of a 4510 with 10GE supervisors with 100-300 users. But I do believe that the SAN should have it's own physical switch...

I guess the most important thing I want to answer is will the 10 pNIC diagram above work, and if not, what should I do differently. At the time it seemed very robust, scalable, and redundant...The only thing I wish I did differently was to stack 3750's instead of having a single 3560.

cmsJustin.blogspot.com

Twitter.com/cmsJustin

[jhanekom]

Ah, OK. The MS initiator will give you better load balancing. But even if you're not using VMFS you'd probably still be better off using the hardware initiator (with RDMs.)

(As an aside, be careful about snapshotting a domain controller(s) to roll back things... you'd do well to follow the same recovery steps as you would in a physical environment.)

[cmsJustin]

If it came to that, I would turn both off, rollback the snapshot, and then power them up one at a time.

I would only do this if AD got corrupted somehow across both DCs, and this hasn't happened yet. I just wanted a separate snapshot of them on EqualLogic...

cmsJustin.blogspot.com

Twitter.com/cmsJustin

[Texiwill]

Hello,

vmnic0: 802.1q trunk (default, voip, public)

vmnic1: 802.1q trunk (default, voip, public)

vmnic2: 802.1q trunk (default, voip, public)

vmnic3: 802.1q trunk (default, voip, public) (backup path on the 3560)

vmnic4: iSCSI

vmnic5: iSCSI

vmnic6: iSCSI

vmnic7: iSCSI (backup path on the 4510)

vmnic8: VMotion

vmnic9: VMotion (backup path on the 4510)

I would do the following:

vmnic0, vmnic1 -> SC participating in iSCSI subnet for iSCSI auth only (limited use)

vmnic2, vmnic3 -> voip, public

vmnic4, vmnic5 -> iSCSI

vmnic6, vmnic7 -> vMotion

vmnic8, vmnic9 .... unused or assigned to a different path to the iSCSI server....

Having more than 2 pNICs per network is not really a savings, there is no load balancing for iSCSI unless you set it up from the iSCSI server yourself. There is no aggregation so you in essence have 1 live link and up to 3 failover links. Unless you present iSCSI luns to different paths.... So at most you get failover.

I would split up the SC from the voip/public as the SC is where most backups/VIC/VC/Admin communications happen and this is just a safety net.

You can do what you want but I am not sure the 4 pNIC per vSwitch will really buy you much if there is a failure, as the SC, and iSCSI does not participate in load balancing.

I would use vmnic8, vmnic9 as a way to do by hand load balancing of the iSCSI LUNs. you could present a set of LUNs through vmnic4,5 and another through 8,9... But this is something the Equilogic must support it is not something ESX can do.

Best regards,

Edward L. Haletky, author of the forthcoming 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', publishing January 2008, Copyright 2008 Pearson Education. Available on Rough Cuts at http://safari.informit.com/9780132302074