Network Architecture for VMware

[michaeltz]

Hi,

We have brought in some network consultants to set up our network and we've provided them with a design but they can't execute it and they want to redesign it from scratch.  I would like to share our design and get some thoughts from the experts.

My questions are:

Is there a glaring flaw in our design?

Are there things we didn't take into account such as what Class network to use?

Is this something that should be easy to configure for a company that does networking?

We have identified 3 VLANs:

10 - vSphere Mgmt traffic and ILO

20 - vm traffic (web, sql server, etc)

30 - storage traffic to NFS

We have 2 ESXi hosts with 7 NICs configured as follows:

NIC 1 - ILO

NIC 2&3 - Teamed for VLAN 10

NIC 4&5 - Teamed for VLAN 20

NIC 6&7 - Teamed for VLAN 30

We have a server running vCenter Essentials with 3 NICs (no teaming unfortunately);

NIC 1 - VLAN 10

NIC 2 - VLAN 20

NIC 3 - VLAN 30

We want the NICs on the server running vCenter to communicate all 3 VLANs so they can see all the NICs on the ESXi hosts or the NFS storage.

We have 2 Cisco SG300-28 managed switches.

Thank you,

Mike

[arturka]

Hi

How many pNIC's you have in ESX(i) host ?  If you have 6 are there 2 onboard and 1 quad port or different configuration ?

We have identified 3 VLANs:

10 - vSphere Mgmt traffic and ILO

20 - vm traffic (web, sql server, etc)

30 - storage traffic to NFS

you should add one more vLAN for vMotion (should be on separate vLAN and HW in non routable network - security)

Cheers
Artur

Please, don't forget the awarding points for "helpful" and/or "correct" answers.

[JohnADCO]

Are you attempting to subnet all this out from like one 0/24 network? (255 IP Address block)    Classless routing is a pita,  I could see that being an issue.

[michaeltz]

Artur,

Thank you very much for your reply.  I don't believe our Essentials license supports vMotion (http://www.vmware.com/files/pdf/vsphere_pricing.pdf -page 😎 but you're absolutely correct that if we used vMotion, we should created a dedicated VLAN.

Our ESXi hosts came with 4 NICs on board (ILO on a separate port) and we added a dual port NIC.  We're cross-teaming the dual port to get true redundancy in case the dual port fails.

We do have a resource who can configure the vSwitches in vSphere and team them up after the network has been set up.

Our network consultants are supposed to configure the switches, create the VLANs, plug the cables in - we've given them a map of which port the nics are supposed to go into, create VLANs and test traffic from the NICs on the server running vCenter to make sure that the 3 NICs on that server can communicate with all VLANs.

Is this a simple setup for a network expert?  Do most people run into issues with VLANs or the servers running vCenter which have multiple NICs?

Thank you,

Mike

[JohnADCO]

Vmotion was included for vSphere 4.1 and above.         You want it trust me.

[JohnADCO]

At least Vmotion traffic can be a separate 0/24 network because you don't need to see it.

On the other stuff?    You have identified what needs to be done.   I'd let them archetect it how ever they want I suppose, but if they go with multiple subnets not in the same 0/24?  I think they will need to add an actual router to allow the different subnets to be able to traverse each other.

[michaeltz]

John,

We don't have many devices so our system should be fine with a Class C network 0/24.

Our first attempt was to set each VLAN on their own Class C network.

The IP addresses would be in these ranges:

192.168.10.x  S/M 255.255.255.0

192.168.20.x  S/M 255.255.255.0

192.168.30.x  S/M 255.255.255.0

They could not get cross VLAN communication on the server running vCenter with the 3 NICs.

Mike

[arturka]

Hi

Thank you very much for your reply.  I don't believe our Essentials license supports vMotion (http://www.vmware.com/files/pdf/vsphere_pricing.pdf -page 😎 but you're absolutely correct that if we used vMotion, we should created a dedicated VLAN.

Cause you have posted in ESXi4 community I thought that you have vSphere 4.1 package not vSphere 5.

Our ESXi hosts came with 4 NICs on board (ILO on a separate port) and we added a dual port NIC.  We're cross-teaming the dual port to get true redundancy in case the dual port fails.

Our ESXi hosts came with 4 NICs on board (ILO on a separate port) and we added a dual port NIC.  We're cross-teaming the dual port to get true redundancy in case the dual port fails.

good and I assume that each NIC from vSwitch is connected to different pSwitch that have it redundancy on pSwitch as well

Is this a simple setup for a network expert?  Do most people run into issues with VLANs or the servers running vCenter which have multiple NICs?

It's quite easy set up, without trunking, should goes straight forward. In my vCetner i have 3 vNIC's and all goes fine, without problems. You should check it if your pSwitch and NAS supports JumboFrames, if it does you should give a try and enable JumboFrames on NFS network.

Cheers
Artur

Please, don't forget the awarding points for "helpful" and/or "correct" answers.

[michaeltz]

Thanks Artur,

We have vSphere 4.0 Update 1 installed.  The link I pulled up was for 5.0 - my apologies.

I get the sense that the overall design seems sound and the network engineers should able to set this up especially when they're not configuring the switches in VMware.

Mike

[arturka]

Hi Mike,

If you have 4.0 installed you should prepare networking for upgrade to 4.1 to have vMotion (( why not ? 🙂 )

I see you network config like table below:

nic   name	vSwitch	portgroup	mode	vLAN_ID	TRUNK	pSwitch
NIC1	vSwitch0	mgmt/vMotion	active in   mgmt/passive in vMotion	10 and 40	yes	pSwitch1
NIC2	vSwitch0	mgmt/vMotion	active in vMotion/passive in mgmt	10 and 40	yes	pSwitch2
NIC3	vSwitch1	VM_LAN	active	20	no	pSwitch1
NIC4	vSwitch1	VM_LAN	active	20	no	pSwitch2
NIC5	vSwitch2	NFS	active	30	no	pSwitch1
NIC6	vSwitch2	NFS	active	30	no	pSwitch2

I've added vLAN40 for vMotion

you have full redundancy on each level, separate ESX mgmt traffic from any other traffic, separate vMotion on HW level, in general network config is resilient, no SPOF's (fully redundant) follow design and security best practices

Cheers
Artur

Please, don't forget the awarding points for "helpful" and/or "correct" answers.

[logiboy123]

I have created some network diagrams for vSphere 5 environments. Feel free to check them out as I think they will help you visualize what you are trying to achieve:

http://vrif.blogspot.com/2011/10/vmware-vsphere-5-host-network-designs.html

Some points I would make right now are

1) Plan VLAN's to accommodate all functionality now. So reserve a VLAN for vMotion, FT, Cluster Heartbeats and anything else you may implement in the future. This will save you time and effort later on and will allow seamless integration of future requirements or features.

2) Don't put iLO on the same VLAN as ESX management. Both of these should use different VLAN's and it is always nice when there is an OOB managment network isolated from the rest of the environment. Sometimes this is a hard sell though.

3) I don't understand why your vCenter server needs to have access directly to the NFS network unless you are using some sort of vCenter managed backup system that has vCenter as a broker between the datastores/esx and the backup server.

Further I would consider not having you vCenter with access to the same VLAN as both your VM Networking and Management, doing this creates a point of attack that you really don't need. Instead please consider putting your vCenter server in VLAN 10 for easier communication with your hosts with firewall rules to talk to other server, or VLAN 20 for better communication with the other servers and create firewall rules to allow communications to your hosts. Bridging networks is not usually a good way of building for security.

If you are interested we can take this conversation offline and I can help you build a design that meets your requirements, if you don't mind me putting up a version of it on my blog.

Regards,

Paul