VMware Communities: Message List - Where do I put my networks?

Re: Where do I put my networks?

mike.laspina — Mon, 21 Apr 2008 13:04:28 GMT

2 ESX servers is definitely in the small range I was thinking an upper range of 4-6 ESX server load and you will not have any issues. (AT 6 if most of the VM's are heavy hitters then all bets are off.)

HA on only two servers .... make sure you test that you can successfully load all of your VM's on one server.

You will need to do this when it comes time to patch the hosts and other unforeseen events.

Memory becomes the most important resource, if your VM's are memory starved they will swap active memory to disk and this is not optimum and only testing can tell the real story.

With two physical switches you have the required physical redundancy. In addition to that HA will want to see two IP paths across it, while not critical to HA running you will get warnings. The idea is that if an IP binding is dropped HA can still see hosts on a second logical path. You would need to define the SC logical paths across two vSwitches and have those vSwitches across the respective physical paths.

Sofware iSCSI is OK on teamed NICs, you should try keep it simple. It can be nasty to figure out if something not working intermitantly. Later as you fully understand it then maybe go more complex. You will have your hands full with what you have now I'm sure.

Regards
Mike

Message was edited by: mike.laspina - added iSCSI #2

Re: Where do I put my networks?

mattjk — Mon, 21 Apr 2008 06:11:33 GMT

Yes I did tell you the 1800 will be fine, and it is when an ESX farm is small. The 2900 is much more powerful and would do better under heavy load and thats why I perfer it for iSCSI. But you need it for LAN routing so it will have to stay there.

Our farm definately counts as small - only 2 ESX nodes to begin with At least with the 1800's we can start out cheap, and then replace with better switches if our load increases to the point where it's necessary.

You also were looking at jumbo frames, one caution there is that HP flowcontrol and jumbo frames are sometimes an issue when combined.

Some iSCSI product vendors (Lefthand) recommend jumbo frames be turned off on the lowered end switches if you are using flow control.

Flow control will be the better option to leave on for peek load performance.

The 2900 series can handle both on at the same time. The reason for it is buffering and processing power.

Thanks for the tip, I'll be sure to enable flow control but disable jumbo frames.

I have the 1800 and I don't have issues at 50% load and I have not used jumbo frames yet.

I also have the 2900 and it really is a work horse, you can pin it and it still continue to perform.

Great, thanks.

As for the SC if you use HA in the future this feature will want redundant network paths across the network for all hosts. Software iSCSI also needs an SC to on the same iSCSI network. This will result in multiple SC's across the networks and you may not what to place them solely on one pNIC pair going over the 1800. So think of the SC as more that just a managment console is what I am saying.

We want to use HA from the start so this sounds like it'll be important to us

Are you saying that:

1) the SC needs redundant network paths, and,
2) the SC will need to participate in more than just 1 network - i.e. for iSCSI authentication

?

If so, I think we have #1 covered by the fact that both the 1800 and 2900 will be paired (i.e. we'll actually have 2 x 1800's and 2 x 2900's), meaning we have network path redundancy, and I thought #2 can be managed by adding a SC port to the iSCSI vSwitch.

Have I mis-understood / missed anything?

Re: Where do I put my networks?

mike.laspina — Sun, 20 Apr 2008 15:40:17 GMT

You also were looking at jumbo frames, one caution there is that HP flowcontrol and jumbo frames are sometimes an issue when combined.

Some iSCSI product vendors (Lefthand) recommend jumbo frames be turned off on the lowered end switches if you are using flow control.

Flow control will be the better option to leave on for peek load performance.

The 2900 series can handle both on at the same time. The reason for it is buffering and processing power.

I have the 1800 and I don't have issues at 50% load and I have not used jumbo frames yet.

I also have the 2900 and it really is a work horse, you can pin it and it still continue to perform.

Re: Where do I put my networks?

mattjk — Sun, 20 Apr 2008 04:07:46 GMT

1) iSCSI will utiliize a switch much more heavily that the network, I would use the 2900 series for iSCSI

Didn't you tell me a couple of days ago - http://communities.vmware.com/message/915061#915061 and http://communities.vmware.com/message/918825#918825 - that the 1800's should be fine for iSCSI?

Regardless, I don't think we have much choice: we need some of the features offered in the 2900 on the LAN side (L3 routing, etc), and can't afford to buy 4 x 2900's (2 for SAN, 2 for LAN). So, we're "stuck" with the 1800's for the SAN, unless an alternative solution can be found. I might be able to afford 2810-24G's instead of 1800's, but it'll be pushing it.

2) SC and VMotion are the lower priority in terms of normal activity. If you have a large farm the activity increases but what is the percentage of the whole....

We only have a small farm

3) You may need more than one SC in an average config so you have to rethink it from the physical side

Could you elaborate on what you mean here please?

4) VMotion and SC do need security isolation but do not mix it with iSCSI because it is normal network traffic and this goes against best practice

OK, that's what I was looking for. Thanks.

Re: Where do I put my networks?

mattjk — Sun, 20 Apr 2008 04:01:28 GMT

There are lots of posts about this out there check out these:

<snip>

All pretty much say the same thing however, you really want 6 pNIC for your load not just 2. But this depends on your Security Policy, Regulatory Stance, and comfort level.

I think you may have mis-understood my question as none of those posts seem to answer it.

It's not regarding how many NICs - we already will have lots of pNICs (10 actually, not 6) - in my original post I said "Our ESX deployment will include the following networks, with 2 x pNICs in each ESX host for each network".

The question is regarding where to physically connect the SC and vMotion networks - i.e. which switch pair (out of the two described) should they connect to?

Re: Where do I put my networks?

mike.laspina — Sat, 19 Apr 2008 13:29:36 GMT

Hi,

Here is my thinking.

1) iSCSI will utiliize a switch much more heavily that the network, I would use the 2900 series for iSCSI

2) SC and VMotion are the lower priority in terms of normal activity. If you have a large farm the activity increases but what is the percentage of the whole....

3) You may need more than one SC in an average config so you have to rethink it from the physical side

4) VMotion and SC do need security isolation but do not mix it with iSCSI because it is normal network traffic and this goes against best practice.

Re: Where do I put my networks?

Texiwill — Sat, 19 Apr 2008 13:27:12 GMT

Where do I put my networks?

mattjk — Sat, 19 Apr 2008 09:41:59 GMT

Hi all,

Quick question regarding where it is best to "place" vMotion and SC (Management) networks.

Our ESX deployment will include the following networks, with 2 x pNICs in each ESX host for each network:

vMotion
SAN (iSCSI and/or NFS)
SC (Management network)
LAN (a couple of different ones seperated by VLANs)
DMZ

These will be connected to 4 physical switches - 2 pairs of 2 for redundancy. I think we're going to use 2 x ProCurve 1800-24G for the SAN network, and 2 x ProCurve 2900-24G for the LAN & DMZ networks (separated by VLANs).

This leaves the vMotion and SC (Management) networks.

Which of the pairs of switches should we run these two through? Are these any special considerations?

I'm leaning towards putting both the vMotion and SC networks on the SAN switches, firstly because they will both be rather simple networks in our small environment, and secondly because the SAN switches will be isolated from other traffic so vMotion and SC security will be higher.

Whichever way we go we'll of course be using VLANs to seperate the different networks inside the switch.

Thoughts? Opinions?

Cheers,
Matt