I understand what you are saying (and in fact have redudant 3850's dedicated for back end storage traffic only in my environment) but realize that for smaller shops the questions is "do I buy a switch for iSCSI (or NFS as the case may be) or use my existing switches"?

In my opinion redundancy is more important than isolated hardware. So I'd rather use a dedicated VLAN for iSCSI across multiple switches then a single switch for iSCSI only.

Obviously if you can afford redundancy and it makes business sense to pay for it, do so. Keep in mind that the VMware best practices are written from VMware's point of view, i.e. what's going to make their product as bulletproof as possible. That does not always equate to what's best for your business, which is why there is so much discussion on the topic.

Great cautionary tale.

To me though this isn't a tle of why you should have seperate switches, the outage would have occurred when the update was applied to the seperate switches.

The way to avoid the outage would be to have a test environment that replicates the production environment and to test the update before applying it to your core switches.

More the better.. ask your iSCSI SAN / storage array vendor for recommended/supported/tested switches..

If you need to choose between jumbos and flow control you should choose flow control.. in a bigger environment it wil help more than jumbos. The point with good flow control is that it will prevent the buffers of the switch (port) getting full and the switch having to drop packets.. this could happen often in a bigger environment with many initiators and/or switches.
if the switch drops packets it will cause TCP retransmits and that makes TCP go slow.. which makes your iSCSI slow.

So good flow control will use pause frames preventing the switch buffers from overflowing and this will keep your traffic flows (iSCSI) running smoothly and fast.

Note that you need to run flow control on every NIC/HBA and on every switch and on your storage arrays.

Bad/poor flow control implementation in the switch will also kill your performance.. good and cheap come in a different packages


I think even Cisco has some models with bad/poor flow control implementation.. usually Cisco 3750G-24 has been safe choise (and recommended by the storage vendors for iSCSI traffic).

HP is known to have many bad/problematic models, and some OK models too..

Hopefully that helps.

Yes, you absolutely need to use switches with "wirespeed" backplace..

The problem I was referring to happens with every switch, even with good ones if you're not running (good) flow control..

Example: Your storage array is connected to your switch with a single gigabit connection. You have two initiators (servers) accessing the storage.
Now think about both of these servers writing to the storage array at the same time..

What happens? Obviously the port on the switch where storage array is connected gets full, because it can only take one gigabit/sec in.. and you have two servers trying to push gigabit/sec into it, total of 2 gigabit/sec into it..
now the switch needs to drop packets when the port buffers get full -> tcp needs to do retransmits (causing delays) and adapt to slower speed to prevent packet loss.. in the end both servers talk at about 500 Mbit/sec max (sharing the gigabit connection), but all of this could have been done without packets loss and tcp retransmits.. by using flow control.

flow control in the switch sends "pause frames" before the buffers get full and tell the servers to wait for a short while so there's no need to drop packets.. these flow control "pause frames" are much shorter than tcp retransmits -> you get better and smoother performance.

Now, this very same problem happens also in a bigger environment when you have more bandwitdth from the initiators than you have bandwitdth to your storage array.. and this is usually always the case.

Fibre Channel has same kind of "flow control" built into it.. FCOE btw needs ethernet flow control too.

Matt-

Was wondering how well the 1800's worked out on the SAN side? Did you experience any performance issues?

I too am wondering how it went with the 1800s. We are quoting a small ESX implementation with HP iSCSI storage, and wanted to use the 1800s for our iSCSI (dedicated) network.

Thanks,