iSCSI considerations Host vs VMHost and HBA vs NIC ESX 3.5

[Brian_Wing]

Howdy Y'all

We're delving into the iSCSI, shared storage VMWare foray and I'm looking for opinions, experiences and recommendations.

First off, is there any major benefit of using a dedicated iSCSI HBA vs a regular ethernet NIC? If so brand/model recomendations? I'm perusing the I/O compat guide now...

Secondly,

what considerations should I look at on whether to have ESX connect to and manage the storage/LUN or should I be configuring LUNs for each and every host I create on ESX? Am I confused on this concept? Any good reads, whitepapers, etc...?

Finally,

we do want to multipath our iSCSI SAN and utilize VLANs to manage network separation. Are there any major gotchas on configuration of the networking for this SAN? Frame size, ARP cache settings, other things a regular network administrator might overlook?

Thanks everyone, in advance for your assistance.

Reference threads and documents:

Ethernet-based Storage Configuration (a vmware technical note attached)

[kukacz]

Brian,

the HBA vs. NIC question was discussed many times before, try to search in the history.

Basically, common NIC provides more options of iSCSI usage like aggregating paths, connecting from VM-level initiator etc. Perhaps the only HBA advantage is some CPU time saved by performing the iSCSI protocol offload.

Common ESX storage scenario I'm applying is creating a single 200-500GB LUN formated as VMFS datastore for virtual machines' system drives. Then I attach LUNs for storing database files, file depots etc. as a raw device mapping or through the VM-level iSCSI initiator. The VM-level initiator is a must in cases where your SAN vendor requires it to perform integrated LUN snapshots.

Regarding the switches - just consider two things: redundancy and physical separation of iSCSI and LAN traffic. In an extreme case you'll come to 4 switches needed. As a compromise you might use 2 switches running a shared iSCSI/LAN traffice separated by VLANs.

About the frame sizes (Jumbo frames): they might help you gain few percents of performance but will need lots of settings at the ESX side + switch level + array level. Sometimes it's not worth of the work. The change helps mostly in sequential traffic (file transfers) but the real effect differs between vendors.

--

Lukas Kubin

[Brian_Wing]

Thanks Lukas, good stuff. We're evaluating a couple of different storage vendors. We are a netapp shop currently and I hear they're getting ready to release something like snap manager for exchange for VMWare, so I'm thinking we may hold off on deciding our storage solution until that's been released. We're also looking at the Dell EqualLogic product, so who knows exactly.

I'm also hearing scuttlebut about using NFS instead of iSCSI but not sure what that will end up looking like.

Thanks again.