iSCSI and over subsciption of nics

[max08]

I have 4 ESX hosts in a cluster, they are connected via gig to a 3750X which is connected to our san via 10gig.  I have roughly 10 vms per hosts, give or take 2 or 3 on each.  Each host has a single gig link to the storage switch and all vms on each host traverse that nic to the storage.  We are seeing output drops on each of the nics attached to the hosts and I am wondering if there is anything I can do with the equipment I have to lessen the load on each nic?

[manfriday]

Am I understanding your config correctly?

Each host has only a single 1gb connection for it's storage? I can't see how that could be good for performance, and certainly it is not good for redundancy.

You should have at least two for redundency preferably more for performance. Before I went to 10gb nics I used to do 4 1gb nics per host just for iSCSI.

[max08]

Correct, there are 4 1gb connections, one per host to the san switch, then from the san switch to san is 10gig.  My san currently only has 1 target IP associated with it.

Those 4 1gb connections are saturdated on the way back from the SAN, so I am trying to alleviate it by adding more nics but wasn't sure how to do-so.

Any suggestions?

[manfriday]

When you say there is only one target IP.. does that mean there are only one 1gb uplink to the san itself?

Or is it like an equallogic where there is a single target IP, but then it manages the connections to multiple (4 in my case) actual uplinks on the controllers?

[golddiggie]

As mentioned, it's always a good idea to have multiple NIC's (on each host) for iSCSI traffic. I would configure them in an active/active configuration. Not the normal just two active NIC's assigned to one port group, but two port groups with each NIC assigned as either active or standby, then flipping the configuration for the other NIC. If possible, also look to enable jumbo frames for the iSCSI network...

Personally, in any production environment, with a decent number of VM's, I would have at least 4 [1Gb] NIC's for iSCSI traffic. I typically have 8-12 pNIC's in each host, allowing for proper redundancy as well as throughput in the configuration. If you're not using iSCSI storage, then you can get away with less pNIC's on each host. I've found that it really does depend on how many VM's you have, what they are, as well as what you have for a SAN and even switch hardware. If you line everything up well, you could get away with less pNIC's... But, I'd not go with anything less than 8 in each host.

Of course, in a lab setting, things are different. I have a home lab setup using 6 pNIC's, with two ports assigned across three vSwithces. One for the management network, one for VM traffic and the last for iSCSI traffic. I have jumbo frames enabled on the switch, for the VM and iSCSI ports, as well as on the SAN I'm using. It really does make a difference.

[max08]

No what I mean is the SAN has one iSCSI target.  Meaning there is only path to the SAN right now and that is via the 10gig interface. 

I am using jumbo frames but not flow control.

Can someone give me a quick run-down of what needs to happen to load-balance the traffic from the VMs to the SAN?  I assume I Just can't add another pnic to the vswitch and be done with it.  I would like to see it load balance the traffic across two pnics per host.  Just not exactly clear on how to accomplish that.

[golddiggie]

I would take a look at this page... http://pubs.vmware.com/vsp40_e/wwhelp/wwhimpl/js/html/wwhelp.htm#href=iscsi_san_config/t_connect_sof...

As well as review this pdf: http://www.vmware.com/pdf/vsphere4/r41/vsp_41_iscsi_san_cfg.pdf

[manfriday]

The quick and dirty run down of what you would have to do is:

Add n number of pnics to your vswitch

for each vmnic, create a VMKernel port.

for each vmkernel port, change the default failover order so that each has a single "active" adapter, and the others are "unused"

So if you had 4 vmnics (vmnic2,3,4,5) and 4 iscsi vmkprts (iscsi1,2,3,4)

You would do something like:

iscsi1:

Active adapter: vmnic2

Unused adapter: vmnic3,4,5

iscsi2:

Active adapter: vmnic3

Unused adapter: vmnic2,4,5

iscsi3:

Active adapter: vmnic4

Unused adapter: vmnic2,3,5

iscsi4:

Active adapter: vmnic5

Unused adapter: vmnic2,3,4

Then, bind the vmk ports to the iscsi HBA

Assuming your vmkport is vmk1, and your iscsi HBA is VMHBA33, the command would look something like:

esxcli swiscsi nic add -n vmk1 -d vmhba33

This is a very quick and dirty outline of how your would do this. I am sure that your SAN vendor has a whitepaper on the proper way to do this, and you should check out their documentation. Which SAN are you using?

[max08]

Aberdeen chassis with nexenta on top of it.   Can I get a screenshot?

[admin]

what are you actually running on these vm's? 1gb is a lot of bandwidth, more than people give credit too. I have a hard time pushing 25% on some of ours hosts with over 70 guests and 700 user file servers on them (dual quad's with 288gb memory).

[max08]

I think you're right.

I was looking at the bandwidth on the gig interfaces while everything slowed and the bandwidth was nowhere near a gig. So it appears to be something else.

My boss and cisco believe it might have something to do with the amount of packets per second we are pushing through the gig interfaces.

[max08]

Ok

I went ahead and did this on one of my hosts using two pnics and two vmk ports which I bound to my iscsi adaptor.  After re-scanning it came back with 3 paths instead of two for each my LUNs.  Weird no?  Only using two phsyical nics for this but for whatever reason it found 3 paths to my SAN all via the iSCSI IP.

Any idea why?

[GeekoCraig]

Did you ever figure all this out? BTW- Nexenta does support RoundRobin so that might help performance. Also those of you not familiar with Nexenta and ZFS it uses system memory as cache. This can be configured as 'read' or 'read/write' depending on your hardware-trust factor. Because of this, if given a lot of system memory (>8GB is common) one could achieve some high levels of IOPs. I have 8GB and some desktop 1TB 7200rpm/32MB drives and at times have seen over 6000IOPs and 100MB/sec on iSCSI.

-Craig

[max08]

Yes and no.   The output drops were because of the storage switch in the middle, it was a 3750X with very small buffers and since our VMs were doing heavy I/O the switch couldnt keep up and started dropping a shit ton of packets.

So we ripped almost everything out and replaced it with beefier equipment.  We were using a single 3750X that we replaced with (2) 4948s.  We also got a new vnxe3300 with 15TB to replace the aberdeen box.  I configured the vnxe for roundrobin and have (4) 10gig paths to and from the esx hosts.  As of now performance seems to be ok and a lot less latency than we were seeing with the aberdeen/nexenta POS.  The aberdeen at this point has been nothing but trouble.  From small DOM to failed chassis/back plane/drives I would not recommend it for any level of production envrionment.

Hope that helps.

[GeekoCraig]

Thanks for the feedback. I've not personally used the Aberdeen boxes but the Nexenta piece has been good. There are other storage vendors that OEM the Nexenta piece.

The setup you purchased is pretty nice. I have a little experience with the EMC stuff and yes, it does perform well. Also switches in an iSCSI network can make a HUGE difference. In production environments I look for ones with at least a 4-6 meg buffer. In the HP world the 2910s or something bigger or Cisco the 3750s of the ~$5000 price tag mark. Although recently I bought a Cisco SG300 and has a nice 4meg buffer for around 500$.