ESX and TOE with ISCSI

Hello,

We are looking at putting a couple of ESX boxes in and connecting to a Equallogic SAN (HP DL380 G5's). Is there a general recommendation on whether or not to use TOE cards or just normal multiport network cards? I have seen articles that say not to bother, but wondering if anyone can advise?

Cheers

Are TOE Nic's supported in ESX yet? Not sure on this.

I would use the TOE cards since they supported with ESX 3.5.

Here's a summary of the what's new in this version:

Storage VMotion
Swapfiles-less VMotion
Update Manager
Server consolidation advisor
Lockdown Mode
Distributed Power Management (DPM)
Guest OS disaster recovery capability
Provisioning across datacenters objects
Integration with VMware Consolidate Backup (VCB) and VMware Converter 4.0
Integration with Wyse multimedia redirection engine
Support for VMware Server 2.0
Support for AMD Rapid Virtualization Indexing (formerly Nested Page Tables or NPT)
Support for Cisco Discovery Protocol (CDP)
Support for InfiniBand network cards
Support for 10Gbit Ethernet network cards
Support for TCP/IP Offload Engine (TOE) network cards
Support for network load balancing algorithms
Support for N_Port ID Virtualization (NPIV)
Support for VCB over iSCSI SANs
Support for IPv6 in virtual networking
Support for para-virtualized Linux guest OSes
Support for Intel I/O Acceleration Technology (IOAT)
Support for Jumbo Frames
Support for Open Virtual Machine Format (OVF)
Support for SATA storage devices
Support for 200 hosts and 2000 virtual machines
Support for 128GB RAM per host and for 64GB RAM per virtual machine

That's misleading. TOE cards are not supported ion ESX 3.5. I think what they mean is that iSCSI offload cards (which are TOE cards under the hood) are supported.

That's misleading. TOE cards are not supported ion ESX 3.5. I think what they mean is that iSCSI offload cards (which are TOE cards under the hood) are supported.

Umm... almost correct. A select number of TOE cards ARE supported for VM and VMkernel traffic, so it would benefit NFS from what i've read.

iSCSI otoh doesn't support TOE when using a normal NIC.

Here's some threads on the topic:
Re: TCP Offload Engine and other similar "Inside NIC" Features
TOE/TSO/IOAT Functionality
Re: VI3 v3.5: Software iSCSI with TCP offload
Re: VMware 3.5 - dissapointing news!
--
Wil

Hitchhiker

Where did you get that information that TOE cards are not supported in ESX 3.5.

The 3.5 release notes dont specifically specify if TOE NICs are supported or not.

All it says is

Support for networking performance optimizations such as TCP Segment Offload and jumbo frames that reduce the overhead associated with processing network i/o.

I might be wrong but I got this information from VMware support last time I spoke to them also from a few moderators blogs.

http://virtualaleph.blogspot.com/2007/12/esx-35-released.html

http://vmware-land.com/

As I said I might be wrong but it would really helpful if you would be able to provide us with a link to an official vmware white paper or document that clarifies this.

Thanks

ed

I don't have any reference papers, but look at the HCL and the nics supported - there is no TOE NIC on the HCL. Also, as you noted, there is no mention of TOE in the release notes - I take that as an assertion that it's not supported. TSO and Jumbo frames are supported ofcourse.

A select number of TOE cards ARE supported

That's news to me. Can you tell me which TOE cards?

Good question, if i recall it correctly then the intel PRO 1000 should work. I'm using these in my lab, but they are now configured for iSCSI traffic and don't have the time to change the config for testing this out.

More info can be found in the ESX Server 3 Configuration Guide page 61 and onwards.

--
Wil

intel PRO 1000 is not a TOE nic, afaik.

You might be right there, I probably need to get my abbreviations sorted if TOE != TSO
The adapter does support the latter

thanks

Wil

ESX 3.5 does have support for jumbo frames, however the Software iSCSI initiator (and the NFS module) does not support jumbo frames. The following blog posting gives a basic overview and some general usage tips: http://www.yellow-bricks.com/2008/01/03/what-about-those-jumbo-frames/

If you want to invest in hardware to improve iSCSI performance, get a proper iSCSI HBA that is listed on the IO compatibility guide. Also take into consideration that 10G Ethernet is now an option with 3.5.

Message was edited by: jhanekom
("use recovered post" ate half my post.)

I am almost certain that our Dell 2950 got TOE support when ESX 3.0.1 was released. I think the network cards in that system were Broadcom-based network cards.

Thanks for all the comments so far.

So on a DL380 G5 (where the two onboard ports support TOE I believe), you would be best to have:

1) The two internal ports connected to the SAN (and the VMkernal mapped to those).

2) Your other network ports (on a none-TOE card) for ethernet client/server traffic for the VM's.

Does that sound correct?

And if I'm not mistaken, those HP NC373i nics on DL380 G5 (that HP claims are TOE nics) are driver by the bnx2 driver (looking at esx3_io_guide.pdf) - which is a standard L2 etherner driver. Better get some clarification from HP on what kind of TOE benefit you might get from these nics.

FWIW, here's what the linux guys have to say about TOE: http://www.linux-foundation.org/en/Net:TOE

This document was generated from the following thread: ESX and TOE with ISCSI