ESX and TOE with ISCSI

ESX and TOE with ISCSI


We are looking at putting a couple of ESX boxes in and connecting to a  Equallogic SAN (HP DL380 G5's). Is there a general recommendation on  whether or not to use TOE cards or just normal multiport network cards? I  have seen articles that say not to bother, but wondering if anyone can  advise?


Are TOE Nic's supported in ESX yet? Not sure on this.

I would use the TOE cards since they supported with ESX 3.5.

Here's a summary of the what's new in this version:

  • Storage VMotion
  • Swapfiles-less VMotion
  • Update Manager
  • Server consolidation advisor
  • Lockdown Mode
  • Distributed Power Management (DPM)
  • Guest OS disaster recovery capability
  • Provisioning across datacenters objects
  • Integration with VMware Consolidate Backup (VCB) and VMware Converter 4.0
  • Integration with Wyse multimedia redirection engine
  • Support for VMware Server 2.0
  • Support for AMD Rapid Virtualization Indexing (formerly Nested Page Tables or NPT)
  • Support for Cisco Discovery Protocol (CDP)
  • Support for InfiniBand network cards
  • Support for 10Gbit Ethernet network cards
  • Support for TCP/IP Offload Engine (TOE) network cards
  • Support for network load balancing algorithms
  • Support for N_Port ID Virtualization (NPIV)
  • Support for VCB over iSCSI SANs
  • Support for IPv6 in virtual networking
  • Support for para-virtualized Linux guest OSes
  • Support for Intel I/O Acceleration Technology (IOAT)
  • Support for Jumbo Frames
  • Support for Open Virtual Machine Format (OVF)
  • Support for SATA storage devices
  • Support for 200 hosts and 2000 virtual machines
  • Support for 128GB RAM per host and for 64GB RAM per virtual machine

That's misleading. TOE cards are not supported ion ESX 3.5. I think what they mean is that iSCSI offload cards (which are TOE cards under the hood) are supported.

That's misleading. TOE cards are not supported  ion ESX 3.5. I think what they mean is that iSCSI offload cards (which  are TOE cards under the hood) are supported.

Umm... almost correct. A select number of TOE cards ARE supported for VM  and VMkernel traffic, so it would benefit NFS from what i've read.

iSCSI otoh doesn't support TOE when using a normal NIC.

Here's some threads on the topic:
Re: TCP Offload Engine and other similar "Inside NIC" Features
TOE/TSO/IOAT Functionality
Re: VI3 v3.5: Software iSCSI with TCP offload
Re: VMware 3.5 - dissapointing news!


Where did you get that information that TOE cards are not supported in ESX 3.5.

The 3.5 release notes dont specifically specify if TOE NICs are supported or not.

All it says is

Support  for networking performance optimizations such as TCP Segment  Offload and jumbo frames that reduce the overhead associated with  processing network i/o.

I might be wrong but I got this information from VMware support last time I spoke to them also from a few moderators blogs.

As I said I might be wrong but it would really helpful if you would be  able to provide us with a link to an official vmware white paper or  document that clarifies this.



I don't have any reference papers, but look at the HCL and the nics  supported - there is no TOE NIC on the HCL. Also, as you noted, there is  no mention of TOE in the release notes  - I take that as an assertion  that it's not supported. TSO and Jumbo frames are supported ofcourse.

A select number of TOE cards ARE supported

That's news to me. Can you tell me which TOE cards?

Good question, if i recall it correctly then the intel PRO 1000 should  work. I'm using these in my lab, but they are now configured for iSCSI  traffic and don't have the time to change the config for testing this  out.

More info can be found in the ESX Server 3 Configuration Guide page 61 and onwards.


intel PRO 1000 is not a TOE nic, afaik.

You might be right there, I probably need to get my abbreviations sorted if TOE != TSO
The adapter does support the latter



ESX 3.5 does have support for jumbo frames, however the Software iSCSI  initiator (and the NFS module) does not support jumbo frames.  The  following blog posting gives a basic overview and some general usage  tips:

If you want to invest in hardware to improve iSCSI performance, get a  proper iSCSI HBA that is listed on the IO compatibility guide.  Also  take into consideration that 10G Ethernet is now an option with 3.5.

Message was edited by: jhanekom
("use recovered post" ate half my post.)

I am almost certain that our Dell 2950 got TOE support when ESX 3.0.1  was released.  I think the network cards in that system were  Broadcom-based network cards.

Thanks for all the comments so far.

So on a DL380 G5 (where the two onboard ports support TOE I believe), you would be best to have:

1) The two internal ports connected to the SAN (and the VMkernal mapped to those).

2) Your other network ports (on a none-TOE card) for ethernet client/server traffic for the VM's.

Does that sound correct?

And if I'm not mistaken, those HP NC373i nics on DL380 G5 (that HP  claims are TOE nics) are driver by the bnx2 driver (looking at  esx3_io_guide.pdf) - which is a standard L2 etherner driver. Better get  some clarification from HP on what kind of TOE benefit you might get  from these nics.

FWIW, here's what the linux guys have to say about TOE:

This document was generated from the following thread: ESX and TOE with ISCSI

Version history
Revision #:
1 of 1
Last update:
‎12-02-2009 02:30 PM
Updated by: