VMware Cloud Community
gheywood
Enthusiast
Enthusiast
Jump to solution

ESX and TOE with ISCSI

Hello,

We are looking at putting a couple of ESX boxes in and connecting to a Equallogic SAN (HP DL380 G5's). Is there a general recommendation on whether or not to use TOE cards or just normal multiport network cards? I have seen articles that say not to bother, but wondering if anyone can advise?

Cheers

Reply
0 Kudos
1 Solution

Accepted Solutions
jhanekom
Virtuoso
Virtuoso
Jump to solution

ESX 3.5 does have support for jumbo frames, however the Software iSCSI initiator (and the NFS module) does not support jumbo frames. The following blog posting gives a basic overview and some general usage tips: http://www.yellow-bricks.com/2008/01/03/what-about-those-jumbo-frames/

If you want to invest in hardware to improve iSCSI performance, get a proper iSCSI HBA that is listed on the IO compatibility guide. Also take into consideration that 10G Ethernet is now an option with 3.5.

Message was edited by: jhanekom

("use recovered post" ate half my post.)

View solution in original post

Reply
0 Kudos
14 Replies
Milton21
Hot Shot
Hot Shot
Jump to solution

Are TOE Nic's supported in ESX yet? Not sure on this.

Reply
0 Kudos
EdiB
Enthusiast
Enthusiast
Jump to solution

I would use the TOE cards since they supported with ESX 3.5.

Here's a summary of the what's new in this version:

  • Storage VMotion

  • Swapfiles-less VMotion

  • Update Manager

  • Server consolidation advisor

  • Lockdown Mode

  • Distributed Power Management (DPM)

  • Guest OS disaster recovery capability

  • Provisioning across datacenters objects

  • Integration with VMware Consolidate Backup (VCB) and VMware Converter 4.0

  • Integration with Wyse multimedia redirection engine

  • Support for VMware Server 2.0

  • Support for AMD Rapid Virtualization Indexing (formerly Nested Page Tables or NPT)

  • Support for Cisco Discovery Protocol (CDP)

  • Support for InfiniBand network cards

  • Support for 10Gbit Ethernet network cards

  • Support for TCP/IP Offload Engine (TOE) network cards

  • Support for network load balancing algorithms

  • Support for N_Port ID Virtualization (NPIV)

  • Support for VCB over iSCSI SANs

  • Support for IPv6 in virtual networking

  • Support for para-virtualized Linux guest OSes

  • Support for Intel I/O Acceleration Technology (IOAT)

  • Support for Jumbo Frames

  • Support for Open Virtual Machine Format (OVF)

  • Support for SATA storage devices

  • Support for 200 hosts and 2000 virtual machines

  • Support for 128GB RAM per host and for 64GB RAM per virtual machine

Reply
0 Kudos
hitchhiker
Enthusiast
Enthusiast
Jump to solution

That's misleading. TOE cards are not supported ion ESX 3.5. I think what they mean is that iSCSI offload cards (which are TOE cards under the hood) are supported.

Reply
0 Kudos
wila
Immortal
Immortal
Jump to solution

That's misleading. TOE cards are not supported ion ESX 3.5. I think what they mean is that iSCSI offload cards (which are TOE cards under the hood) are supported.

Umm... almost correct. A select number of TOE cards ARE supported for VM and VMkernel traffic, so it would benefit NFS from what i've read.

iSCSI otoh doesn't support TOE when using a normal NIC.

Here's some threads on the topic:

--

Wil

| Author of Vimalin. The virtual machine Backup app for VMware Fusion, VMware Workstation and Player |
| More info at vimalin.com | Twitter @wilva
Reply
0 Kudos
EdiB
Enthusiast
Enthusiast
Jump to solution

Hitchhiker

Where did you get that information that TOE cards are not supported in ESX 3.5.

The 3.5 release notes dont specifically specify if TOE NICs are supported or not.

All it says is

Support for networking performance optimizations such as TCP Segment Offload and jumbo frames that reduce the overhead associated with processing network i/o.

I might be wrong but I got this information from VMware support last time I spoke to them also from a few moderators blogs.

As I said I might be wrong but it would really helpful if you would be able to provide us with a link to an official vmware white paper or document that clarifies this.

Thanks

ed

Reply
0 Kudos
hitchhiker
Enthusiast
Enthusiast
Jump to solution

I don't have any reference papers, but look at the HCL and the nics supported - there is no TOE NIC on the HCL. Also, as you noted, there is no mention of TOE in the release notes - I take that as an assertion that it's not supported. TSO and Jumbo frames are supported ofcourse.

Reply
0 Kudos
hitchhiker
Enthusiast
Enthusiast
Jump to solution

>> A select number of TOE cards ARE supported

That's news to me. Can you tell me which TOE cards?

Reply
0 Kudos
wila
Immortal
Immortal
Jump to solution

Good question, if i recall it correctly then the intel PRO 1000 should work. I'm using these in my lab, but they are now configured for iSCSI traffic and don't have the time to change the config for testing this out.

More info can be found in the ESX Server 3 Configuration Guide page 61 and onwards.

--

Wil

| Author of Vimalin. The virtual machine Backup app for VMware Fusion, VMware Workstation and Player |
| More info at vimalin.com | Twitter @wilva
Reply
0 Kudos
hitchhiker
Enthusiast
Enthusiast
Jump to solution

intel PRO 1000 is not a TOE nic, afaik.

Reply
0 Kudos
wila
Immortal
Immortal
Jump to solution

You might be right there, I probably need to get my abbreviations sorted if TOE != TSO

The adapter does support the latter

thanks

Wil

| Author of Vimalin. The virtual machine Backup app for VMware Fusion, VMware Workstation and Player |
| More info at vimalin.com | Twitter @wilva
Reply
0 Kudos
jhanekom
Virtuoso
Virtuoso
Jump to solution

ESX 3.5 does have support for jumbo frames, however the Software iSCSI initiator (and the NFS module) does not support jumbo frames. The following blog posting gives a basic overview and some general usage tips: http://www.yellow-bricks.com/2008/01/03/what-about-those-jumbo-frames/

If you want to invest in hardware to improve iSCSI performance, get a proper iSCSI HBA that is listed on the IO compatibility guide. Also take into consideration that 10G Ethernet is now an option with 3.5.

Message was edited by: jhanekom

("use recovered post" ate half my post.)

Reply
0 Kudos
Sly
Contributor
Contributor
Jump to solution

I am almost certain that our Dell 2950 got TOE support when ESX 3.0.1 was released. I think the network cards in that system were Broadcom-based network cards.

Reply
0 Kudos
gheywood
Enthusiast
Enthusiast
Jump to solution

Thanks for all the comments so far.

So on a DL380 G5 (where the two onboard ports support TOE I believe), you would be best to have:

1) The two internal ports connected to the SAN (and the VMkernal mapped to those).

2) Your other network ports (on a none-TOE card) for ethernet client/server traffic for the VM's.

Does that sound correct?

Reply
0 Kudos
hitchhiker
Enthusiast
Enthusiast
Jump to solution

And if I'm not mistaken, those HP NC373i nics on DL380 G5 (that HP claims are TOE nics) are driver by the bnx2 driver (looking at esx3_io_guide.pdf) - which is a standard L2 etherner driver. Better get some clarification from HP on what kind of TOE benefit you might get from these nics.

FWIW, here's what the linux guys have to say about TOE: http://www.linux-foundation.org/en/Net:TOE