We are looking at putting a couple of ESX boxes in and connecting to a Equallogic SAN (HP DL380 G5's). Is there a general recommendation on whether or not to use TOE cards or just normal multiport network cards? I have seen articles that say not to bother, but wondering if anyone can advise?
Are TOE Nic's supported in ESX yet? Not sure on this.
I would use the TOE cards since they supported with ESX 3.5.
Here's a summary of the what's new in this version:
That's misleading. TOE cards are not supported ion ESX 3.5. I think what they mean is that iSCSI offload cards (which are TOE cards under the hood) are supported.
Umm... almost correct. A select number of TOE cards ARE supported for VM and VMkernel traffic, so it would benefit NFS from what i've read.
iSCSI otoh doesn't support TOE when using a normal NIC.
Here's some threads on the topic:
Re: TCP Offload Engine and other similar "Inside NIC" Features
Re: VI3 v3.5: Software iSCSI with TCP offload
Re: VMware 3.5 - dissapointing news!
Where did you get that information that TOE cards are not supported in ESX 3.5.
The 3.5 release notes dont specifically specify if TOE NICs are supported or not.
All it says is
Support for networking performance optimizations such as TCP Segment Offload and jumbo frames that reduce the overhead associated with processing network i/o.
I might be wrong but I got this information from VMware support last time I spoke to them also from a few moderators blogs.
As I said I might be wrong but it would really helpful if you would be able to provide us with a link to an official vmware white paper or document that clarifies this.
I don't have any reference papers, but look at the HCL and the nics supported - there is no TOE NIC on the HCL. Also, as you noted, there is no mention of TOE in the release notes - I take that as an assertion that it's not supported. TSO and Jumbo frames are supported ofcourse.
That's news to me. Can you tell me which TOE cards?
Good question, if i recall it correctly then the intel PRO 1000 should work. I'm using these in my lab, but they are now configured for iSCSI traffic and don't have the time to change the config for testing this out.
More info can be found in the ESX Server 3 Configuration Guide page 61 and onwards.
intel PRO 1000 is not a TOE nic, afaik.
You might be right there, I probably need to get my abbreviations sorted if TOE != TSO
The adapter does support the latter
ESX 3.5 does have support for jumbo frames, however the Software iSCSI initiator (and the NFS module) does not support jumbo frames. The following blog posting gives a basic overview and some general usage tips: http://www.yellow-bricks.com/2008/01/03/what-about-those-jumbo-frames/
If you want to invest in hardware to improve iSCSI performance, get a proper iSCSI HBA that is listed on the IO compatibility guide. Also take into consideration that 10G Ethernet is now an option with 3.5.
Message was edited by: jhanekom
("use recovered post" ate half my post.)
I am almost certain that our Dell 2950 got TOE support when ESX 3.0.1 was released. I think the network cards in that system were Broadcom-based network cards.
Thanks for all the comments so far.
So on a DL380 G5 (where the two onboard ports support TOE I believe), you would be best to have:
1) The two internal ports connected to the SAN (and the VMkernal mapped to those).
2) Your other network ports (on a none-TOE card) for ethernet client/server traffic for the VM's.
Does that sound correct?
And if I'm not mistaken, those HP NC373i nics on DL380 G5 (that HP claims are TOE nics) are driver by the bnx2 driver (looking at esx3_io_guide.pdf) - which is a standard L2 etherner driver. Better get some clarification from HP on what kind of TOE benefit you might get from these nics.
FWIW, here's what the linux guys have to say about TOE: http://www.linux-foundation.org/en/Net:TOE
This document was generated from the following thread: ESX and TOE with ISCSI