We've been doing software iSCSI initiator for a while now with regular broadcom nics.
If we buy the more expensive nics with TCP/IP offload engine, will vSphere be able to make use of it to do iSCSI?
VMware does not at this time support TOE enabled NICs. TOE enabled NICs will function as regular GbE devices under ESX(i). If you want iSCSI offloading under ESX(i) you’ll need an iSCSI HBA.
This is the case for 3.5, I don't think it has changed for 4.
Also vSphere does not use yet TOE or IOAT.
But a VM with a card assigned with VMDirectPath can use this features.