Have you tried watching the %DRPRX in ESXTOP ? see if any packets are getting dropped? Is this a gigE adapter? could also check the physical port and see if there are any errors present there.
Here is a couple of docs that may help.
http://communities.vmware.com/docs/DOC-5240
http://communities.vmware.com/docs/DOC-5500
