What is your physical NIC? There is a driver issue with the ConnectX-3 NIC that causes symptoms like that. I had issues with mine and changed to an Intel brand NIC and no more issues.
