Ok, I have an aberdeen san running Nexenta. Two interfaces configured on the box, 1 for MGMT: 192.168.55.0/24 and one dedicated for iSCSI tengi 192.99.99.0/24. That SAN is advertising itself on both interfaces to my ESX hosts. ESX hosts have 2 vmkernels:
vmotion: 192.168.55.x
iSCSI: 192.99.99.x
Since the SAN is advertising both targets my esx hosts have 2 paths to every LUN and they are using the vmotion kernel instead othe iSCSI one. I removed the vmotion kernel on a couple of hosts, one took a shit and lost connectivity to the SAN so I had to add it back. The other kept on chugging without fail but for my sanity I added it back.
Now why would it choose the mgmt lan route over the iSCSI route?