The failure seems to be at the vSphere networking level though, so it doesn't look like a Windows nor SAN vendor problem.
I have built a two node ESX4i (4.0.0U1) cluster using an EqualLogic PS4000XV which is connected as per the EqualLogic best practice documents. Namely, on each node:
. The vSphere software initiator is bound to four separate vmkernel ports (one per path is recommended).
. The iSCSI vSwitch uses two physical NICs, one into each dedicated iSCSI network switch (PowerConnect 5424).
. EqualLogic say you should bind half of the iSCSI vmkernels to a single pNIC, with the other pNIC configured as unused, and the other half of the iSCSI vmkernels should reverse that arrangement..
. Jumbo frames are enabled on the iSCSI vSwitch on each ESXi node at creation time, on the vNICs, and on the physical network switches.
. The two physical iSCSI switches use flow control, and have a 6Gbps link aggregation trunk between them.
So far so normal. This all works and vSphere fails over its storage connections as expected when cables are disconnected.
The problem is with guest VMs. For some of these I need to connect iSCSI LUNs using the Microsoft iSCSI Initiator within the VM so I can use Exchange- and SQL-aware off-host backup using Backup Exec - which leverages the EqualLogic Hardware VSS provider. For this I need to create two additional Virtual Machine Port Groups ("VM iSCSI 1" and "VM iSCSI 2") attached to the same vSwitch the vSphere iSCSI initiator is using, and create a vNIC in each (see screenshot):
As with those vmkernel ports, one is bound to the 1st pNIC with the 2nd pNIC unused, and the other Virtual Machine Port Group is configured in the opposite way. In the Guest OS (Windows 2003R2) the EqualLogic Host Integration Tools (3.3.1) see the two adapters and enable multipathing. The tools show an iSCSI connection to the SAN from each adapter.
What I discovered recently when I needed to do some maintenance is that if I disconnect one of the iSCSI pNICs the vSphere iSCSI connections fail over fine, but those iSCSI sessions that originate within the VMs do not. In the failed state I could not ping the SAN from these VMs. Since production systems were down I couldn't afford time to troubleshoot so I had to reconnect the physical cable and reboot them. All three VMs with Guest-attached LUNs failed in this way. I have of course double-checked the Networking configs on both ESXi cluster nodes. I am soon going to move all VMs onto one node for some unrelated maintenance, so I shall be able to test in more detail.
So, has anyone else encountered this behaviour? Is there perhaps a better way I should configure these VMs for iSCSI multipathing?
VMware have finally come back with the same opinion as Dell - that I need to use a separate vSwitch for it to be supported, so two more physical NICs.
Their tech said I could set up the guest with 1 vNIC and have that VM port group configured to use both physical NICs in an active/standby config. This would take care of the failover at layer 2 instead of relying on the Eql MPIO driver, however I would like the guest VMs to have true multipath since I want the max. I/O possible (2Gbps).
I may not have time to test soon because I have a week of holiday, then a week on training - to become VCP 
Thanks Will. These descriptions are often quite complex to follow, but from what I've understood from your post, it sounds as if to achieve what I wanted using 2 physical NICs I'd need to change to the kind of setup below which I drew in MSPAINT (so a single vSwitch per Physical NIC).
Is there any downside to this? I can't really see any, but of course you'll need to rewrite your Dell EqualLogic best practice guide. If you think that's fine then I'll give that a try as soon as I can.
As for why there is a path fail issue in the first place - would you agree that the issue is quite likely to be with the EqualLogic DSM driver? The best way to test would be to simulate the same failure on a physical machine running HIT, refer to Jelle's diagram of the network connectivity (have two additional physical switches to simulate the vmnics), and kill not the NIC uplink but the connection between these switches for one of the paths like so:
If my suspicion is correct, the HIT will continue to send traffic down the now dead network segment, just because the uplink is still ok, whereas presumably it should regularly be testing the path end to end.
I'm so glad you were able to replicate the problem though - huge thanks for that! Sometimes I get the impression I must be the first guy to try this...
I've tested failover with my ESXi host config matching that network config above and whilst it does now fail over without killing the VM's instance of SQL I'm still not convinced it's working properly.
Here's what happens:
1) If I disconnect vmnic1, the hypervisor marks the two disconnected paths as dead in the Storage Manage Paths view. If I look in the Eql MPIO tab in MS iSCSI Initiator on the Guest I can see that only one connection exists to the SAN, and in a few minutes it doubles up. So two connections from a single IP at the Guest end (one to eth0 and one to eth1 on the SAN). The vss-control connection is maintained throughout. So in theory, my off-host VSS backups would be ok in this scenario.
2) I reconnected vmnic1 and let it settle until the MPIO tab in MS iSCSI Initiator showed a connection from each of the Guest's vNICs.
3) If I disconnect vmnic5, the hypervisor again fails over fine. The Guest froze up for about 5 seconds but crucially didn't fail. If I look in the Eql MPIO tab in MS iSCSI Initiator I can see only one connection, however - no vss-control connection:
Now this is where things seem to go wrong. I left it like this for about 20 minutes and it never initiated a second connection to the SAN from 192.168.200.49. So if a SAN interface failed later - I'd lose the guest. Improbable, but perfectly possible.
The vss-control connection was not re-established either, meaning that my backups would fail for this guest. I paid a lot of money to make sure this was redundant - but it seems that it isn't. Even clicking Refresh it stays like this:
4) I reconnected vmnic5. The hypervisor re-established all paths immediately, and on the Guest the vss-control connection was also reconnected promptly. I notice that it's never blue, and it's in the "Managed:No" category. Does this mean that its lack of redundancy is actually to be expected?
An entire 30mins after reconnecting vmnic5 I notice that there is still no reconnection for the failed Guest path. This VM has now lost its redundancy until a reboot forces it to reset its MPIO.
So all in all, given how much these companies market these technologies - they seem to be suprisingly rough around the edges. Should I really be testing to this degree? This is core product functionality. Can anyone else replicate this?