Well, my guess is, like seemingly everyone else who tries to setup this configuration, you've probably built an invalid network configuration. Having two nics in a team that go to two different switches that are not connected to each other is not a valid configuration. I don't know what your configuration looks like and I don't know anything about the MD3000i, but the typical configuration is something like this:
vSwtich1pNIC1pSwtich1Controller1/Port110.10.10.1
vSwitch1pNIC1pSwitch1Controller2/Port110.10.10.2
vSwitch1pNIC2pSwitch2Controller1/Port210.10.10.3
vSwitch1pNIC2pSwitch2Controller2/Port210.10.10.4
If the two physical switches are not trunked together this is a completely invalid configuration and you will probably only be able to see half of your ports at any given time. Why? Well, you've created a network where a given IP address is only reachable via one physical path from the server but the vSwitch has no way of knowing which pNIC any give IP might be reachable on. Let's look at what happens if you choose various load balancing algorithms:
IP Hash -- This load balancing policy simply picks a port using a simple algorithm which simply hashes the IP address and uses that hash to send the packet out. If the hash for 10.10.10.2 sends the packet via pNIC2 then it will never arrive at the storage array since the only way to reach 10.10.10.2 is via pNIC1.
VPort ID -- This load balancing policy simply pretends that the virtual switch has "ports" just like a physical switch, it then hashes the "port numbers" and sends all traffic for the given "virtual port" out a single pNIC. Since the software iSCSI initiator uses only a single connection via the VMkernel, thus only a single "virtual port" then every packet from the iSCSI initiator will go out of the same pNIC so basically the iSCSI initiator will try to reach all four IP addresses out of the same pNIC and thus will only be able to reach half of them.
Of course, if you pull the cable failover will kick in and the packets are forced to go out of the other pNIC and it will magically "find" the other paths.
The main reason for this issue is that the software iSCSI initiator in ESX only supports a single session. With hardware iSCSI this is not an issue since each HBA acts as a seperate initiator. Each hardware HBA would ack as a separate initiator and would only see two paths, but the HBA multipathing software would combine the 2 paths from the 2 HBA's into a single LUN with 4 paths.
In my opinion the only workable solution is to create an actual network that meets the normal rules of networking (the virtual world does not somehow suspend the rules of networking 101). The easiest way to do this is to simply interconnect the two switches, either with a trunk, or, since you have Cisco 3750's, the StackWise cable. This provides a significant level of redundancy while providing a single network that will work well with the software iSCSI initiators failover methods. It's not perfect since the two fabrics are not completely separate. SAN admins from a fiber channel background will really balk at not having completely separate fabrics, which helps in the case of an administrative failure in zoning or a complete fabric failure, but it's still probably the best option with the current limitations of the software iSCSI in ESX as this failure mode is fairly uncommon in IP networks. Even with the switches stacked you can still loose a swtich or replace a switch without loosing connectivity or incurring downtime but it is still possible that an administrative error or traffic storm could potentially take down the network, even if that likelyhood is remote.
Now, some people have reported that they can leverage the IP Hash load balancing policy to still make the separate switches work. Since the IP hashing algorithm is deterministic (the same IP will always be sent out of the same pNIC) you could theoretcially determine with pNIC would be used for each IP address and configure the storage array ports to use these IP. I'm not conviced this is a good idea as it seems fragile especially if you start thinking about failure modes like storage array controller failures rather than just network failures but since I think the MD3000i appears like an "active/active" array I guess it might work.
Others have reported that you can just use the pNICS with a failover policy and that the iSCSI software initiator will rescan when the link fails. I'm not convinced that this would be truly robust either, but I suppose it's not out of the question as long as you put all of the IP addresses for all of the ports in as discovery addesses and made sure the iSCSI initiator is conigured for continous discovery (which I think is the default). Still, that doesn't change the fact that the configuration is technically "broken".
Later,
Tom
