We recently migrated 4 IBM HS20 and 2 IBM HS21 blades running VMware ESX from a Fiber Channel Boot-from-SAN to an iSCSI Boot-from-SAN configuration. We used the IBM 32R1923 iSCSI HBA which uses the qla4022 driver. It's basically the IBM blade server equivalent of the Qlogic QLA4052C.
Setting up Boot-from-SAN with these adapters was easy, but was not without a few issue. Mainly, involving failover of the COS (Console OS) in the event of a complete path failure. One of the issues was pretty simple, the default KeepAliveTimeout (KATO) for the boot volumes was set to 30 seconds as that's what the HBA is set to "out of box". The problem with this is that the failover time can basically be calculated by n2+5 where in is basically the KATO time (the ql4xportdownretrycount, which defaults to 14, sets this value for the dynamically discovered volumes -- volumes discovered by the settings you use in ESX itself -- but the boot LUN has to be configured in the HBA BIOS since it obviously has to exist before ESX is loaded). Well, 302+5 is ~65 seconds, so it will take at least that long for the boot LUN to fail over, however, the COS also has a timeout in where it will set the /dev/sda device offline if it can't complete a write in ~60 seconds. Since the two timeouts are so close together it can happen that the console OS times out the device before the failover happens. Once this happens the COS becomes effectively unusable, the VM's will survive, but since the COS is pretty much unusable you can't Vmotion them off or anything. The easy fix here is to simply use the iscli utility to set the KATO timeout for the HBA to 14 globally for both ports, and also individually for the boot LUN targets (target 0 set the TGT_KATO value).
The above is enought for a normal Qlogic HBA, unfortunately the IBM 32R1923 has another issue which impacts failover and causes it to fail. It appears that IBM defaults the value of the "TaskManagementTimeout" to a very high value (2560) whereas normal Qlogic 4052HBA's use a more typical value (10). Getting the IBM adapters changed to 10 was painful. It was very easy to actually change the value for each port globally using the iscli utility, however, any statically bound volumes keep the value that was sent when they were bound. Since boot volumes have to be bound before acutally running the iscli utility the boot volume retained the 2560 value for it's targets. The iscli utility has an option to allow you to change the individual target values for the parameter but attempting to change them kept giving an error "HBA parameter value 10 invalid for TGT_TaskManagementTimeout". To correct this issue I had to set the TaskManagementTimeout value at the HBA level for both ports, reboot the system, use the HBA BIOS setup to clear all persistent bindings, and finally reconfigure the Boot LUN. This finally allowed the boot LUN to pick up the TaskManagementTimeout value of 10.
After getting the KATO and TaskManagementTimeout values configured properly for the Boot LUN's we've found that iSCSI boot from SAN works great. We can time a failover with a stop watch and get nearly exactly 33 seconds every time, just like our old FC setup and the COS survives every time.
It sounds like a lot of work, and I guess it is, but once you figure it out it's actually pretty easy, about 5 minutes a blade in our case. Making sure that COS failover worked was a hugely important deal for us because, in a blade center environment a switch module failure could potentially kill an entire path for every blade. You wouldn't want this scenario to take down the COS on all of your ESX servers because, even though the VM's actually survive you'll still have to eventually reboot every ESX server and since you can't Vmotion the VM's once the COS fails it would still mean downtime for the VM's eventually.
Later,
Tom
