stisidore,
Are you running multiple, enterprise applications inside virtual machines on x86 servers today? You avoid the single point of failure of x86 motherboards through the magic of VMware Enterprise's HA, Vmotion, and DRS. Think of LeftHand Networks' SAN/iQ as ESX for storage.
In my time on the planet, I have seen how standards and market forces win over elegant, proprietary hardware designs for the majority of applications. See ethernet vs. token ring, PCI vs. MCA, VHS vs. Beta, and on and on. My electrical engineering degree cries about this often.
But the proof, as they say, is in the pudding. The performance of million dollar enterprise solutions like those you mentioned can be matched and exceeded by a cluster of HP ProLiant DL320s SAS storage modules powered by LeftHand Networks' SAN/iQ. See:
http://technet.microsoft.com/en-us/exchange/bb412164.aspx
On backplanes:
One of my former employers built storage arrays whose backplanes would flex on drive insertion / removal. This was enough to disrupt the signaling to neighboring drives. I have personally watched a RAID5 array go offline in an unrecoverable fashion due to a drive swap. I've also seen SANs go offline on controller insertion (misaligned pins caused a marvelous arc). Stuff happens.
RAID6 should be increasingly viewed as a must-have option for storage solutions - especially those employing SATA drives. The bit error rate of a SATA drive combined with their large capacities make the probability of an unrecoverable read error on rebuild fairly frightening. On a 41 array of 36GB drives, the probability was low. On a 61 array of 750GB SATA drives (two of these in a PS400E configured in RAID50), you have to hope that 6x750GB of data is readable for your rebuild to succeed. For a discussion on the benefits of "drive burn in" (or lack thereof), please read:
http://www.networkcomputing.com/channels/storageandservers/showArticle.jhtml?articleID=198700132
LeftHand has RAID6, but even better than that, we've got 2, 3, and 4-way replication (configurable by LUN/volume on-the-fly). This provides multiple copies of your data on the SAN, so you can tolerate multiple disk or controller failures without downtime. You get the ability to tolerate site failures for your virtual machines and data with automatic failover and failback. For example, with a simple 2-node configuration using the 12-drive HP DL320s in RAID5, I have 24 spindles at work. I can configure any or all of the volumes on the SAN such that I can lose 14 of these drives and still be running. I could put one of the two nodes in one building and the other one in another building - complete site-level redundancy for my virtual machines and data. With a single site, I could choose to split the nodes into different racks. And all of this scales - with 10 nodes, I could tolerate 5 controller failures. How well do your "million dollar enterprise solutions" survive 5 controller failures? How well to they hold up to the idiot from the vendor when he pulls the wrong power tail during a power supply swap?
On Broadcom:
The homework I've done is from EQL's website. It shows that the 4-year-old, SATA-based PS100E turns out the same IOPS as the recently released, top-of-the-line PS3900XV. 60,000 IOPS to cache. The chipset spec sheets from Broadcom indicate a dual core processor at about 1GHz with an architecture that can't support 10 gigabit Ethernet. I'm sure Broadcom has some exciting stuff; none of it seems to be in PS-arrays though. Can someone explain to me why you can team the Broadcom interfaces in a $3,000 server but you can't team the interfaces in a $65,000 SAN using Broadcom hardware?
At the end of the day, you're locked in to the "enterprise" storage hardware vendor's price for a disk drive, rather than the market's price. You're locked in to that storage vendor's hardware roadmap. You'll get 10 gigabit when they're ready, not when you're ready. You'll get RAID6 when they're ready, not when you need it.
These are my opinions and they are not necessarily shared by my employer, LeftHand Networks.
