At this point, I have concluded the following: 1) Good shared storage is very expensive compared to good local storage. 2) Good shared storage is much more prone to complex failures than local storage. Because live storage migration exists, I am thinking that I am going to dump shared storage in order to decrease complexity, increase uptime and *performance*. I added emphasis to that last one because my anecdotal experimentation hasn't been good to this end: my MegaRAID cards that perform very well under regular linux perform like absolute garbage under ESXi 5.x. Can someone help me here? Is there a good RAID card that can enable local datastore performance within ESXi? Are the LSI cards cursed? Do I need to sacrifice a goat or do some sort of song and dance for the EMC gods? Because of the HUGE pricing overhead on shared storage, I plan on running all SSD (the new Samsung 850s will do 150TBW, which is great for my purposes) for the local datastore. Any guidance would be appreciated.
For posterity, the solution was to build a SAN with Windows Server 2012 and export "Storage Spaces" over iSCSI. This is a very affordable way to serve high-performance shared storage because they do SSD caching. At this point, we're probably going to move whole-hog to Hyper-V, since VMware does not offer compelling storage solutions (I get it - EMC is the majority holder - no need to explain).
If you are going to use local storage, make sure you use a RAID controller with write-cache. ESXi doesn't do caching on the OS level, so it is required to have write-back mode (usually with flash/battery backed cache) enabled on the RAID controller. Without this RAID controllers operate in write-trough mode which is what's most likely happening in your case.
André
You might want to consider hyper-converged solutions like Nutanix.
For posterity, the solution was to build a SAN with Windows Server 2012 and export "Storage Spaces" over iSCSI. This is a very affordable way to serve high-performance shared storage because they do SSD caching. At this point, we're probably going to move whole-hog to Hyper-V, since VMware does not offer compelling storage solutions (I get it - EMC is the majority holder - no need to explain).
Oh, you must have missed VSAN then, its a very affordable storage solution using local disk.
Storage Spaces is a joke compared to VSAN.
// Linjo
How so? For less than $3000 in hardware and a Microsoft Server 2012 Standard license, I was able to get 9 terabytes of 10GbE iSCSI that I'm peaking at 45,000 write iops and 21,000 read iops. Please show me your recipe to do that with VMware vSAN. HINT: the first $2500 is spent on a vSAN license. EDIT: Make that $5000 for the vSAN license since I am serving two ESXi hosts with my Windows SAN.