very slow sequential write using SAS 6/iR (LSI 1068E) and 15k rpm SAS drives on ESXi, run fast on a native installation of Win 2008

[mephistopoa]

Hi gents,

I'm having an issue on a Dell PowerEdge 2950 (quad core 2.66 ghz, 18GB ram) with my SAS drives having a very low write speed. The tests were ran inside a Windows XP VM using thick virtual disk of 32 GB. I've tried a single seagate 15k RPM 500GB and got 26MB sec, 2 of them in raid-0 and still got 26MB sec, then I tried a different SAS 2.5 inches 10k rpm from HP, and got again the same 26MB sec.

This controller is a hardware raid, not the best around but it is good for raid 1 and 0. So I've taken off ESXi from the machine and installed Windows 2008 baremetal. The drive now writes at 100MB sec sequentially on single drive and using software raid 0 does 145MB sec. I'm going to set up hardware raid 0 tomorrow on Windows 2008 to check if the write gets a bit better.

But anyway, there seems to be a huge difference between a VM in ESXi and a native machine running on these harddisks. I'm going to install VMware workstation on top of Windows 2008 and run again some tests with a thick virtual disk.

It is very strange that all disks are capped at 26MB sec. Also the read speed is very low in raid 0. I get on Windows 2008 340MB sec, on ESXi I get 200.

The controller doesn't have write cache so far I now as there is no battery on board. Can Windows 2008 use write cache on the harddisks without any sort of configuration needed from me? But even with or without write cache, is there so much overhead to drop a sequential read from 340 to 200 mb on a quad core xeon with 12mb cache and SAS controller?

Thank you!

[DSTAVERT]

The forums are littered with performance issues on controllers without write back caching. If you want any performance you will need a battery module and enable write back caching.




Forum Upgrade Notice - We will be upgrading VMware Communities systems between 10-12 December 2010. During this time, the system will be placed in READ-ONLY mode.

[mephistopoa]

Hum, I just confirmed on Windows 2008 that I was not using write cache to get 100MB write speed on a single drive. With write cache enabled it got 140MB, and in raid-0 it got 340MB sec of sequential write speed.

But even without write cache controller, why ESXi has such a low performance? Considering Windows 2008 natively reach 100MB easily on sequential write on these drives without write cache, I was expecting ESXi to get at least 80MB, but 26MB is really a pain.

Do you think there is a way to resolve this without changing the controller? This is not a option for me at the moment.

Thanks

[DSTAVERT]

I doubt there is anything you can do. The virtual requirements are much different than a standalone OS. ESX(i) is providing shared IO access to storage. Without the ability to cache reads and writes it must wait until the operation is complete before moving on.




Forum Upgrade Notice - We will be upgrading VMware Communities systems between 10-12 December 2010. During this time, the system will be placed in READ-ONLY mode.

[mephistopoa]

And in your opinion using a good perc controller with battery and write cache enabled, for example if on a Win 2008 i'm getting 140MB sec on sequential writes do you believe I can get close to this on a VM using writing to the same parts of the disk?

I'm not aware of the overhead on the disk system yet, but so far disconsidering the writes, the drop is large, sequential reads drops from 140 to 90-100.

what do you think about the storage overhead on ESXi ? Is it that much?

[DSTAVERT · Accepted solution]

You can't directly compare a standalone OS and a Virtual one. There is more than just the write caching but it can make a 10 fold difference in some environments (depending on controller, cache size, number of spindles etc.)

If you search the forums you will find many many similar posts and invariably the answer is battery module and caching.

http://communities.vmware.com/docs/DOC-5490

Forum Upgrade Notice - We will be upgrading VMware Communities systems between 10-12 December 2010. During this time, the system will be placed in READ-ONLY mode.

[mephistopoa]

thank you for this link, very useful information guide.

I appreciate your time reading my questions.

I think I'll need to spend more money on a decent perc controller...

[DSTAVERT]

Glad it helped. It doesn't look lake the card supports the BBU. There are places like ebay.

Forum Upgrade Notice - We will be upgrading VMware Communities systems between 10-12 December 2010. During this time, the system will be placed in READ-ONLY mode.

[mephistopoa]

Just one more thing, I was thinking about a single drive on the controller and it is really strange with write cash disabled on both windows 2008 and esxi, but I still get a huge difference in write speed between them.

Why a single drive is capped at 26MB sec on ESXi? Are the drivers so different to enable windows to perform 3x faster using the same conditions?

I'm afraid that I will get a proper raid controller and in the end find that ESXi can't really do much better even with write cache enabled.

What do you think?