Adaptec 6405 poor write performance

[geforce20111014]

Hi,

My System specs are as follows:

CPUs: 2x Intel Xeon E5620

Motherboard: Intel S5520HC
Ram: 12x Kingston KVR1066D3D4R7S/4GI for a total of 48GB
USB: 4GB Kingston DataTraveler for booting ESXi 4.1
RAID Controller: Adaptec 6405 (firmware build 18301) with AFM-600 NAND FLASH MEMORY BACKUP for 6 SERIES
HDD: 4x Seagate Constellation ES SAS 1TB ST31000424SS (firmware 0006)

After suffering poor write performance the first time I went ahead and produced a fresh install of ESXi 4.1 Update 1 installed to the usb stick listed above.

I then installed the VMware ESX/ESXi 4.1 Driver CD for PMC Sierra aacraid from http://downloads.vmware.com/d/details/dt_esx41_pmc_aacraid_11728000/ZCV0YnR0aipiZColcA==

After installing and restarting ESXi I could then see the raid array and create a datastore.

Currently the raid is configured as a RAID 10 I have tried RAID 6 with the same poor write performance. Write Cache is forced on and ZMM is at optimal charge!

I have been testing performance with Crystal Disk Mark running in a Windows 2008 R2 x64 VM.

Sequential Read is 286MB/s and Sequential Write is 28MB/s.

On another system which is an i7 950 with Adaptec 5805 with BBU running ESXi 4.1 with the same hard drives but configured in a RAID6.

Sequential Read is 246MB/s and Sequential Write is 207MB/s.

I would like to get the 6405 writing atleast 5x faster than its currently operating instead of sending it off as ewaste!

Thank you in advance for any assistance you can offer me.

Update: Tried Paravirtual SCSI Controller and a vdisk on the Windows vm for a bit of fun. The test results are 325MB/s Sequential Read and 28MB/s Sequential Write.

[amstim]

Indeed and its not just 32megs.  It's 32Megs times the number of drives in the array being used for the VM or set of VMs, etc.  Also the risk goes up significantly with higher utilization of the drives or arrays.  I may get away with it due to the lazy nature of most of our application servers and that they are not VMed.

[znebzuga]

I'm using 3ware 9650 and 9690 raid controller. When Write back cache is enable with BBU, the drive cache is also protected by the BBU too and there is no option to disable/enable cache drive level... Simply enable/disable the both caches.

I'm interested on a 6405...

[amstim]

Update.......boo.

The 6405 write speed is performing poorly again, no amount of toggling cache setting is restore proper speed.....

I just switched my pending order for two 6805s for two 5805Zs.  I don't need this kind of aggravation on my ESXi boxes...

znebzuga:

I've never heard of BBU powering drive cache before....sounds fishy! :smileymischief:

[znebzuga]

Yeah but 3ware's do this. However my 3ware's are slowly for SSD's.

I ordered a 6405 + ZMCP module. I think it will be ok with cache drive OFF (since my SSD's don't have cache) and turnn on the controller cache...

Will bo ok ,don't you?

[amstim]

Normally I would say yes, but now my test machine running 2008 server with no VMware or ESX whatsoever is having trouble getting more the 28 MBs.....

Right now I am hoping desperately that its just Crystal Disk mark that is messed up with the Adaptec 6 series.  Installing more test tools now.  I will update when I know more.

[znebzuga]

Oh man...you are worring me... no matter what setting you use , can't do more than 28mb/s sequencial write? Even with cache drive off and controller cache on?

[znebzuga]

Try SISandra software and Winsat disk...

[geforce20111014]

Before I posted on here I did have the one occassion where the 6405 did perform without the write lock in esxi but I have never seen it happen more than once.

I have benched the performance with a freebsd vm with dd and that showed similar to seq write in crystal.

I havent herd anymore from Adaptec yet.

[znebzuga]

Geforce:

Tell me a thing. Did you perform controller in sequencial write below and above 512mb of data? Did it show the same 28mb/s? I mean, on Crystal disk mark without VM or ESXI, just on windows...

[amstim]

I went ahead and deployed the 6405 in the customer server.  It was performing inconsistently, but it was "not that bad".  I expect Adaptec will be coming out with an update shortly.

I also have reworked my ESXi setup and am using a 6805 with 8x 300 Gig SAS Cheetah 15.7.   I never encountered the 28MB/s issue but performance was a tad erratic, but always acceptable.

RAW Benches from a 2008 Machine, NO VM.  RAID6, 256k stripe. Onboard HDD Cache OFF

In a 2003 VM, RAID 6, 256k stripe.  This is with 11 other VMs running. Onboard HDD Cache OFF

Below is a test with a 50meg data set.  Same VM as above same RAID 6, 256k stripe. Onboard HDD Cache OFF. This reflects the nice speed boosts you get from the 512 meg controller cache.

In short to sum up.  I think the 6 series cards need a firmware update, they do perform a bit wacky.  Choosing to deploy them or not ATM will depend on your specific needs.  I think once the bugs are worked out the 6 series is going to be a very good card.  The 512 megs of cache is really nice.

[znebzuga]

Hello:

The test of 50Mb file is very strange. All 4k random on this test shows the cache speed but Sequencial's not. 50 and 100mb file lenght should show numbers around of 2000mb/s (cache speed). Yes, definetly Adaptech has to build a new firmware as soon as possible.

1000mb test shows as expected I think... but 500mb looks strange...

[amstim]

Actually I think the 500 meg test is just fine.  Remember that with the other VMs running the controller cache is not dedicated to the one VM running the tests.

When running on the RAW 2008 machine tests of 500 megs were registering about 1200 MB/s as expected.  Never saw 2000 MB/s on any tests.

As I said though, it needs a firmware update for sure.

[znebzuga]

I would say 2000mb/s in seqeuncial read at 50 and 100mb test file.

At 500mb of course is nearly out of cache and not show this result.

Although, 1200mb/s on 500mb test is very good

Here is my 3ware 9690SA-8i 512mb DDR2 533 cache + BBU with 2x Corsair F120 SSD.

50 and 100mb test shows the cache in every nature of test.

1000mb test shows the weakness of this controller, doesn't scale so well since each SSD does +/- 250mb/s read and write, so less than 400mb/s seq is bad, and even worse is the seq write, 2x in raid 0 shows the half speed of ONE drive...I know, 3ware 9600 series was not made for SSD's.

And as you can see, 4k random is pratically nothing comparing to the new controllers like an Adaptec 6805...

[znebzuga]

Amstim:

I purchased my 6405 + ZMM today.

I've been looking for BIOS of controller and I need your help please...

1- Where is Read Ahead Cache that you talked when you found the magic formula for unlock 28mb/s...I just see Enable/disable Read only.

I let most setting on default:

HDD drive cache in Enable All

Controller cache in Enable with ZMM

Performance on dynamic

Background task in High

This is right for the optimal  performance?

Regards

[amstim]

The read cache should be a property of the array just like write cache.  I believe the setting is right above or below the setting for write cache.  I don't have a server here that I can check on because they are all in production.

As I said I am my other post I don't think there is a "magic formula".  I concluded that the card really needs a firmware update, and that the only reason that I ended up deploying it was that at least with my 8 drive setup I never saw performance issues that were bad enough to halt my project.  That being said the performance numbers did vary from test to test more then I thought they should and I expect this will be sorted out with firmware patches.

[znebzuga]

Hello

Well I'm playing with my Adaptec 6405, I'm satisfied with performance.

Yes, it needs new firmware, especially to make a better job on cache support for 4k linear read.

4k random I think it's fine. This card is speedy thanks to powerfull dual core cpu. It's away faster than my old 3ware 9690...

4GB SLC NAND is a must not even on power failure but in a global system performance gain too.

[znebzuga]

Hello:)

Good news! The tech support guy of Adaptec told me that we have a new release/firmware for 6405/6805 up to 2 - 4 weeks:)

Let's see if this unlocks the potential of this cards. Stay tune

[znebzuga]

New firmware build 18668 is out.

Performance seems to be the same, but there is stability enhancements.

New performance mode - big block bypass. Uses all the cache for small blocks reads.