<sigh> I may have finally found my performance issue. Unfortunately I cannot go through steps to put an older linux on the machine again to see if the kernel version is what is causing this.
BUT, with that said. It seems that under Hardy Heron, my cache is not enable for the controller.
http:// 163.044342 sd 0:2:0:0:
sda 1464074240 512-byte hardware sectors (749606 MB)
http:// 163.044388 sd 0:2:0:0:
sda Write Protect is off
http:// 163.044390 sd 0:2:0:0:
sda Mode Sense: 1f 00 10 08
http:// 163.044492 sd 0:2:0:0:
sda Write cache: disabled, read cache: disabled, supports DPO and FUA
http:// 163.044617 sd 0:2:0:0:
sda 1464074240 512-byte hardware sectors (749606 MB)
http:// 163.044662 sd 0:2:0:0:
sda Write Protect is off
http:// 163.044665 sd 0:2:0:0:
sda Mode Sense: 1f 00 10 08
http:// 163.044744 sd 0:2:0:0:
sda Write cache: disabled, read cache: disabled, supports DPO and FUA
http:// 163.044749 sda: sda1 sda2 <<6>ata1.00: ATAPI: TSSTcorp CD-RW/DVD-ROM TS-H492C, DE02, max UDMA/33
http:// 163.067845 sda5 >
http:// 163.067958 sd 0:2:0:0:
sda Attached SCSI disk
http:// 163.068118 sd 0:2:1:0:
sdb 1464074240 512-byte hardware sectors (749606 MB)
http:// 163.068160 sd 0:2:1:0:
sdb Write Protect is off
http:// 163.068163 sd 0:2:1:0:
sdb Mode Sense: 1f 00 10 08
http:// 163.068240 sd 0:2:1:0:
sdb Write cache: disabled, read cache: disabled, supports DPO and FUA
http:// 163.068377 sd 0:2:1:0:
sdb 1464074240 512-byte hardware sectors (749606 MB)
http:// 163.068419 sd 0:2:1:0:
sdb Write Protect is off
http:// 163.068422 sd 0:2:1:0:
sdb Mode Sense: 1f 00 10 08
http:// 163.068518 sd 0:2:1:0:
sdb Write cache: disabled, read cache: disabled, supports DPO and FUA
http:// 163.068523 sdb: sdb1
http:// 163.068713 sd 0:2:1:0:
sdb Attached SCSI disk
http:// 163.068804 sd 0:2:2:0:
sdc Very big device. Trying to use READ CAPACITY(16).
http:// 163.068843 sd 0:2:2:0:
sdc 4392222720 512-byte hardware sectors (2248818 MB)
http:// 163.068897 sd 0:2:2:0:
sdc Write Protect is off
http:// 163.068900 sd 0:2:2:0:
sdc Mode Sense: 1f 00 10 08
http:// 163.068991 sd 0:2:2:0:
sdc Write cache: disabled, read cache: disabled, supports DPO and FUA
http:// 163.069095 sd 0:2:2:0:
sdc Very big device. Trying to use READ CAPACITY(16).
http:// 163.069149 sd 0:2:2:0:
sdc 4392222720 512-byte hardware sectors (2248818 MB)
http:// 163.069217 sd 0:2:2:0:
sdc Write Protect is off
http:// 163.069220 sd 0:2:2:0:
sdc Mode Sense: 1f 00 10 08
http:// 163.069306 sd 0:2:2:0:
sdc Write cache: disabled, read cache: disabled, supports DPO and FUA
http:// 163.069311 sdc: sdc1
It is a PERC 5/i from Dell/LSI. We opted not to go with the battery backed cache because it is on a 2kva UPS which is attached to a standby generator.
I CANNOT get the cache enabled. It seems I need a program called megacli which is only provided in 32bit, not 64bit. I can't find any resources for a ready to use 64bit version or instructions on how to run the 32 bit one on my 64bit Hardy. If I ever get the cache enabled I'll report back if it fixed my issue.