LSI IR writecaching performance fix

[Smite]

Hi,

WARNING: Enabling WC without BBU will cause data corruption when system is shutoff incorrectly.

I have found quite a few posting regarding performance issues with ESXi and LSI IR. Probably due to the write caching being disabled by default on these controllers.

You can change this with LSIUtil (http://www.lsi.com/DistributionSystem/AssetDocument/support/downloads/hbas/fibre_channel/hardware_drivers/LSIUtil%20Kit_1.60.zip), but by default this does not work on ESXi because it's slightly different:

/tmp # ./lsiutil

LSI Logic MPT Configuration Utility, Version 1.60, July 11, 2008

sh: /sbin/modprobe: not found

sh: /bin/mknod: not found

Couldn't open /dev/mptctl or /dev/mpt2ctl!

0 MPT Ports found

/tmp #

I made the following change to the source of LSIUtil and then it works on my ESXi 4.0:

--- lsiutil/mpt2sas_ctl.h 2008-04-17 20:30:40.000000000 +0200

+++ lsiutil.fix/mpt2sas_ctl.h 2009-07-05 22:58:58.000000000 +0200

@@ -56,8 +56,8 @@

/**

• HACK - changeme (MPT_MINOR = 220 )

*/

-#define MPT2SAS_MINOR MPT_MINOR+1

-#define MPT2SAS_DEV_NAME "mpt2ctl"

#define MPT2SAS_MINOR MPT_MINOR2

+#define MPT2SAS_DEV_NAME "mptctl_sas"

#define MPT2_MAGIC_NUMBER 'm'

#define MPT2_IOCTL_DEFAULT_TIMEOUT (10) /* in seconds */

-

---

/tmp # ./lsiutil.new

LSI Logic MPT Configuration Utility, Version 1.60, July 11, 2008

sh: /sbin/modprobe: not found

sh: /bin/mknod: not found

1 MPT Port found

Port Name Chip Vendor/Type/Rev MPT Rev Firmware Rev IOC

1. /proc/mpt/ioc0 LSI Logic SAS1068E B3 105 011c0200 0

Select a device: 1

1. Identify firmware, BIOS, and/or FCode

2. Download firmware (update the FLASH)

4. Download/erase BIOS and/or FCode (update the FLASH)

8. Scan for devices

10. Change IOC settings (interrupt coalescing)

13. Change SAS IO Unit settings

16. Display attached devices

20. Diagnostics

21. RAID actions

22. Reset bus

23. Reset target

42. Display operating system names for devices

45. Concatenate SAS firmware and NVDATA files

59. Dump PCI config space

60. Show non-default settings

61. Restore default settings

66. Show SAS discovery errors

69. Show board manufacturing information

97. Reset SAS link, HARD RESET

98. Reset SAS link

99. Reset port

e Enable expert mode in menus

p Enable paged mode

w Enable logging

Main menu, select an option: 21

1. Show volumes

2. Show physical disks

3. Get volume state

4. Wait for volume resync to complete

23. Replace physical disk

26. Disable drive firmware update mode

27. Enable drive firmware update mode

30. Create volume

31. Delete volume

32. Change volume settings

33. Change volume name

50. Create hot spare

51. Delete hot spare

99. Reset port

e Enable expert mode in menus

p Enable paged mode

w Enable logging

RAID actions menu, select an option: 32

Volume 0 is Bus 0 Target 0, Type IM (Integrated Mirroring)

Volume 0 Settings: write caching disabled, auto configure

Volume 0 draws from Hot Spare Pools: 0

Enable write caching: yes

Offline on SMART data:

Auto configuration:

Priority resync:

Hot Spare Pools (bitmask of pool numbers):

RAID ACTION returned IOCLogInfo = 00010005

RAID actions menu, select an option: 0

Main menu, select an option: 0

-

---

I was doing a slow SCP while enabling WC and you can see my improvement in the attached screenshot.

Let me know if it works for you.

[ceemour]

Does this change survive a reboot. If so which file is being changed ?? any ideas

chris

[Smite]

Yes, I have just tested and it survives a reboot.

The change is probably being made in the NVRAM of the LSI RAID controller, not anywhere on the VMware ESXi filesystem.

[hzuerker]

I can't get the lsiutil to run under ESXi 4.0.

I applied the patch you provided (thanks!) and compiled it via

gcc -Wall -static -O lsiutil.c -o lsiutil

When I try to run it on the ESXI host I only get

LSI Logic MPT Configuration Utility, Version 1.60, July 11, 2008

Segmentation fault

Thx

Heiko

[Smite]

I compiled it on i386.

If I compile it on x86-64, I get the same error as you.

On which architecture do you compile it?

[hzuerker]

I did first compile it on x86_64, but I also tried i386. I also tried it stripped and not-stripped.

This is what I currently got:

file lsiutil

lsiutil: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.0, statically linked, stripped

Maybe it's the glibc version, unfortunately I don't have an older one available.

Would you mind posting your working file here?

Heiko

[Smite]

I'd rather not post binaries of this. I'm not sure about the license of this software.

However, I do not build it statically, I just:

zoiah@matryoshka:~/tmp/Sourcefix/lsiutil.fix$ make

gcc -Wall -O lsiutil.c -o lsiutil

cp -p lsiutil /usr/bin

cp: cannot create regular file `/usr/bin/lsiutil': Permission denied

make: *** Error 1

zoiah@matryoshka:~/tmp/Sourcefix/lsiutil.fix$ file lsiutil

lsiutil: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.8, not stripped

zoiah@matryoshka:~/tmp/Sourcefix/lsiutil.fix$

This is on a mostly vanilla Debian Lenny installation.

Please let me know if you can get it to work or not. If not, I'll see what I can do.

[hzuerker]

Okay I got it working.

It needs to be compiled for 32bit and as a dynamic executable. I was just thinking to much....

Thanks for the help!

Heiko

[Smite]

Can you confirm the same performance increase?

[hzuerker]

I'm copying my test VM over to the IR volume right now, it'll take a bit.

I'll post my findings as soon as I have them.

[hzuerker]

I'm seeing a HUGE difference in write performance!

The write speed is at least 2-3 times to what it was before.

I'm only testing in a Raid-1 right now, I hope I'll get a chance to try it on a Raid-1E this weekend.

Heiko

[DAPBENJAMIN]

Hi,

Could I get a copy of your binary please?

Regards,

Darren

[DAPBENJAMIN]

Hi,

Could I get a copy of your binary please?

Regards,

Darren

[alphenit]

I created a blogpost about this subject (You cut the power to our ESXi with VMFS running on RAID with WriteCache on

without a BBU What happens next?) which also hosts the ESXi4 LSI Util I compiled.

You can find it here:

http://www.virtualistic.nl/archives/526

Hope you guys enjoy it (please visit some of the sponsors to keep the site alive)

Please consider awarding points if my response was helpful

[J1mbo]

Great bit of work this.

Assuming the controllers in question have no battery or flash cache backup though I would be sure to be running ESXi build 208xxx since LSI controller caches are not properly flushed at shutdown on earlier builds. Also a machine in that case a UPS and a UPS shutdown mechanism would be required.

[alphenit]

Thanks..!!!

The sleepless night was actually caused by a remark I read on techhead.co.uk which inspired me to test it out. (remark probably made by you I guess )

I'm currently running VMware ESXi 4.0.0 build-219382 and am also working on getting the UPS to work with ESXi (using the excellent blogpost from techhead)

Aleks

Please consider awarding points if my response was helpful

[Scribe]

Hi guys,

I'm having similar performance issues with an LSI SAS2008 controller mpt2sas drivers. Is there such a utility for this device on ESXi 5.0.0?

Many thanks

[EDV-Schuster]

i would also be interested in a lsiutil that runs on esxi 5.0

can someone post a link to a working binary?

[pasmo]

no news for a esxi5 solution?

[tyauvil]

This utility works in ESXi 5.1. I just tested it and I went from 1-10MB/s a second to 60MB/s+ steady copying over my gigabit link.