How to tweak the 3ware 9650SE SATA RAID controller driver into ESX3i

[gi-minni]

IMPORTANT: First of all please consider that following my instructions below to embed a 3rd-party driver

into ESX3i is and remain UNSUPPORTED forever. You will not get any help from the vmware stuff personnel

until the below mentioned driver will be officially integrated as an update into the main upstream distribution.

Be also aware that integrating 3rd party driver in this way could seriously damage or alter any data stored

in your datastore, if the driver has not passed VMware's strong quality regression,NFR, harness tests before.

Please consider also that using the oem.tgz file can be somedays overwritten without notice.

You are alone, be careful on that.

But I want give back something to this lovely community, giving me a lot of information and even VMware for

ESX3i a small-footprint OS that can run 24x7 from a USB-drive having a rock solid filesystem VMFS and

beeing a small subset of the industry proven ESX3. This will be the future of the next generation of computing

platform.

Embedding ESX3i into the BIOS of any server, laptop, desktop and workstation and you get "virtualization at

your fingertips"

Making things easier to maintain and handle even portable across systems, easier to backup and distribute.

Easier for the developer for buiding specialized appliances without the need using a full blown operating

system with all the security concerns in it.

I brought two months ago a 3ware 9650SE SATA-RAID Controller after reading in the 3ware forum that the

newest 3ware driver can run under ESX3i.

(See:[http://www.3ware.com/KB/article.aspx?id=14922])

Where can I find the ESX3i driver?

• Please go to the 3ware download support website ()

• Select the 3ware 9690SA Series software product

• Select the item "Complete 9.5.0.2 CD for VMware ISO" (9.5.0.2-Codeset-Complete.iso checksum: a2939603988521aa6b49d78597a0b39a)

• Burn the iso file or use Winimage to open the ISO without burning (my favorite way:-)* )

• Go to VMupdates/RPMS

• Extract VMware-esx-drivers-scsi-3w-9xxx-2.24.08.014vm-00000.i386.rpm into a temp directory

• From the temp directory use 7-Zip to open the rpm file in a Windows environment

• Double click the rpm file, you will see VMware-esx-drivers-scsi-3w-9xxx-2.24.08.014vm-00000.i386.cpio.gz

• Double click on the gz file, you will see VMware-esx-drivers-scsi-3w-9xxx-2.24.08.014vm-00000.i386.cpio

• Double click on the cpio file, you will see a dot folder

• Double click on the dot folder

• Goto /usr/lib/vmware/vmkmod

• extract the 3w_9xxx.o driver

• upload this driver to the ESXi box into the /mod directory

• check /sbin/vmkload_mod /mod/3w_9xxx.o

BTW here is my calculated MD5 checksum "cb3e6d0cc53b61833d3ba7194f9631fa 3w_9xxx.o"

But how do I enable this driver inside ESX3i?

I read a little bit around and downloaded the newest driver from 3ware, extracted the 3w_9xxx.o driver and

uploaded it with scp into my esx3i server into the /mod directory. Afterward from the unsupported shell I loaded

manually the driver into the ESX3i by executing /sbin/vmkload_mod /mod/3w_9xxx.o. I was very happy to

format the RAID5 with VMFS and see my RAID5 array up and running every time after a rescan while rebooting

the partition.

Testing it a while I could not find any show stopper or strange behavior, so I considered to add this driver into

the system.

But how the hell can I automate this stuff?

First of all is essentially to know the PCI-ID of the mentioned card after loading the driver manually into the

running kernel with /sbin/vmkload_mod /mod/3w_9xxx.o and executing lspci -p from the unsupported shell.

The output of lspci was ... 13c1:1004 13c1:1004 ... wonderful.

What next?

OK I discovered that /etc/vmware/simple.map

has a list of all supported devices with vendor-id/device-id sub-vendor-id/sub-device-id. So I inserted into

/etc/vmware/simple.map 13c1:1004 0000:0000 storage 3w_9xxx and saved the file

The little thing to know now is the way ESX3i loads this driver automatically. Looking inside

/vmfs/volumes/Hypervisor1 there are all the tgz compressed stuff to load after

booting the kernel.

I tried to be not invasive and exchanged the empty oem.tgz file with my generated oem.tgz file.

The name sounds reasonable to use it for this tweak!

Here are the steps:

- Press Alt-F1 on the DCUI-console and enter -> unsupported <- and the root password if set

- cd /tmp

- make mkdir -p /mod

- make mkdir -p /etc/vmware

- make mkdir -p /usr/share/hwdata

- cd /tmp/mod

- get the driver from another ssh server with scp user@host:/tmp/3w_9xxx.o .

- cd /tmp/etc/vmware

- copy simple.map from /etc/vmware in this directory

- vi simple.map

- insert 13c1:1003 0000:0000 storage 3w_9xxx for the PCI-X Card before the line

beginning with 14e4:1600

- insert 13c1:1004 0000:0000 storage 3w_9xxx for the PCIe Card

- save and quit (:wq)

- cd /tmp/usr/share/hwdata

- copy pci.ids from /usr/share/hwdata in this directory

- vi pci.ids

- search for 13c1 3ware

- insert the following lines

- 1003 3ware 9650SE-series SATA-RAID Controller (PCI-X)

- 1004 3ware 9650SE-series SATA-RAID Controller (PCIe)

- save and quit (:wq)

- cd /tmp

- chown -R 201:201 ./mod

- chown -R 201:201 ./etc

- chown -R 201:201 ./usr

- chmod -R 755 ./mod

- chmod -R 755 ./etc

- chmod -R 755 ./usr

- chmod 644 ./etc/vmware/simple.map

- chmod 644 ./usr/share/hwdata/pci.ids

- chmod 755 ./mod/3w_9xxx.o

- tar -cvzf oem.tgz etc mod usr

Now you have tweaked the oem.tgz file with the 3ware driver in it and simple.map.

Copy this file into /vmfs/volumes/Hypervisor1 and chmod 755 oem.tgz and

chown root:root oem.tgz it. Repeat the above step also for /vmfs/volumes/Hypervisor2

Cross your fingers and reboot. If every thing is in place the 3ware driver will

be loaded automatically.

My questions now to the vmware guys are:

1) At what time can I expect this driver in the upstream esx3i bundle?

2) The 3ware controller type is unknown in VIClient -> Configuration -> Storage

Adapters. Where can I change it?

3) Is there a chance to see an ssh server for maintenance purposes in the future?

Enjoy it and tell me if this advice was useful, please consider to give me some points/credits if

this topic helped you in one way or another.

Message was edited by: gi-minni (updated with driver download, pci.ids and PCI-X information)

[Dave_Mishchenko]

2) The 3ware controller type is unknown in VIClient -> Configuration -> Storage Adapters. Where can I change it?

Edit /usr/share/hwdata/pci.ids and add it to your oem.tgz file.

3) Is there a chance to see an ssh server for maintenance purposes in the future?

I would doubt it - but you can enable SSH to your ESXi host once you have the install done. Edit the file /etc/inetd.conf and remove the # from before SSH and then reboot.

1. These are standard services.

#

1. ftp stream tcp nowait root /usr/sbin/tcpd in.ftpd

• ssh stream tcp nowait root /sbin/dropbearmulti dropbear ++min=0

#telnet stream tcp nowait root /bin/busybox telnetd

#nntp stream tcp nowait root tcpd in.nntpd

#smtp stream tcp nowait root tcpd sendmail -v

[gi-minni]

Thanks Dave,

2) ok, but I recognized the information is there, my pci.ids has the 3ware tags in it. I guess pci.ids will

be built after the boot process and therefore the info could not be shown in the VIC GUI.

3) SSH is implemented via dropbear. I tried to tunnel ports, but

could not gain any control on it. Every time PuTTY ends with an error.

How can I tunnel ports with the dropbear (SSH) server and use PuTTY 0.60 on the other side?

Message was edited by: gi-minni (regarding the pci.ids issue)

[rkelleyrtp]

I just wanted to share my experience with this. I tried the same steps as gi-minni but was unable to get the module loaded properly for either ESX3i or ESX 3.5 on my server (using the 9650SE-4LPML card).

On my first try, I installed the latest version of ESX3i (VMware-ESXi_3.5_u2-103909.i386.iso) on a thumb drive, booted the server, and got into the unsupported shell. After some digging around, I found a 3w-9xxx.o module on 3ware's site (had to d/l the 9690SA VMWare driver disc), copied this file to /mod/3w_9xxx.o, then ran the "/sbin/vmkload_mod /mod/3w_9xxx.o" command. Unfortunately, this module was compiled with a different kernel that ESX3i was running, so the kernel did not install (I got a number of driver/kernel mismatch errors). I also tried various other 3w-9xxx.o files from 3Ware (specifically for the 9650SE card) and they all gave the same sort of issues. Unfortunately, ESX3i does not contain a full install of Linux, so I could not compile the module source code while booted in EX3i (no "make" utility or kernel source code).

On my second try, I installed the full-blown ESX 3.5 software on my server (on a HDD using the onboard SATA controller), downloaded and extracted the kernel source (from the same iso image file), and ran the "make" command. To my suprise, the module built without errors and I was able to run "insmod 3w-9xxx.o". A quick "lsmod" did show the module listed, so everything was looking good. I copied the new kernel module to the proper location and created a new initrd file (again, no errors). Unfortunately, after rebooting the server, the driver kept spitting out messages about LUN timeouts and reset the card for each timeout occurance. It did this about 15 times (Lun0...Lun15) before I shut the server off. While the module was vaild for this kernel release, the LUN timeout issue was a show-stopper.

I was curious if someone has compiled a working module for the latest version of ESX3i (Update 2 - released July 31, 2008). If not, I am afraid we need to wait for 3Ware to release their official VMWare driver set for the 9650SE cards...

[gi-minni]

Believe me it works even with the latest version (ESX3i-u2) you need only the correct driver to fit into the vmkernel

But it is a hard way to get this driver out, I admit the description was not accurate enough, I apologize for this.

Now I re- and double checked every thing. Et voilá here comes the truth

• Please go to the 3ware download support website ()

• Select the 3ware 9690SA Series software product

• Select the item "Complete 9.5.0.2 CD for VMware ISO" (9.5.0.2-Codeset-Complete.iso checksum: a2939603988521aa6b49d78597a0b39a)

• Burn the iso file or use Winimage to open the ISO without burning (my favorite way:-)* )

• Go to VMupdates/RPMS

• Extract VMware-esx-drivers-scsi-3w-9xxx-2.24.08.014vm-00000.i386.rpm into a temp directory

• From the temp directory use 7-Zip to open the rpm file in a Windows environment

• Double click the rpm file, you will see VMware-esx-drivers-scsi-3w-9xxx-2.24.08.014vm-00000.i386.cpio.gz

• Double click on the gz file, you will see VMware-esx-drivers-scsi-3w-9xxx-2.24.08.014vm-00000.i386.cpio

• Double click on the cpio file, you will see a dot folder

• Double click on the dot folder

• Goto /usr/lib/vmware/vmkmod

• extract the 3w_9xxx.o driver

• upload this driver to the ESXi box into the /mod directory

• check /sbin/vmkload_mod /mod/3w_9xxx.o

If this works for you, read carefully my post how to create an oem.tgz file and feel free to

give me some credits by selecting Yes, my question has been answered. Thanks.

BTW here is my calculated MD5 checksum "cb3e6d0cc53b61833d3ba7194f9631fa 3w_9xxx.o"

If you wish I can send you my oem.tgz file supporting 9650 (PCIe & PCI-X) bus interfaces

[2by4]

Hello, all. I'm new to ESXi.

I tried the instructions above and I am definitely able to get my 9650 to be seen by ESXi, beautiful! Thanks.

One question though: I am able to do this for an existing install, can this be done on a fresh install?

Meaning, I'd like to install the ESXi image onto a disk which is attached to the 9650 itself, is there a

way to do this too? I also thought about isstalling ESXi to a USB pen drive, is there a large performace issuse

there?

[rkelleyrtp]

Hi gi-minni,

Your new instructions worked perfectly! Many thanks for updating them. It turns out, I was indeed using the wrong driver from 3Ware's web site. Right now, I have a new server running from a thumb drive using the 3Ware driver on a set of RAID-10 Seagate 1TB drives. The only issue I am having now is slow network performance. But, your updated instructions worked very well.

Many thanks again!

-Ron

[gi-minni]

Hi 2by4,

if you do a fresh install you need only to exchange the empty oem.tgz file with the one you created in an earlier stage.

It means make a copy of oem.tgz and overwrite the original oem.tgz in /vmfs/volumes/Hypervisor1 and /vmfs/volumes/Hypervisor2, dont forget

to set the correct owner,group and flags. To upload the oem.tgz file you can use the scp program inside the ESX3i box or better you enable the

ssh server (dropbear) and do an scp upload from outside into the ESX3i box. I prefer to use the embedded ssh-server, because I gain the ability to execute local commands like vmkfstools for cloning from another VM.

I am running for weeks now an USB pendrive based ESX3i installation and everything is running smoothly.

The instruction how to build a USB pendrive based installation you can grab at:

http://communities.vmware.com/message/825686#825686

The above instruction works also with the latest update.

Please consider to flag this topic as answered if you like.

[matthewk]

Any way to get the tw_cli utility working? It tells me that I do not have a controller installed.

[rkelleyrtp]

Hi gi-minni,

Looks like I may be having some sort of disk I/O issue using the 3Ware controller (9650Se-4LPML) on my ESX3i install (using the 3w_9xxxx.o file from the 9650SE VMWare CD). I have two servers running these cards and they both seem to get only 13-15MB/sec thru-put. I was hoping you could test your server(s) to see if your disk I/O is similar. From the unsupported shell, can you please see how long it takes to create a 200MB junk file? On my server, it takes approximately 17sec (11.7MB/sec) to create this file. I am using this command:

shell# date ; dd if=/dev/zero of=file bs=1024 count=200000 ; date

Mon Aug 11 22:34:23 UTC 2008

200000+0 records in

200000+0 records out

Mon Aug 11 22:34:35 UTC 2008

I have noticed my network thru-put is very slow as well (max 10MB/sec) but that may have something to do with the low disk I/O. Also, here is a dump of /proc/scsi/3w_9xxx/6:

/vmfs/volumes/48a02f9c-974df040-5bb2-00e0814cab6f # cat /proc/scsi/3w-9xxx/6

scsi6: 3ware 9000 Storage Controller

Driver version: 2.24.08.014vm

Current commands posted: 0

Max commands posted: 9

Current pending commands: 0

Max pending commands: 0

Last sgl length: 0

Max sgl length: 32

Last sector count: 1

Max sector count: 256

SCSI Host Resets: 0

AEN's: 2

vmkadapter: vmhba1

Thanks for any info you have.

[matthewk]

Took 53 seconds, although that was to a single 250gb Seagate hooked up to the 3ware controller. The ESX SC is notoriously slow on I/O and I would assume the ESXi unsupported shell is the same way.

I checked ATTO on an XP VM and was getting good performance for reads, but writes were less than 10mb/sec. Writes over the network was much worse at 500k/sec, although reads were at 10mb/sec -typical for my 100base-T network up here.

[rkelleyrtp]

Thanks for the info. I just tested the server again using an NFS-mounted datastore and the VM performance was much better. This leads me to believe the 3Ware controller is causing the slow-down and not the NIC.

I opened a support ticket with the 3Ware folks and they just asked me to verify if the Disk Cache and Performance options were set in the BIOS. On my servers, both options were set properly (max performance). They also asked me to install the 3DM2 software on the ESX server. Unfortunately, I can't as ESXi does not have a full-blown Linux install.

If you have an NFS server available, can you try a similar test? Simply mount an NFS server as a datastore, and create a VM using that datastore. Perform some simple tests (ATTO, etc) to see what you get. This should give you a rough indication if the 3Ware controller is causing your slow-down as well...

[matthewk]

I replaced the 250gb with a pair of 80gb Hitachi's in RAID 0. 29mb/sec. Better.

~ # cd /vmfs/volumes/storage

/vmfs/volumes/48a1b8bc-44ea913c-d31b-0019d1f5ba1c # date ; dd if=/dev/zero of=file bs=1024 count=200000 ; date

Fri Aug 1 20:03:44 UTC 2008

200000+0 records in

200000+0 records out

Fri Aug 1 20:03:51 UTC 2008

The ATTO utility under the Windows XP VM over the network gave a more respectable 10mb read/6mb write, although I have to wonder if those numbers would go up if I had a gigabit port up here.

[gi-minni]

Hi rkelleyrtp,

I am back after the black thuesday, indeed I can confirm that the performance is slow using

internal SATA storage. We need to move out this discussion to a separate thread. I was wondering

how could I get only 100KiB/s download 170KiB/s upload speed over the Internet. Had no time to

look deeper into it. But now is time to look what is causing this bottleneck.

First of all we must normalize our test. It means we have different disks, motherboards, bus

interfaces, NIC's, procs-, mem speed and raid level. We need common tools to check net,IO,cpu,bandwith

I will come back with numbers if I have time to measure and look inside the MB and 3ware BIOS Level

Is there anyone other having this slow I/O out of there?

Is there a moderator (Dave?) that can move out our discussion to another independend thread?

[rkelleyrtp]

Hi gi-minni,

Thanks for confirming the testing results. My suspicion is the 3ware controller has put the drives in "safe" mode because:

• I don't have a BBU installed

• None of the 3ware utilities (tw_cli and 3dm2) are able to recognize the controller using ESX3i (thus, the driver is unable adjust the performance "bit" on the card).

I have tweaked every setting in the 3Ware BIOS (performance, caching, queuing, etc) and nothing has made a difference. Also, I reconfigured my card to provide pass-thru drives (no RAID) and got the same performance figures as before. At this point I can only assume (bad word) the 3ware driver can recoginze the card but not enable any performance features.

[Paul_Lalonde]

Hey guys.

Typically, the ESX VMkernel SCSI / RAID modules disable write-back caching on a controller if no BBWC is found. I will tinker around with the 3Ware driver and see what I can find.

Regards,

Paul

[rkelleyrtp]

Thanks Paul! If you need someone to test your new driver, just let me know.

-Ron

[rkelleyrtp]

Just started a new thread to track the slow disk I/O issues:

http://communities.vmware.com/thread/163076