Poor storage I/O performance with ESX 3.0.1

[jmmarton]

I'm trying to iron out some issues I've noticed with my VI3 implementation, and I've got SR's open with VMware for the local storage performance in ESX and with Novell for iSCSI performance. Right now I'm concerned about the local storage performance as the result of my VMware SR was that this was "normal." That doesn't sound right, so I figured I'd post here.

Here are some R/W performance stats using a 3.4GB DVD ISO copied back & forth via SCP:

test server, dual 933MHz CPUs, 100Mbps LAN connection, local storage,

Ultra3 10K RPM drive, JBOD: read 6159.6KB/s, write 5794.7 KB/s

ESX server, dual quad-core 2.33GHz CPUs, 2GBps LAN connection (2 x 1Gb),

local stroage, SAS 10K RPM drive, RAID1: read 4467.4KB/s, write 5978.4KB/s

ESX server again, 1Gbps iSCSI, storage server is 5 SATA II drives, 7200

RPM, RAID5 (no online spare): read 4350.8KB/s, write 2482KB/s

So as you can see my iSCSI write performance is absolutely horrid, and I've got an SR open with Novell on that one (using SLES10 on the storage server running iscsitarget). But I'm still concerned about the read performance of around 4500KB/s when using local storage, and in fact even the write performance of 6000KB/s seems slow.

[BenConrad]

Are you using a hardware RAID controller, if so you -need- Battery Backed Write Cache (BBWC) enabled so you can enable write-through caching. You should at least triple the write performance.

Also be aware that using SCP is not a good test of disk throughput, SSH uses compression & CPU so you don't have the best test case.

Also, copying into VMFS3 is not a good test of a disk subsystem. Get a file(s) onto a VM on the VMFS3 filesystem and see how fast you can copy.

Ben

[jmmarton]

Yes, this is an array controller with BBWC option. It's an HP Proliant DL380 G5 "high performance" model. It has two quad-core Intel Xeon 5345 (2.33GHz) CPUs, 12GB RAM, and a Smart Array P400 controller with BBWC & 512MB RAM. I'm copying into local storage, which is VMFS. I know that SCP will have some overhead, and I was told by VMware tech support today that VMFS will slow things down as well. But even with all that, considering my hardware, does 4500KB/s reading from ESX and 6000KB/s writing to ESX seem kinda slow if my workstation has a 1Gbps connection to the LAN and ESX service console has 2Gbps (this is also shared by vmkernel)? FYI, the the virtual switch for the service console has its load balance configured for IP hash, the Cisco switch has etherchannel mode set to on in the channel-group, and the switch is using dst-src-ip for th etherchannel load balance method.

I'm just trying to double-check everything to make sure the performance I'm seeing is correct, or if I have something configured incorrectly.

[RParker]

I would like to point out the bottle neck in this scenario is \*NOT* iSCSI, its SATA. Its good for storage, but cannot come CLOSE to SAS drives.

You need to use SAS as local storage for performance, this is where your problem lies, not in 1Gb connection, not iSCSI, its your drives.

Why do you have 1 Dual Core machine with SAS, and another with SATA II, trying to go the cheap route?

Just FYI, you get what you pay for. You want good performance, upgrade to SAS, period.

[jmmarton]

Ignore that config with the SATA drives and iSCSI. That's something else I'm working on, and I actually didn't mean to paste that in. The local storage on the server is via two 10k RPM SAS drives in a RAID1 config via the P400 controller. That's what's giving me 4500KB/s read and 6000KB/s write.

[Paul_Lalonde]

First of all, you should probably establish a baseline of iSCSI performance using a Windows XP or 2003 machine with the MS iSCSI initiator (v2.03).

Then, you should test iSCSI performance again, but this time from a VM running on the ESX server.

If you scan through some of the messages in this forum, you'll see that many people have found the service console to be a poor indicator of true disk performance in ESX 3. This is because VMFS was not designed for read/write operation by the service console, but by the VMkernel.

If you re-run your performance tests using a trusted performance benchmarking tool (like IOMETER) within a VM, you'll definitely see much better performance values than what you're seeing in the service console.

Also, make sure to "tweak" the TCP/IP settings of your Linux iSCSI target. Edit your /etc/sysctl.conf file and add:

net.core.wmem_max=8388608

net.core.wmem_default = 65536

net.core.rmem_default = 65536

net.ipv4.tcp_window_scaling=1

net.ipv4.tcp_mem= 98304 131072 196608

Then restart the iSCSI target system and performance should improve.

Finally, you don't post the specs of the iSCSI system. Remember, to get any kind of performance at all over Gigabit, you need 64-bit, 66MHz (or better) PCI-X or PCI-Express Gigabit adapters and a half-decent server chipset driving the I/O.

In my "white box" setup, I get 115MB/s of iSCSI throughput. It's all about knowing where the I/O tweaks need to be made

Paul

[RParker]

Well something isn't right, we have Quad Core, but similar drives, and I am getting 80Mb/s Read.. I think your RAID config must be setup incorrectly..

[RParker]

Oh yeah, and just to add what Paul said, we use Intel 1000/MT NIC's on PCI-X 4x slots.. we don't use integrated NIC's at all.. so Paul is right about all the other stuff. Whenever possible, upgrade the NIC's turn of internal use of cards.. they don't perform well either.

[Paul_Lalonde]

Yes, there is a

LOT

of truth to this. Depending on the server board vendor, many on-board GigE NICs are "plumbed" into the server chipset in direct competition with other high I/O devices, like RAID controllers.



For example, I have seen two onboard GigE NIC controllers piped into a Southbridge chip, and the Southbridge was also servicing three PCI-X slots. When you throw one (or more) RAID controllers in the system and combine that I/O with two GigE NICs, many inter-bridge link arrangements (ie. Intel HL 1.5) just can't pass that much I/O up to the Northbridge. You end up with a significant bottleneck.

Paul

[jmmarton]

First of all, you should probably establish a

baseline of iSCSI performance using a Windows XP or

2003 machine with the MS iSCSI initiator (v2.03).

Yeah, I was actually thinking of taking another SLES box I have and configure the iSCSI initiator (open-iscsi) to do some benchmarking. But since both of my targets have been configured as VMFS by ESX, I'm assuming I'd have to blow at least one of them away to do any benchmarking either via SLES or Windows, right?

Then, you should test iSCSI performance again, but

this time from a VM running on the ESX server.

I'm getting poor performance that way as well. I have a SLES10 (64-bit) VM running MySQL. When the VM is on local storage (SAS & RAID1), it takes about 25 minutes to do a database restore. When the VM is running on iSCSI, it takes about 55 minutes. I've opened an SR with Novell to try and figure out what might be happening over on the iSCSI side of things. I know, I know, there I'm using SATA II drives and not SAS. But still, since I'm spreading across five drives to get more spindles and I'm using a SATA II RAID controller (3Ware 9550SX) I should be getting better performance than what I'm seeing.

If you scan through some of the messages in this

forum, you'll see that many people have found the

service console to be a poor indicator of true disk

performance in ESX 3. This is because VMFS was not

designed for read/write operation by the service

console, but by the VMkernel.

This is what I'm trying to find out. I'm really new to all of this, so I don't know if using SCP to copy stuff via the service console gives me a true indicator of what's going on. I just didn't want to trust what one tech support guy told me, as I've found tech support in general (not necessarily VMware) isn't always 100% correct. I figured I'd get a number of second opinions here.

If you re-run your performance tests using a trusted

performance benchmarking tool (like IOMETER) within a

VM, you'll definitely see much better performance

values than what you're seeing in the service

console.

Hmm... I had never heard of this, but I've just looked it up. I'll do some benchmarking tomorrow using that tool from within my SLES VM.

Also, make sure to "tweak" the TCP/IP settings of

your Linux iSCSI target. Edit your /etc/sysctl.conf

file and add:

\[text deleted]

Cool, thanks for the tweaks, I'll put them in tomorrow. Perhaps you can help me out with another Linux question as far as tweaking performance goes. 3ware has some tips on tweaking performance when using their controller with the 2.6 kernel. They give stuff like this that can be done on the fly:

echo “512” > /sys/block/sda/queue/nr_requests

I'm assuming that needs to go into /etc/sysctl.conf to make it happen on boot, but I'm not sure of the exact syntax to use.

Finally, you don't post the specs of the iSCSI

system. Remember, to get any kind of performance at

all over Gigabit, you need 64-bit, 66MHz (or better)

PCI-X or PCI-Express Gigabit adapters and a

half-decent server chipset driving the I/O.

Sorry, that would probably help. The box is a Nixsys NIX-200-12RD storage server with two dual-core AMD Opteron 2.00GHz CPUs and 4GB RAM. It has (12) 500GB Seagate ST3500630AS drives. (Barracuda SATA-300, 7200 RPM). Those drives are connected to a 3Ware 9550SX-12MI SATA-II controller (64-bit/133MHz PCI-X). Two drives are in a RAID1 for the OS (SLES10 64-bit), five drives are in a RAID5 and configured as one target, the other five drives are in a second RAID5 configured as a second target.

In my "white box" setup, I get 115MB/s of iSCSI

throughput. It's all about knowing where the I/O

tweaks need to be made

I guess so! Thanks!!!

[jmmarton]

Oh yeah, and just to add what Paul said, we use Intel

1000/MT NIC's on PCI-X 4x slots.. we don't use

integrated NIC's at all.. so Paul is right about all

the other stuff. Whenever possible, upgrade the

NIC's turn of internal use of cards.. they don't

perform well either.

Well, in my ESX server I also have a quad port Intel PRO/1000 PT (PCI Express) card that I'm using. Three of them are bonded together to be used by VMs on the internal LAN, the fourth is for the DMZ. I'm only using the onboard for service console & vmkernel.

I should've mentioned that I'm doing something similar with my storage server. I had two onboard Broadcom gigabit NICs, but I popped in a PRO/1000 PT as well. I wasn't sure if I was having bonding problems, so at this point I've removed all bonding configs from the storage server and I have a single Intel NIC plugged in. Once I'm sure everything else is configured appropriately I'll plug the other three Intel ports in and work on my teaming/bonding.

[Paul_Lalonde]

Well, my "home grown" iSCSI storage server is similarly set up. 3Ware 9550SX-12 with 12 SATA-II drives, Intel dual XEON server motherboard, Intel Pro/1000 Gig NICs, etc.

I get well over 220MB/s when I connect to it with a Windows 2003 Server running the MS iSCSI initiator and MPIO (across two GigE NICs).

There's no question the performance is there. After all, companies like EqualLogic are making incredible inroads with performance on their SATA-based systems.

Paul

[BenConrad]

4.5 - 6MB/s is pretty good for a typical implementation, even on a box that costs $50,000 and has a ton of 15K disks. From my workstation to my ESX servers I usually get 3-4MB/s when copying an ISO. I get 30-40MB/s when importing a VM into a VMFS3 filesystem and on my iSCSI HBA I can get it to 105MB/s if I'm testing with IOMeter... you can see a large difference depending what you are doing.

You probably have some local non-VMFS3 storage on your box right? Put your .ISO file on a fast Windows FTP server and do a 'get' from the ESX console. You should see nice Read/Write performance since it will be going to EXT3.... it should be fast even on RAID-1 as long as you have BBWC

Ben

[jmmarton]

4.5 - 6MB/s is pretty good for a typical

implementation, even on a box that costs $50,000 and

has a ton of 15K disks. From my workstation to my

ESX servers I usually get 3-4MB/s when copying an

ISO. I get 30-40MB/s when importing a VM into a

VMFS3 filesystem and on my iSCSI HBA I can get it to

105MB/s if I'm testing with IOMeter... you can see a

large difference depending what you are doing.

So it sounds like I need to not freak out about the performance copying ISOs, and instead I need to use IOMeter to test performance from a VM. That's where I know I'm definitely having performance issues, with a database restore taking over 2x as long when the VM is on iSCSI storage vs when it's running on local storage.

You probably have some local non-VMFS3 storage on

your box right? Put your .ISO file on a fast Windows

FTP server and do a 'get' from the ESX console. You

should see nice Read/Write performance since it will

be going to EXT3.... it should be fast even on RAID-1

as long as you have BBWC

Well, I do have an NFS mount that's simply an EXT3 partition on my storage server, and all three ESX servers access that NFS mount. That's the real place I'm going to store ISOs. I was simply using an ISO yesterday because it was a quick and dirty way of doing some benchmarking.

[jmmarton]

Well, my "home grown" iSCSI storage server is

similarly set up. 3Ware 9550SX-12 with 12 SATA-II

drives, Intel dual XEON server motherboard, Intel

Pro/1000 Gig NICs, etc.

I get well over 220MB/s when I connect to it with a

Windows 2003 Server running the MS iSCSI initiator

and MPIO (across two GigE NICs).

So does that mean you're connecting to it via the MS iSCSI initiator and not from ESX Server? What performance do you get from ESX? BTW, and I should've mentioned this beforeI'm sure this makes a differenceI'm using a QLogic QLA4052 iSCSI HBA to connect ESX to the storage server. I'm not using VMware's software iSCSI initiator. Presumably that's a good thing.

There's no question the performance is there. After

all, companies like EqualLogic are making incredible

inroads with performance on their SATA-based

systems.

Well at this point I'd assume I'd have "ultimate" performance if I was using SAS instead of SATA. But the problem is that since we had to purchase two new servers (the third ESX server was purchased last fiscal year), two copies of VI3, and some form of shared storage, we had to go with a less expensive storage for now. We needed to spread the costs out over multiple fiscal years. This box has to last a year, then next year we get a real SAN while we move the SATA storage box to our DR site. Then a year after that we'll implement a real SAN up there as well. Eventually we'll have a really nice fully redundant and automated setup, but we have to take baby steps. So I figured this $7300 white box storage server was a way to take a small step forward.

[BlueDragon]

hi,

i've read all these post. Nice answers

however,

For copying an Iso throw LAN between a workstation and ESX production serveur, i use Bitvise tunnelier, works like a charm ! give it a try => http://www.bitvise.com/tunnelier. While SCP took 1H15 to upload an 650Mo ISO, tunnelier only took 15Min.

I've also try the VCB with ISCSI. (Ms inititor for the W2K3 VCB Box, software initiator on ESX and VM Linux OpenFiler on ESX localstorage). Works like a charm as well and took about 5min to download a 10Go vmdk.

hope this will help

cya

Message was edited by:

BlueDragon

[CWedge]

I just wanted to add that the Console is NOT a good idea to test performance with.

To get best results, use a VM.

I know it sounds weird but it is what it is.

Disk and Network I/O are all Blazingly a lot fast through a VM when compared to the console.

[bertdb]

to lessen the effect of the encryption overhead, configure sshd to allow blowfish-cbc encryption (/etc/ssh/sshd_config) and copy using scp -c blowfish-cbc . That will be less heavy on the CPU compared to aes128 and aes256 encryption. Still, everything gets encrypted, so there is substantial overhead.

How fast is a vmkfstools -i to one of your VMFS LUNs ? scp into VMFS is slow because it does a lot of block allocations, which VMFS isn't made to do well. VMFS expects file creations with the correct size in one go, which is what vmkfstools does.

[bertdb]

it doesn't even sound weird. Writing to a VMFS happens through the VMkernel, so the console Linux kernel will have to ask the VMkernel to write everything, rather than being able to do it all by itself.

[CWedge]

it doesn't even sound weird. Writing to a VMFS

happens through the VMkernel, so the console Linux

kernel will have to ask the VMkernel to write

everything, rather than being able to do it all by

itself.

in general Consoles are usually the quickest in most cases.

ESX happeneds to be the exception to the rule.