Strange disk performance during manual file transfers

[jeffmeden]

I am using the data store browser to manually transfer some virtual hosts (just issuing a 'move to...' command on the whole directory) between iSCSI partitions and between the local scsi disk and the iSCSI partitions. It is software iSCSI to a dell MD device. The strange thing I am noticing is that there seems to be an artificial bottle neck around 200 mbit/s despite a dedicated gigabit connection to the iscsi device. Running multiple transfers causes the overall throughput to climb closer to the theoretical maximum (600-700 mbit/s) so something strange is happening in software (i think). There is relatively little virtual host activity during these transfers, so I really doubt it is a resource contention issue (at least not with the virtual machines).

The final clue that something strange is going on is the performance graph of disk and network activity. The network activity graph shows that much more is being received than being sent (when I am moving between iSCSI partitions and there should be a 1:1 transfer ratio, it is closer to 2:1 sent/rec). Disk usage shows similar activity, but it is even more bizarre as there will be times during the tranfer that read activity goes to at/near 0 and write activity shoots up, as if it is reading from some sort of buffer but since the read rate has been consistently lower I am not sure how this is possible. The disk activity is around 2:1 sent/rec as well, which pretty much always matches the network i/o numbers.

Here is a performance chart showing the curious activity. In this case, the transfer was from a local scsi disk (purple line) to a software iSCSI disk (yellow line).

Can anyone offer some advice on how to better measure this, or where to look for possible problems? I am willing to accept performance issues related to the use of software iSCSI but I really would like to know more about what the limitations are and the practical ways to get the most out of the system performance. I have read most of the relevant documents I could find (the iscsi san config guide, the vsphere 4.0 performance tuning guide, etc.) but am willing to read more if you could point me in the right direction. Thanks in advance!

[J1mbo]

When copying files using (eg) datastore browser, the write rate is always doubled. I have concluded that this is because the system is counting the "eager-zero'ing" of the new disk space being allocated as write (and indeed, appears to be actually doing so).

This behaviour is also observable when undertaking large block-sequential-write to a VM's file system that is stored on a thin-provisioned volume. In this example, the first time the data is written we see the doubled-write counters to the performance observed within the VM. But where the OS level file is deleted and then re-created, the counters tally with the OS observed performance (since the physical disk space is then allocated already).

Please award points to any useful answer.

[jeffmeden]

That would make sense, the partition the files are moving to is a new one. Any idea why the performance is pathetic even compared to manually running two transfers simultaneously? Where should I look for the bottleneck?

[J1mbo]

How many drives does the array have, and of what type? Also, could you provide a little info on the network/switching infrastructure between them?

Please award points to any useful answer.

[jeffmeden]

Its a 15 drive array of 15k disks, and a simple 1GBit switch sits between the servers running esxi and the disk box, and the only traffic on that switch is disk activity.

[J1mbo]

Should be good then. I would focus more on performance within VMs than performance of ESX operations.

Download IOMeter if you haven't already done so, test 32K sequential read (and seperately, write) against a 4GB test file with 32 IOs outstanding. Then test 100% random performance. With that array you should also be able to increase outstanding IOs quite a bit before latency gets up to about 50ms.

You can also test the 100% cached throughput by selecting "test connection rate" and selecting about 500 in the box. This will (hopefully) be very close to the max (about 110MB/s per GigE path to the storage).

Please award points to any useful answer.