VMware Cloud Community
tys
Contributor
Contributor
‎04-13-2008 08:14 AM

ESX 3.5 disk write performance issue

Hi,

I've serious disk performance issue on VM running on ESX 3.5.

Here are issue results and a procedure to duplicate.

Please let me know if you have same result and I also would like to know how to fix the issue.

I've already opened SR however VMware Support says that the behavior of the system is normal and also by design.

But I can't accept their response and I'd like to know if another ESX 3.5 users know this issue.

HW Environment:

HP Blade C3000

BL460c (2xQuadCore/16GBRAM/2xFCHBA)

MSA1000(ActiveActiveFirmware)

LUN01 RAID5

LUN02 RAID10

SW Environment:

Host ESX 3.5

Guest1 WindowsServer2003 32bit

C:\ on LUN01(RAID5)

D:\ on LUN02(RAID5)

vRAM 1GB

vCPU 2

Guest2 WindowsServer2003 64bit

C:\ on LUN02(RAID10)

D:\ on LUN02(RAID10)

vRAM 2GB

vCPU 4

Problem:

When disk write operation happens on VM, write performance dramatically degraded and as the result, ping reply delayed also. No problem for read operation.

Procedure to duplicate:

1. Create the new additional vDisk and assign it to the VM(with default settings). Don't reuse vDisk which has been using.

2. Format the disk with NTFS(Quick or Normal format, whichever) and create Windows Share.

3. Prepare e.g. 5xISO(500MB) images on some client PC.

4. Start "ping vm -t" from another machine which is out of ESX systems.

5. Copy 5xISO one by one from client PC (use 5 session) to VM's share folder.

Result:

After about 3mins copy started,

Reply from 192.168.22.77: bytes=32 time=77ms TTL=127

Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=127ms TTL=127
Reply from 192.168.22.77: bytes=32 time=13ms TTL=127
Reply from 192.168.22.77: bytes=32 time=89ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=60ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=87ms TTL=127
Reply from 192.168.22.77: bytes=32 time=34ms TTL=127
Reply from 192.168.22.77: bytes=32 time=105ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=1ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=9ms TTL=127
Reply from 192.168.22.77: bytes=32 time=322ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=93ms TTL=127
Reply from 192.168.22.77: bytes=32 time=19ms TTL=127
Reply from 192.168.22.77: bytes=32 time=95ms TTL=127
Reply from 192.168.22.77: bytes=32 time=47ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=17ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=17ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=12ms TTL=127
Reply from 192.168.22.77: bytes=32 time=44ms TTL=127
Reply from 192.168.22.77: bytes=32 time=222ms TTL=127
Reply from 192.168.22.77: bytes=32 time=119ms TTL=127
Reply from 192.168.22.77: bytes=32 time=28ms TTL=127
Reply from 192.168.22.77: bytes=32 time=6ms TTL=127
Reply from 192.168.22.77: bytes=32 time=91ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=47ms TTL=127
Reply from 192.168.22.77: bytes=32 time=16ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=27ms TTL=127
Reply from 192.168.22.77: bytes=32 time=51ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=47ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=2ms TTL=127
Reply from 192.168.22.77: bytes=32 time=2ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=9ms TTL=127
Reply from 192.168.22.77: bytes=32 time=29ms TTL=127
Reply from 192.168.22.77: bytes=32 time=142ms TTL=127
Reply from 192.168.22.77: bytes=32 time=213ms TTL=127
Reply from 192.168.22.77: bytes=32 time=27ms TTL=127
Reply from 192.168.22.77: bytes=32 time=91ms TTL=127
Reply from 192.168.22.77: bytes=32 time=30ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=25ms TTL=127
Reply from 192.168.22.77: bytes=32 time=76ms TTL=127
Reply from 192.168.22.77: bytes=32 time=87ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=42ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=52ms TTL=127
Reply from 192.168.22.77: bytes=32 time=46ms TTL=127
Reply from 192.168.22.77: bytes=32 time=61ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=47ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=201ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=31ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=25ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=17ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=188ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=12ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=117ms TTL=127
Reply from 192.168.22.77: bytes=32 time<1ms TTL=127
Reply from 192.168.22.77: bytes=32 time=28ms TTL=127

Generating VM Local disk write opearation such as copy&pasting of mutiple big files will cause same result.

My investigation so far:
1. Build another simple HW and network config ESX3.5 server(DL380+Local SCSIDisks) to reduce the possiblity of network or hardware issue. (because My blade system is somewhat complicated especially for networking.)
>>> Same issue happened.
2. Test with ESX3.0.1.
>>> See similar issue, however from the "degree" point of view acceptable.
3. Test with VMware Server 1.0.3. (with PreAllocated&NonPreAllocated disk)
>>> No trouble at all.

If you have any information about this, please kindly let me know.

Regards,

Tys

Preview file
15 KB
0 Kudos
5 Replies
Anders_Gregerse
Hot Shot
Hot Shot
‎04-13-2008 10:43 AM

Any performance difference between Quick and normal NTFS format? Normally, Quick will result in the OS need to initialize each disk cluster before the first write, slowing performance.

0 Kudos
christianZ
Champion
Champion
‎04-13-2008 10:44 AM

My thought is that the write cache on raid controller is not enabled.

0 Kudos
kastlr
Expert
Expert
‎04-13-2008 02:37 PM

Hi,

at first, you should align the VMFS datastore AND the NTFS.

Check the following article for more details.

http://www.vmware.com/pdf/esx3_partition_align.pdf

To check the storage performance, you should create (at least) the following conditions.

- use CPU and memory reservations to avoid ESX swapping while running your tests

- use multi instead of singlethreading tools to generate disk i/o's (i.e. IOMeter with several dynamos)

- use a proper IO profile (i.e. 70% read IO's, 30 % write IO's)

- perform a long test to reduce cache effects

If you need high performance, you shouldn't use RAID5.

While RAID5 does provide good read performance, write performance isn't that good.

Each host write operation requires at least 3 IO's in the backend.

This activity is usually transparent to the host because each write will be acknowledged by the cache.

But when the cache is filled, it must destage content from the cache to the disks.

Now the write performance would be an issue.

And you should verify if you're talking about a disk or network performance.

When performing copy jobs to a share, you need to verify the NIC configuration.

VMWare does recommend to disable Auto Negotiation, so set your NIC's to a fixed speed and duplex mode.

Hope this helps a bit.


Hope this helps a bit.
Greetings from Germany. (CEST)
0 Kudos
tys
Contributor
Contributor
‎04-13-2008 05:46 PM

Hi, thank you all for your comments.

- In my case whether format is QUICK or Normal, there are no difference in the issue results.

- I configured MSA1000's cache from the beggining. (Read:Write = 30:70)

- This is not a network issue, because copying the data inside VM cause same performance issue. Also my test machine in quite different network and with very simple network configuration also shows the same issue.

- I understand RAID5 performance is low and accept the lower performance, however my issue also happen on RAID10 LUN. Even if disk performance is low, is this ping result normal? This is my point.

I'm reading Disk Allignment document now...

Tys

0 Kudos
tys
Contributor
Contributor
‎04-13-2008 06:48 PM

Hi,

Let me correct the question. The title "ESX 3.5 disk write performance issue" may cause the confution.

Disk performance itself doesn't matter, because RAID5, RAID10 each array has it's performance and I of course understand that.

However the ping results during file copy, is this normal? All ESX3.5 user recognize and accept this results?

Regards,

Tys

0 Kudos