VMware Communities: I/O performance of vSphere

Up to Discussions in Performance & VMmark

This Question is Not Answered

1 "correct" answer available (10 pts) 2 "helpful" answers available (6 pts)

12,375 Views 31 Replies Last post: Jul 9, 2009 10:23 AM by chethank

1 2 3 Previous Next

chethank

10 posts since
Jan 20, 2009

Currently Being Moderated

May 26, 2009 11:55 AM

I/O performance of vSphere

I am a senior performance engineer at VMware. I have spent quite a bit of time on performance analysis of ESX storage stack starting from ESX 3.0. We at VMware are constantly working on improving I/O performance of ESX. I have done few experiments to drive extremely high I/O load on single instance of ESX. With 3.5, I obtained 100,000 I/OPs with an I/O load that is most representative of real applications (100,000 I/Ops) until I ran out of hardware. The performance envelope was pushed furthe with vSphere when we achieved 350,000 I/Ops in an experiment done at EMC labs. I wrote a blog highlighting the results with some details on the experiments in VMware's performance blog VROOM.

Having a well configured I/O system is critical for good application performance. Very often I hear questions from customers on storage performance, choice of virtual disk format - VMFS vs RDM, best practices etc., The answer is simple - follow the best practices that you would normally follow in native world when designing an I/O infrastructure for your application. ESX provides excellent I/O performance and can support even extreme I/O demands from applications as the results discussed in the blogs indicate. VMFS or RDM - you can expect similar performance though RDM can help during certain scenarios which are purely non-performance related.

We can discuss more in this thread. Feel free to post your questions, comments on the blogs or any I/O related issues on this thread. I will try my best to respond. May be some one who has already faced a similar situation will jump in with a solution even we at VMware wouldn't have thought of!

Chethan

Tags: performance, san, storage, vmfs, disk, 4.0, vsphere_performance

rDale

71 posts since
Mar 22, 2007

Currently Being Moderated

1. May 26, 2009 2:43 PM

in response to: chethank

Re: I/O performance of vSphere

I have a question in order to get tose kinds of IOPS did you do any disk alignment with the vmfs vol or the vmdk files ?

I often hear alot of blah about disk alignment being needed for SQL servers and Exchnage servers.

what block size did you work with on the ntfs partions ?

Report Abuse

Gabrie

1,079 posts since
Jun 6, 2005

Currently Being Moderated

2. May 27, 2009 9:40 AM

in response to: chethank

Re: I/O performance of vSphere

Q1: A while ago I saw a discussion on VMFS blocksize. I learned that VMFS blocksize doesnot matter when Windows Guest is reading for example a 64kb block from vmdk on vmfs. So my question is, when does the blocksize matter if it has no influence on performance and why should I choose between 1 or 8MB blocksize?

Q2: What is the best way to track performance from Guest disk through Guest SCSI controller through Host HBA Queue depth to LUN Queue depth? Only vScsiStats and esxtop? Or other tools aswell?

http://www.GabesVirtualWorld.com

Report Abuse

chethank

10 posts since
Jan 20, 2009

Currently Being Moderated

3. May 27, 2009 10:34 AM

in response to: rDale

Re: I/O performance of vSphere

The virtual disks were created using the VI client, which automatically aligns the vmdk files. As far as guest OS is concerned, no file system was created on these virtual disks. For the guest OS (and iometer) they appeared as raw disks.

But, if you were to create a file system for a particular application, my suggestion would be to go with the best practices suggested by Microsoft for particular applications. For e.g., MS recommends 64K NTFS block size for SQL server files.

Hope that helps.

Report Abuse

chethank

10 posts since
Jan 20, 2009

Currently Being Moderated

4. May 27, 2009 10:51 AM

in response to: Gabrie

Re: I/O performance of vSphere

You choose different block sizes depending on the size of the largest file (vmdks) on the VMFS datastore. For e.g., with 1MB block size, maximum size for a single vmdk is 256GB. The limit increases as you increase the block size.

vScsiStats and esxtop are very popular. You can also use tools provided by storage vendors (such as Navisphere Analyzer from EMC) to complement esxtop data.

Report Abuse

Gabrie

1,079 posts since
Jun 6, 2005

Currently Being Moderated

5. May 27, 2009 11:52 AM

in response to: chethank

Re: I/O performance of vSphere

chethank wrote:
You choose different block sizes depending on the size of the largest file (vmdks) on the VMFS datastore. For e.g., with 1MB block size, maximum size for a single vmdk is 256GB. The limit increases as you increase the block size.

vScsiStats and esxtop are very popular. You can also use tools provided by storage vendors (such as Navisphere Analyzer from EMC) to complement esxtop data.

I understand about the blocksize and the 256GB limit for a VMDK on a 1MB blocksize VMFS. But why shouldn't I just create only 8MB blocked VMFS in the future? Why is it better to create 1MB block size VMFS when you don't need VMDK bigger then 256GB?

http://www.GabesVirtualWorld.com

Report Abuse

chethank

10 posts since
Jan 20, 2009

Currently Being Moderated

6. May 28, 2009 5:47 PM

in response to: Gabrie

Re: I/O performance of vSphere

There is no reason why you shouldn't use 8MB block size instead of 1MB. In case of 'thin' vmdk format unit of space allocation is the minimum block size used for VMFS. As the file grows space is allocated in chunks of the chosen block size. If the application has periods of bursty writes (which causes file growth) followed by period of inactivity, this may lead to some wastage in space. This may not ber very significant compared to the acual file size, but just something to keep in mind.

Report Abuse

Gabrie

1,079 posts since
Jun 6, 2005

Currently Being Moderated

7. May 28, 2009 11:02 PM

in response to: chethank

Re: I/O performance of vSphere

Thx !!!

http://www.GabesVirtualWorld.com

Report Abuse

jasonboche

6,071 posts since
Jan 7, 2004

Currently Being Moderated

8. May 28, 2009 11:08 PM

in response to: chethank

Re: I/O performance of vSphere

chethank wrote:
The virtual disks were created using the VI client, which automatically aligns the vmdk files.

Not in my experience.

The VI Client will align VMFS volumes at creation time, but not the guest OS in the .vmdk.

I've been aligning a lot of .vmdk's lately that were created with VI3.

Would like clarification here.

As far as guest OS is concerned, no file system was created on these virtual disks. For the guest OS (and iometer) they appeared as raw disks.

But, if you were to create a file system for a particular application, my suggestion would be to go with the best practices suggested by Microsoft for particular applications. For e.g., MS recommends 64K NTFS block size for SQL server files.

Hope that helps.

[i]Jason Boche, vExpert[/i]

Re: I/O performance of vSphere[boche.net - VMware Virtualization Evangelist|http://www.boche.net/blog/][/i]

Re: I/O performance of vSphere[VMware Communities User Moderator|http://www.vmware.com/communities/content/community_terms/][/i]

Re: I/O performance of vSphere[Minneapolis Area VMware User Group Leader|http://communities.vmware.com/community/vmug/us-central/minneapolis][/i]

VCDX3 #34, VCDX4, VCAP4-DCA #14, VCAP4-DCD #35, VCPx4, vEXPERTx3, MCSEx3, MCSAx2, MCP, CCAx2, A+

Report Abuse

depping

4,238 posts since
Jan 17, 2005

Currently Being Moderated

9. May 28, 2009 11:10 PM

in response to: Gabrie

Re: I/O performance of vSphere

I wrote an article about it a while ago. Especially with the new VMFS Grow feature it makes sense to always use 8MB Blocks. If you do you will always have the option to create the largest file possible. And indeed for Thin Provisioning you will grow in increments of 8MB vs 1MB which should lead to less locking and less growing. The overhead should be minumum. (max = 7MB x total amount of VMs)

http://www.yellow-bricks.com/2009/05/14/block-sizes-and-growing-your-vmfs/

Duncan

VMware Communities User Moderator | VCP | VCDX

Blogging: http://www.yellow-bricks.com

Twitter: http://www.twitter.com/depping

If you find this information useful, please award points for "correct" or "helpful".

Duncan | Yellow-Bricks.com | Author of the vSphere 5.0 Clustering Deepdive

Report Abuse

dconvery

2,235 posts since
May 10, 2006

Currently Being Moderated

10. May 29, 2009 5:39 AM

in response to: chethank

Re: I/O performance of vSphere

Chethank -

I would like a clarification on alignment as well. What I am hearing you say is to create VMFS in the VIC and it would be automatically aligned (I already knew that part). I am not sure I understand if the guest needs anything special to aling from default setting. Are you saying that the guest will be aligned as well? or is there something special I need to do within the guest?

Dave Convery

VMware vExpert 2009

http://www.dailyhypervisor.com

Careful. We don't want to learn from this.

Bill Watterson, "Calvin and Hobbes"

Dave Convery, VCDX3,4 #20 ** http://www.tech-tap.com ** http://twitter.com/dconvery ** "Careful. We don't want to learn from this." -Bill Watterson, "Calvin and Hobbes"

Report Abuse

depping

4,238 posts since
Jan 17, 2005

Currently Being Moderated

11. May 29, 2009 5:47 AM

in response to: dconvery

Re: I/O performance of vSphere

I think he only stated that he did not need to allign the additional disk cause IOMeter would access the raw disk without it being formatted with a file system?

When a it's formatted with a filesystem it will need to be alligned depending on the filesystem used.

Duncan

VMware Communities User Moderator | VCP | VCDX

Blogging: http://www.yellow-bricks.com

Twitter: http://www.twitter.com/depping

If you find this information useful, please award points for "correct" or "helpful".

Duncan | Yellow-Bricks.com | Author of the vSphere 5.0 Clustering Deepdive

Report Abuse

IRivera

9 posts since
Jun 16, 2008

Currently Being Moderated

12. May 29, 2009 6:00 AM

in response to: depping

Re: I/O performance of vSphere

I think you'll find this article very helpful on aligning Windows .vmdk

http://www.vmware.com/pdf/esx3_partition_align.pdf

Report Abuse

dconvery

2,235 posts since
May 10, 2006

Currently Being Moderated

13. May 29, 2009 6:09 AM

in response to: depping

Re: I/O performance of vSphere

Yeah, I think I was reading into Jason's questions too much. I use the steps outlined in the partition alignment whitepaper for the guests right now.

Let me ask this: Is there an easy way to align the C: drive on a windows system? I have gone through the process using a helper VM to partition and format the VMDK and then install the OS on the newly-formatted disk, but there must be a better way. Is it really necessary?

Dave Convery

VMware vExpert 2009

http://www.dailyhypervisor.com

Careful. We don't want to learn from this.

Bill Watterson, "Calvin and Hobbes"

Dave Convery, VCDX3,4 #20 ** http://www.tech-tap.com ** http://twitter.com/dconvery ** "Careful. We don't want to learn from this." -Bill Watterson, "Calvin and Hobbes"

Report Abuse

chethank

10 posts since
Jan 20, 2009

Currently Being Moderated

14. May 29, 2009 10:56 AM

in response to: chethank

Re: I/O performance of vSphere

CLARIFICATION: As some of the readers have pointed out, creation of a VMFS datastore from VI client automatically aligns VMFS on a LUN. And yes, this is what I did too. But this doesn't align a file system created on a virtual disk (.vmdk) in the guest OS. That has to be done manually. This document has some helpful information on alignment: http://www.vmware.com/pdf/esx3_partition_align.pdf . Sorry for the confusion in my earlier response.

>> As far as guest OS is concerned, no file system was created on these virtual disks. For the guest OS (and iometer) they appeared as raw disks.

In our case, since no file system was created on the virtual disk in the guest OS, the virtual disks didn't require any alignment.

Hope the clarification helps.

Report Abuse

Go to original post 1 2 3 Previous Next

Go to original post

This Question is Not Answered

May 26, 2009 11:55 AM

I/O performance of vSphere

Actions

Bookmarked By (0)

Share This Page

Communities