VMware Communities: I/O performance of vSphere ...

Up to Discussions in Performance & VMmark

This Question is Not Answered

1 "correct" answer available (10 pts) 2 "helpful" answers available (6 pts)

1 2 3 Previous Next 31 Replies Last post: Jul 9, 2009 10:23 AM by chethank

I/O performance of vSphere posted: May 26, 2009 11:55 AM

chethank

8 posts since
Jan 20, 2009

I am a senior performance engineer at VMware. I have spent quite a bit of time on performance analysis of ESX storage stack starting from ESX 3.0. We at VMware are constantly working on improving I/O performance of ESX. I have done few experiments to drive extremely high I/O load on single instance of ESX. With 3.5, I obtained 100,000 I/OPs with an I/O load that is most representative of real applications (100,000 I/Ops) until I ran out of hardware. The performance envelope was pushed furthe with vSphere when we achieved 350,000 I/Ops in an experiment done at EMC labs. I wrote a blog highlighting the results with some details on the experiments in VMware's performance blog VROOM.

Having a well configured I/O system is critical for good application performance. Very often I hear questions from customers on storage performance, choice of virtual disk format - VMFS vs RDM, best practices etc., The answer is simple - follow the best practices that you would normally follow in native world when designing an I/O infrastructure for your application. ESX provides excellent I/O performance and can support even extreme I/O demands from applications as the results discussed in the blogs indicate. VMFS or RDM - you can expect similar performance though RDM can help during certain scenarios which are purely non-performance related.

We can discuss more in this thread. Feel free to post your questions, comments on the blogs or any I/O related issues on this thread. I will try my best to respond. May be some one who has already faced a similar situation will jump in with a solution even we at VMware wouldn't have thought of!

Chethan

Tags: vsphere_performance, performance, disk, san, storage, vmfs, 4.0

Re: I/O performance of vSphere

1. May 26, 2009 2:43 PM

in response to: chethank

rDale

71 posts since
Mar 22, 2007

I have a question in order to get tose kinds of IOPS did you do any disk alignment with the vmfs vol or the vmdk files ?

I often hear alot of blah about disk alignment being needed for SQL servers and Exchnage servers.

what block size did you work with on the ntfs partions ?

Re: I/O performance of vSphere

2. May 27, 2009 9:40 AM

in response to: chethank

Gabrie

887 posts since
Jun 6, 2005

Q1: A while ago I saw a discussion on VMFS blocksize. I learned that VMFS blocksize doesnot matter when Windows Guest is reading for example a 64kb block from vmdk on vmfs. So my question is, when does the blocksize matter if it has no influence on performance and why should I choose between 1 or 8MB blocksize?

Q2: What is the best way to track performance from Guest disk through Guest SCSI controller through Host HBA Queue depth to LUN Queue depth? Only vScsiStats and esxtop? Or other tools aswell?

http://www.GabesVirtualWorld.com

Re: I/O performance of vSphere

3. May 27, 2009 10:34 AM

in response to: rDale

chethank

8 posts since
Jan 20, 2009

The virtual disks were created using the VI client, which automatically aligns the vmdk files. As far as guest OS is concerned, no file system was created on these virtual disks. For the guest OS (and iometer) they appeared as raw disks.

But, if you were to create a file system for a particular application, my suggestion would be to go with the best practices suggested by Microsoft for particular applications. For e.g., MS recommends 64K NTFS block size for SQL server files.

Hope that helps.

Re: I/O performance of vSphere

4. May 27, 2009 10:51 AM

in response to: Gabrie

chethank

8 posts since
Jan 20, 2009

You choose different block sizes depending on the size of the largest file (vmdks) on the VMFS datastore. For e.g., with 1MB block size, maximum size for a single vmdk is 256GB. The limit increases as you increase the block size.

vScsiStats and esxtop are very popular. You can also use tools provided by storage vendors (such as Navisphere Analyzer from EMC) to complement esxtop data.

Re: I/O performance of vSphere

5. May 27, 2009 11:52 AM

in response to: chethank

Gabrie

887 posts since
Jun 6, 2005

chethank wrote:
You choose different block sizes depending on the size of the largest file (vmdks) on the VMFS datastore. For e.g., with 1MB block size, maximum size for a single vmdk is 256GB. The limit increases as you increase the block size.

vScsiStats and esxtop are very popular. You can also use tools provided by storage vendors (such as Navisphere Analyzer from EMC) to complement esxtop data.

I understand about the blocksize and the 256GB limit for a VMDK on a 1MB blocksize VMFS. But why shouldn't I just create only 8MB blocked VMFS in the future? Why is it better to create 1MB block size VMFS when you don't need VMDK bigger then 256GB?

http://www.GabesVirtualWorld.com

Re: I/O performance of vSphere

6. May 28, 2009 5:47 PM

in response to: Gabrie

chethank

8 posts since
Jan 20, 2009

There is no reason why you shouldn't use 8MB block size instead of 1MB. In case of 'thin' vmdk format unit of space allocation is the minimum block size used for VMFS. As the file grows space is allocated in chunks of the chosen block size. If the application has periods of bursty writes (which causes file growth) followed by period of inactivity, this may lead to some wastage in space. This may not ber very significant compared to the acual file size, but just something to keep in mind.

Re: I/O performance of vSphere

7. May 28, 2009 11:02 PM

in response to: chethank

Gabrie

887 posts since
Jun 6, 2005

Thx

http://www.GabesVirtualWorld.com

Re: I/O performance of vSphere

8. May 28, 2009 11:08 PM

in response to: chethank

jasonboche

5,896 posts since
Jan 7, 2004

chethank wrote:

The virtual disks were created using the VI client, which automatically aligns the vmdk files.

Not in my experience.
The VI Client will align VMFS volumes at creation time, but not the guest OS in the .vmdk.
I've been aligning a lot of .vmdk's lately that were created with VI3.
Would like clarification here.

As far as guest OS is concerned, no file system was created on these virtual disks. For the guest OS (and iometer) they appeared as raw disks.

Hope that helps.

Jason Boche, vExpert
boche.net - VMware Virtualization Evangelist
VMware Communities User Moderator
Minneapolis Area VMware User Group Leader

Re: I/O performance of vSphere

9. May 28, 2009 11:10 PM

in response to: Gabrie

depping

2,997 posts since
Jan 17, 2005

I wrote an article about it a while ago. Especially with the new VMFS Grow feature it makes sense to always use 8MB Blocks. If you do you will always have the option to create the largest file possible. And indeed for Thin Provisioning you will grow in increments of 8MB vs 1MB which should lead to less locking and less growing. The overhead should be minumum. (max = 7MB x total amount of VMs)

http://www.yellow-bricks.com/2009/05/14/block-sizes-and-growing-your-vmfs/

Duncan
VMware Communities User Moderator | VCP | VCDX

Blogging: http://www.yellow-bricks.com
Twitter: http://www.twitter.com/depping

If you find this information useful, please award points for "correct" or "helpful".

Re: I/O performance of vSphere

10. May 29, 2009 5:39 AM

in response to: chethank

dconvery

1,887 posts since
May 10, 2006

Chethank -

I would like a clarification on alignment as well. What I am hearing you say is to create VMFS in the VIC and it would be automatically aligned (I already knew that part). I am not sure I understand if the guest needs anything special to aling from default setting. Are you saying that the guest will be aligned as well? or is there something special I need to do within the guest?

Dave Convery
VMware vExpert 2009
http://www.dailyhypervisor.com

Careful. We don't want to learn from this.
Bill Watterson, "Calvin and Hobbes"

Re: I/O performance of vSphere

11. May 29, 2009 5:47 AM

in response to: dconvery

depping

2,997 posts since
Jan 17, 2005

I think he only stated that he did not need to allign the additional disk cause IOMeter would access the raw disk without it being formatted with a file system?

When a it's formatted with a filesystem it will need to be alligned depending on the filesystem used.

Duncan
VMware Communities User Moderator | VCP | VCDX

Blogging: http://www.yellow-bricks.com
Twitter: http://www.twitter.com/depping

If you find this information useful, please award points for "correct" or "helpful".

Re: I/O performance of vSphere

12. May 29, 2009 6:00 AM

in response to: depping

IRivera

5 posts since
Jun 16, 2008

I think you'll find this article very helpful on aligning Windows .vmdk

http://www.vmware.com/pdf/esx3_partition_align.pdf

Re: I/O performance of vSphere

13. May 29, 2009 6:09 AM

in response to: depping

dconvery

1,887 posts since
May 10, 2006

Yeah, I think I was reading into Jason's questions too much. I use the steps outlined in the partition alignment whitepaper for the guests right now.

Let me ask this: Is there an easy way to align the C: drive on a windows system? I have gone through the process using a helper VM to partition and format the VMDK and then install the OS on the newly-formatted disk, but there must be a better way. Is it really necessary?

Dave Convery
VMware vExpert 2009
http://www.dailyhypervisor.com

Careful. We don't want to learn from this.
Bill Watterson, "Calvin and Hobbes"

Re: I/O performance of vSphere

14. May 29, 2009 10:56 AM

in response to: chethank

chethank

8 posts since
Jan 20, 2009

CLARIFICATION: As some of the readers have pointed out, creation of a VMFS datastore from VI client automatically aligns VMFS on a LUN. And yes, this is what I did too. But this doesn't align a file system created on a virtual disk (.vmdk) in the guest OS. That has to be done manually. This document has some helpful information on alignment: http://www.vmware.com/pdf/esx3_partition_align.pdf . Sorry for the confusion in my earlier response.

As far as guest OS is concerned, no file system was created on these virtual disks. For the guest OS (and iometer) they appeared as raw disks.

In our case, since no file system was created on the virtual disk in the guest OS, the virtual disks didn't require any alignment.

Hope the clarification helps.

1 2 3 Previous Next Go to original post

Go to original post

Search VMware Communities Advanced Search

Actions

View as PDF

Up to Discussions in Performance & VMmark

This Question is Not Answered

I/O performance of vSphere posted: May 26, 2009 11:55 AM

Re: I/O performance of vSphere

Re: I/O performance of vSphere

Re: I/O performance of vSphere

Re: I/O performance of vSphere

Re: I/O performance of vSphere

Re: I/O performance of vSphere

Re: I/O performance of vSphere

Re: I/O performance of vSphere

Re: I/O performance of vSphere

Re: I/O performance of vSphere

Re: I/O performance of vSphere

Re: I/O performance of vSphere

Re: I/O performance of vSphere

Re: I/O performance of vSphere

Actions

Communities

Community Resources

VMworld

Get Support

Other Links

This Question is Not Answered

I/O performance of vSphere posted: May 26, 2009 11:55 AM

Actions

VMware Developer

Developer Sample Code

VMworld Sessions & Labs

Purchase PSO Credits Online

Community Hardware Software

VMware vSphere

Community Resources

VMworld

Get Support

Other Links