VMware Cloud Community
admin
Immortal
Immortal

I/O performance of vSphere

I am a senior performance engineer at VMware. I have spent quite a bit of time on performance analysis of ESX storage stack starting from ESX 3.0. We at VMware are constantly working on improving I/O performance of ESX. I have done few experiments to drive extremely high I/O load on single instance of ESX. With 3.5, I obtained 100,000 I/OPs with an I/O load that is most representative of real applications (100,000 I/Ops) until I ran out of hardware. The performance envelope was pushed furthe with vSphere when we achieved 350,000 I/Ops in an experiment done at EMC labs. I wrote a blog highlighting the results with some details on the experiments in VMware's performance blog VROOM.

Having a well configured I/O system is critical for good application performance. Very often I hear questions from customers on storage performance, choice of virtual disk format - VMFS vs RDM, best practices etc., The answer is simple - follow the best practices that you would normally follow in native world when designing an I/O infrastructure for your application. ESX provides excellent I/O performance and can support even extreme I/O demands from applications as the results discussed in the blogs indicate. VMFS or RDM - you can expect similar performance though RDM can help during certain scenarios which are purely non-performance related.

We can discuss more in this thread. Feel free to post your questions, comments on the blogs or any I/O related issues on this thread. I will try my best to respond. May be some one who has already faced a similar situation will jump in with a solution even we at VMware wouldn't have thought of!

Chethan

Reply
0 Kudos
31 Replies
rDale
Enthusiast
Enthusiast

I have a question in order to get tose kinds of IOPS did you do any disk alignment with the vmfs vol or the vmdk files ?

I often hear alot of blah about disk alignment being needed for SQL servers and Exchnage servers.

what block size did you work with on the ntfs partions ?

Reply
0 Kudos
Gabrie1
Commander
Commander

Q1: A while ago I saw a discussion on VMFS blocksize. I learned that VMFS blocksize doesnot matter when Windows Guest is reading for example a 64kb block from vmdk on vmfs. So my question is, when does the blocksize matter if it has no influence on performance and why should I choose between 1 or 8MB blocksize?

Q2: What is the best way to track performance from Guest disk through Guest SCSI controller through Host HBA Queue depth to LUN Queue depth? Only vScsiStats and esxtop? Or other tools aswell?

http://www.GabesVirtualWorld.com
Reply
0 Kudos
admin
Immortal
Immortal

The virtual disks were created using the VI client, which automatically aligns the vmdk files. As far as guest OS is concerned, no file system was created on these virtual disks. For the guest OS (and iometer) they appeared as raw disks.

But, if you were to create a file system for a particular application, my suggestion would be to go with the best practices suggested by Microsoft for particular applications. For e.g., MS recommends 64K NTFS block size for SQL server files.

Hope that helps.

Reply
0 Kudos
admin
Immortal
Immortal

You choose different block sizes depending on the size of the largest file (vmdks) on the VMFS datastore. For e.g., with 1MB block size, maximum size for a single vmdk is 256GB. The limit increases as you increase the block size.

vScsiStats and esxtop are very popular. You can also use tools provided by storage vendors (such as Navisphere Analyzer from EMC) to complement esxtop data.

Reply
0 Kudos
Gabrie1
Commander
Commander

You choose different block sizes depending on the size of the largest file (vmdks) on the VMFS datastore. For e.g., with 1MB block size, maximum size for a single vmdk is 256GB. The limit increases as you increase the block size.

vScsiStats and esxtop are very popular. You can also use tools provided by storage vendors (such as Navisphere Analyzer from EMC) to complement esxtop data.

I understand about the blocksize and the 256GB limit for a VMDK on a 1MB blocksize VMFS. But why shouldn't I just create only 8MB blocked VMFS in the future? Why is it better to create 1MB block size VMFS when you don't need VMDK bigger then 256GB?

http://www.GabesVirtualWorld.com
Reply
0 Kudos
admin
Immortal
Immortal

There is no reason why you shouldn't use 8MB block size instead of 1MB. In case of 'thin' vmdk format unit of space allocation is the minimum block size used for VMFS. As the file grows space is allocated in chunks of the chosen block size. If the application has periods of bursty writes (which causes file growth) followed by period of inactivity, this may lead to some wastage in space. This may not ber very significant compared to the acual file size, but just something to keep in mind.

Reply
0 Kudos
Gabrie1
Commander
Commander

Thx !!!

http://www.GabesVirtualWorld.com
Reply
0 Kudos
jasonboche
Immortal
Immortal

The virtual disks were created using the VI client, which automatically aligns the vmdk files.

Not in my experience.

The VI Client will align VMFS volumes at creation time, but not the guest OS in the .vmdk.

I've been aligning a lot of .vmdk's lately that were created with VI3.

Would like clarification here.

As far as guest OS is concerned, no file system was created on these virtual disks. For the guest OS (and iometer) they appeared as raw disks.

But, if you were to create a file system for a particular application, my suggestion would be to go with the best practices suggested by Microsoft for particular applications. For e.g., MS recommends 64K NTFS block size for SQL server files.

Hope that helps.






[i]Jason Boche, vExpert[/i]

[boche.net - VMware Virtualization Evangelist|http://www.boche.net/blog/][/i]

[VMware Communities User Moderator|http://www.vmware.com/communities/content/community_terms/][/i]

[Minneapolis Area VMware User Group Leader|http://communities.vmware.com/community/vmug/us-central/minneapolis][/i]

VCDX3 #34, VCDX4, VCDX5, VCAP4-DCA #14, VCAP4-DCD #35, VCAP5-DCD, VCPx4, vEXPERTx4, MCSEx3, MCSAx2, MCP, CCAx2, A+
Reply
0 Kudos
depping
Leadership
Leadership

I wrote an article about it a while ago. Especially with the new VMFS Grow feature it makes sense to always use 8MB Blocks. If you do you will always have the option to create the largest file possible. And indeed for Thin Provisioning you will grow in increments of 8MB vs 1MB which should lead to less locking and less growing. The overhead should be minumum. (max = 7MB x total amount of VMs)

http://www.yellow-bricks.com/2009/05/14/block-sizes-and-growing-your-vmfs/

Duncan

VMware Communities User Moderator | VCP | VCDX

-


Blogging: http://www.yellow-bricks.com

Twitter:

If you find this information useful, please award points for "correct" or "helpful".

Reply
0 Kudos
dconvery
Champion
Champion

Chethank -

I would like a clarification on alignment as well. What I am hearing you say is to create VMFS in the VIC and it would be automatically aligned (I already knew that part). I am not sure I understand if the guest needs anything special to aling from default setting. Are you saying that the guest will be aligned as well? or is there something special I need to do within the guest?

Dave Convery

VMware vExpert 2009

Careful. We don't want to learn from this.

Bill Watterson, "Calvin and Hobbes"

Dave Convery, VCDX-DCV #20 ** http://www.tech-tap.com ** http://twitter.com/dconvery ** "Careful. We don't want to learn from this." -Bill Watterson, "Calvin and Hobbes"
Reply
0 Kudos
depping
Leadership
Leadership

I think he only stated that he did not need to allign the additional disk cause IOMeter would access the raw disk without it being formatted with a file system?

When a it's formatted with a filesystem it will need to be alligned depending on the filesystem used.

Duncan

VMware Communities User Moderator | VCP | VCDX

-


Blogging:

Twitter:

If you find this information useful, please award points for "correct" or "helpful".

Reply
0 Kudos
IRivera
Contributor
Contributor

I think you'll find this article very helpful on aligning Windows .vmdk

http://www.vmware.com/pdf/esx3_partition_align.pdf

Reply
0 Kudos
dconvery
Champion
Champion

Yeah, I think I was reading into Jason's questions too much. I use the steps outlined in the partition alignment whitepaper for the guests right now.

Let me ask this: Is there an easy way to align the C: drive on a windows system? I have gone through the process using a helper VM to partition and format the VMDK and then install the OS on the newly-formatted disk, but there must be a better way. Is it really necessary?

Dave Convery

VMware vExpert 2009

Careful. We don't want to learn from this.

Bill Watterson, "Calvin and Hobbes"

Dave Convery, VCDX-DCV #20 ** http://www.tech-tap.com ** http://twitter.com/dconvery ** "Careful. We don't want to learn from this." -Bill Watterson, "Calvin and Hobbes"
Reply
0 Kudos
admin
Immortal
Immortal

CLARIFICATION: As some of the readers have pointed out, creation of a VMFS datastore from VI client automatically aligns VMFS on a LUN. And yes, this is what I did too. But this doesn't align a file system created on a virtual disk (.vmdk) in the guest OS. That has to be done manually. This document has some helpful information on alignment: . Sorry for the confusion in my earlier response.

>> As far as guest OS is concerned, no file system was created on these virtual disks. For the guest OS (and iometer) they appeared as raw disks.

In our case, since no file system was created on the virtual disk in the guest OS, the virtual disks didn't require any alignment.

Hope the clarification helps.

Reply
0 Kudos
dconvery
Champion
Champion

Is there an easy way to align the C: drive on a windows system? I have

gone through the process using a helper VM to partition and format the

VMDK and then install the OS on the newly-formatted disk, but there

must be a better way. Is it really necessary?

Dave Convery

VMware vExpert 2009

http://www.dailyhypervisor.com

Careful. We don't want to learn from this.

Bill Watterson, "Calvin and Hobbes"

Dave Convery, VCDX-DCV #20 ** http://www.tech-tap.com ** http://twitter.com/dconvery ** "Careful. We don't want to learn from this." -Bill Watterson, "Calvin and Hobbes"
Reply
0 Kudos
admin
Immortal
Immortal

CLARIFICATION: As some of the readers have pointed out, creation of a VMFS datastore from VI client automatically aligns VMFS on a LUN. And yes, this is what I did too. Sorry for the confusion in my earlier response.

But creation of a partition on a virtual disk (.vmdk) inside a guest OS doesn't align the partition. This has to be done manually (diskpart in windows; fdisk in linux). This article has some helpful information on alignment:

>> As far as guest OS is concerned, no file system was created on these virtualdisks. For the guest OS (and iometer) they appeared as raw disks.

In our case, since we didn't create any file system on the virtual disks in the guest, we didn't need to align the .vmdks.

Hope it is clearer now.

Reply
0 Kudos
kcollo
Contributor
Contributor

I actually have nothing to add, but wanted to at least say great thread! Should make all people looking to post to the communities read this first. Will solve a lot of issues and have good information concearning the typical IO testing for vmware (using raw storage with iometer). Also, still a lot of people do not know about alignment issues with vmfs. Again, this would be a great intro post for VMware and storage Smiley Happy

Kevin Goodman

kevin@colovirt.com

http://blog.colovirt.com

Reply
0 Kudos
jasonboche
Immortal
Immortal

Is there an easy way to align the C: drive on a windows system? I have

gone through the process using a helper VM to partition and format the

VMDK and then install the OS on the newly-formatted disk, but there

must be a better way. Is it really necessary?

Yes. NetApp mbrscan will scan your .vmdk file(s) to see if they need alignment and mbralign will perform the alignment on both Windows and Linux VMs. The utilities run from the command line on an ESX host. I use the tools on both NFS and block storage and it works well. There's a caveat with Linux that I'll tell you about if you have any Linux guest VMs.

Jas






[i]Jason Boche, vExpert[/i]

[boche.net - VMware Virtualization Evangelist|http://www.boche.net/blog/][/i]

[VMware Communities User Moderator|http://www.vmware.com/communities/content/community_terms/][/i]

[Minneapolis Area VMware User Group Leader|http://communities.vmware.com/community/vmug/us-central/minneapolis][/i]

VCDX3 #34, VCDX4, VCDX5, VCAP4-DCA #14, VCAP4-DCD #35, VCAP5-DCD, VCPx4, vEXPERTx4, MCSEx3, MCSAx2, MCP, CCAx2, A+
Reply
0 Kudos
depping
Leadership
Leadership

there's no point in aligning your C:\ drive unless you have a lot of data traffic on the C:\ which is unlikely. The mbr-tools by NetApp could do this indeed.

Duncan

VMware Communities User Moderator | VCP | VCDX

-


Blogging:

Twitter:

If you find this information useful, please award points for "correct" or "helpful".

Reply
0 Kudos