VMware Communities: Message List - I/O performance of vSphere

Re: I/O performance of vSphere

chethank — Thu, 09 Jul 2009 17:23:45 GMT

To support the the number of I/O operations we generated we required large amount of I/O bandwidth which meant multiple I/O paths. Since we didn't have FC switches available at our disposal, we had to use direct links. We kept on adding I/O paths until we hit 350,000 number. At this stage we had twelve 4Gbps FC links and for that we needed 3 arrays with 4 FC ports each. Hence 3 arrays.

I am sure EMC has sizing guidelines for designing a storage infrastructure based on I/O requirements. You can check EMC website or talk to a local EMC rep.

Chethan

Re: I/O performance of vSphere

smalldust — Wed, 08 Jul 2009 04:27:04 GMT

Hi,

I noticed that in the blog article named "350,000 I/O operations per Second, One vSphere Host",you said

Instead only 30 EFDs housed in three CX4-960 arrays provided enough
storage bandwidth for vSphere to drive just above 350,000 I/O requests per
second.

I am wondering why you need 3 boxes and hope there is a way to calculate how many storage arrays are required for a certain IOPS.

Is there any data available on EMC web site?

Thank you.

Re: I/O performance of vSphere

dconvery — Mon, 08 Jun 2009 22:18:47 GMT

So...Since VI3 anly aligns VMFS when created via the VIC, how about ESX4? This requires a VMFS on install for the COS VM. Is it aligned?

Dave Convery
VMware vExpert 2009
http://www.dailyhypervisor.com
http://twitter.com/dconvery

Careful. We don't want to learn from this.
Bill Watterson, "Calvin and Hobbes"

Re: I/O performance of vSphere

za_mkh — Mon, 08 Jun 2009 13:54:50 GMT

Thanks Harley,

That makes sense, and means it our workload will be made us. Since we
are re architecting our Virtual Infrastructure, this comes at a good
time!

Many thanks once again
za_mkh

Re: I/O performance of vSphere

hstagner — Mon, 08 Jun 2009 12:39:59 GMT

Hello za_mkh,

I did write an article on this in January.

http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1344730_mem1,00.html

Basically, I used a WinPE disk to align a system partition, then I created a template. Every VM cloned from that template will be aligned.

However: Now I question whether this should be done. As I said above, this note is in the Disk Alignment whitepaper from VMware:

Note: Aligning the boot disk in the virtual machine is neither recommended nor required.
* Align only the data disks in the virtual machine.*

I hope this helps.

Don't forget to use the buttons on the side to award points if you found this useful (you'll get points too).

Regards,

Harley Stagner

Re: I/O performance of vSphere

za_mkh — Mon, 08 Jun 2009 11:02:41 GMT

Great discussion, especially as we are busy implementing our vmware infrastructure onto our new SAN.

I do have a question/solution... Since it is a tedious process to align the system drive on a Win 2K3 / etc box, would the following scenario be valid:

1) In an existing VM - add a new VMDK (say 15GB) and align this disk as per normal - I don't format it.

2) I then use this VMDK as the system drive for a new VM and configure/format as necessary. I then convert this VM to a Template e.g. Windows 2003 Std Template

3) Would all subsequent VM's cloned from this new template also have their drives aligned, or what I need to go down the knoppix route to solve this

If the above solution does work, then, it could be an easy fix for new VM's created but we would still have to go through the pain for the existing VM's,

Re: I/O performance of vSphere

cxo — Fri, 05 Jun 2009 14:37:00 GMT

Reading the instructions for MBRALIGN, it notes with Linux and Solaris running GRUB there may be issues on bootup. The instructions are quite clear on how to remedy this for Linux (as well as documented here), but they leave out the analogous step-by-step instructions for Solaris.

So, the question I have is, has anyone used MBRALIGN on a Solaris 10 VM and what were the caviates and solutions to such issues?

Charlie

Re: I/O performance of vSphere

hstagner — Sat, 30 May 2009 17:35:35 GMT

I have aligned the system partition (and created a template from an aligned VM) in the past as well. I have even written an article on it. Then it occured to me after reading the VMware whitepaper again that I must be missing something. Why does this question (how to align C: drives) keep coming up when the VMware whitepaper on partition alignment has this note:

Note: Aligning the boot disk in the virtual machine is neither recommended nor required.
Align only the data disks in the virtual machine.
The following sections discuss how to align guest operating system partitions in Linux and
Windows environments.

What am I missing guys? I found another thread (can't find it at the moment) claiming that aligned C: drives may cause problems with VSS on certain arrays. Is this true? Does anyone know what arrays have this issue? Should the best practice going forward be to not align the C: drive at all because of the note in the VMware whitepaper? I know I have more questions than answers. I am just trying to get to the bottom of this.

Don't forget to use the buttons on the side to award points if you found this useful (you'll get points too).

Regards,

Harley Stagner

Re: I/O performance of vSphere

LucasAlbers — Sun, 31 May 2009 05:27:15 GMT

Windows 2008 automatically aligns the drive starting at a 1024 k offset.
Aligning windows 2003 was tedious. Set alignment with diskpart, format.
Install and use existing file system.

Re: I/O performance of vSphere

dconvery — Sat, 30 May 2009 13:59:08 GMT

VERY NICE! Thanks Jas. I always forget about Knoppix (I have the latest DVD image) and always revert to DSL or a RHEL rescue CD.

Dave Convery
VMware vExpert 2009
http://www.dailyhypervisor.com

Careful. We don't want to learn from this.
Bill Watterson, "Calvin and Hobbes"

Re: I/O performance of vSphere

jasonboche — Sat, 30 May 2009 13:54:45 GMT

dconvery wrote:

Jas -

Give it up! What's the caveat for Linux? I have some customers with sizeable Linux environments too. Also, is mbrscan a freeware tool? I don't work for a NetApp partner right now.

Sometimes (most of the time) after running NetApp's MBRALIGN on Linux VMs, the VM is no longer bootable. It will immediately hang at a GRUB prompt. There is a repair procedure that involves booting from a Knoppix CD and running a few commands to repair the boot loader of the affected Linux VM. I've been through it dozens of time. Once the boot loader is repaired, the Linux VM will boot and its partitions will be aligned. I have seen rare cases where the fix does not actually work and you have no choice but to revert to the backed up .vmdk files that mbralign automatically creates for you. At that point, you can try the align process again or give up. The align process takes a while and it varies upon how large each .vmdk file is of course. On average what I see is an alignment of a 50GB .vmdk file in half an hour or less.

Once the knoppix CDROM has booted, From the 'boot>' prompt type 'knoppix 2' and hit RETURN

From the Command Line, type 'grub' to get to the grub prompt

Run "find /boot/grub/stage1" and note all of the drives it finds (e.g., "(hd0,0)")

From the GRUB prompt, for each Drive, Run the following:

grub> find /boot/grub/stage1
(hd0,0)
(hd1,0)
(hd2,0)

grub> root (hd0,0)
Filesystem type is ext2fs, partition type 0x83

grub> setup (hd0)
Checking if "/boot/grub/stage1" exists... yes
Checking if "/boot/grub/stage2" exists... yes
Checking if "/boot/grub/e2fs_stage1_5" exists... yes
Running "embed /boot/grub/e2fs_stage1_5 (hd0)"... 15 sectors are embedded.
succeeded
Running "install /boot/grub/stage1 (hd0) (hd0)1+15 p (hd0,0)/boot/grub/stage2 /boot/grub/menu.lst"... succeeded
Done.

grub>

You can batch process multiple boot drives, just attach all of the drives you wish to fix to a dedicated knoppix appliance that boots from the Knoppix CD.

You should be able to download the NetApp tools. Just go create yourself a now.netapp.com account and download the mbrscan and mbralign tools
http://communities.netapp.com/docs/DOC-2563;jsessionid=3FAC4EB6245FD8344D98EA9247C2FE34

Don't forget that MBRALIGN creates backups of your .vmdk files which will chew up double the amount of storage you use so go back and delete those backup files once you've determined the alignment is a success.

Duncan -

That's what I thought. If your data is on an aligned drive, ther is probably no reason to worry about the system drive, but I wanted some other opinions as well.

I align all drives as a best practice. Not as a performance benefit for the individual VM, but as a performance benefit for the storage array that all VMs point back to. Disk alignment converts all unaligned disk I/Os from a maximum possible value of 2 IOs to a value of 1 IO. If you multiply that factor by hundreds or thousands you start to see a little performance increase for the VMs. Now multiple that savings by X number of VMs on each LUN, disk group, etc. and you might see how aligning C: drives collectively improves the maximum amount of performance you can squeeze out of that LUN, disk group, storage array, etc. This is an example of where the value of the savings is greater than the sum of all of its parts.

Dave Convery
VMware vExpert 2009
http://www.dailyhypervisor.com

Careful. We don't want to learn from this.
Bill Watterson, "Calvin and Hobbes"

Jason Boche, vExpert
boche.net - VMware Virtualization Evangelist
VMware Communities User Moderator
Minneapolis Area VMware User Group Leader

Re: I/O performance of vSphere

dconvery — Sat, 30 May 2009 11:30:25 GMT

Jas -

Give it up! What's the caveat for Linux? I have some customers with sizeable Linux environments too. Also, is mbrscan a freeware tool? I don't work for a NetApp partner right now.

Duncan -

That's what I thought. If your data is on an aligned drive, ther is probably no reason to worry about the system drive, but I wanted some other opinions as well.

Dave Convery
VMware vExpert 2009
http://www.dailyhypervisor.com

Careful. We don't want to learn from this.
Bill Watterson, "Calvin and Hobbes"

Re: I/O performance of vSphere

depping — Sat, 30 May 2009 07:04:53 GMT

there's no point in aligning your C:\ drive unless you have a lot of data traffic on the C:\ which is unlikely. The mbr-tools by NetApp could do this indeed.

Duncan
VMware Communities User Moderator | VCP | VCDX

Blogging: http://www.yellow-bricks.com
Twitter: http://www.twitter.com/depping

If you find this information useful, please award points for "correct" or "helpful".

Re: I/O performance of vSphere

jasonboche — Sat, 30 May 2009 03:17:31 GMT

dconvery wrote:
Is there an easy way to align the C: drive on a windows system? I have
gone through the process using a helper VM to partition and format the
VMDK and then install the OS on the newly-formatted disk, but there
must be a better way. Is it really necessary?

Yes. NetApp mbrscan will scan your .vmdk file(s) to see if they need alignment and mbralign will perform the alignment on both Windows and Linux VMs. The utilities run from the command line on an ESX host. I use the tools on both NFS and block storage and it works well. There's a caveat with Linux that I'll tell you about if you have any Linux guest VMs.

Jas

Jason Boche, vExpert
boche.net - VMware Virtualization Evangelist
VMware Communities User Moderator
Minneapolis Area VMware User Group Leader

Re: I/O performance of vSphere

kcollo — Fri, 29 May 2009 18:47:13 GMT

I actually have nothing to add, but wanted to at least say great thread! Should make all people looking to post to the communities read this first. Will solve a lot of issues and have good information concearning the typical IO testing for vmware (using raw storage with iometer). Also, still a lot of people do not know about alignment issues with vmfs. Again, this would be a great intro post for VMware and storage

Kevin Goodman
kevin@colovirt.com
http://blog.colovirt.com

Re: I/O performance of vSphere

chethank — Fri, 29 May 2009 18:04:00 GMT

CLARIFICATION: As some of the readers have pointed out, creation of a VMFS datastore from VI client automatically aligns VMFS on a LUN. And yes, this is what I did too. Sorry for the confusion in my earlier response.

But creation of a partition on a virtual disk (.vmdk) inside a guest OS doesn't align the partition. This has to be done manually (diskpart in windows; fdisk in linux). This article has some helpful information on alignment: http://www.vmware.com/pdf/esx3_partition_align.pdf

As far as guest OS is concerned, no file system was created on these virtualdisks. For the guest OS (and iometer) they appeared as raw disks.

In our case, since we didn't create any file system on the virtual disks in the guest, we didn't need to align the .vmdks.

Hope it is clearer now.

Re: I/O performance of vSphere

dconvery — Fri, 29 May 2009 18:03:28 GMT

Is there an easy way to align the C: drive on a windows system? I have
gone through the process using a helper VM to partition and format the
VMDK and then install the OS on the newly-formatted disk, but there
must be a better way. Is it really necessary?

Dave Convery
VMware vExpert 2009
http://www.dailyhypervisor.com

Careful. We don't want to learn from this.
Bill Watterson, "Calvin and Hobbes"

Re: I/O performance of vSphere

chethank — Fri, 29 May 2009 17:56:26 GMT

CLARIFICATION: As some of the readers have pointed out, creation of a VMFS datastore from VI client automatically aligns VMFS on a LUN. And yes, this is what I did too. But this doesn't align a file system created on a virtual disk (.vmdk) in the guest OS. That has to be done manually. This document has some helpful information on alignment: http://www.vmware.com/pdf/esx3_partition_align.pdf . Sorry for the confusion in my earlier response.

As far as guest OS is concerned, no file system was created on these virtual disks. For the guest OS (and iometer) they appeared as raw disks.

In our case, since no file system was created on the virtual disk in the guest OS, the virtual disks didn't require any alignment.

Hope the clarification helps.

Re: I/O performance of vSphere

dconvery — Fri, 29 May 2009 13:09:36 GMT

Yeah, I think I was reading into Jason's questions too much. I use the steps outlined in the partition alignment whitepaper for the guests right now.

Let me ask this: Is there an easy way to align the C: drive on a windows system? I have gone through the process using a helper VM to partition and format the VMDK and then install the OS on the newly-formatted disk, but there must be a better way. Is it really necessary?

Dave Convery
VMware vExpert 2009
http://www.dailyhypervisor.com

Careful. We don't want to learn from this.
Bill Watterson, "Calvin and Hobbes"

Re: I/O performance of vSphere

IRivera — Fri, 29 May 2009 13:00:40 GMT

I think you'll find this article very helpful on aligning Windows .vmdk

http://www.vmware.com/pdf/esx3_partition_align.pdf

Re: I/O performance of vSphere

depping — Fri, 29 May 2009 12:47:28 GMT

I think he only stated that he did not need to allign the additional disk cause IOMeter would access the raw disk without it being formatted with a file system?

When a it's formatted with a filesystem it will need to be alligned depending on the filesystem used.

Duncan
VMware Communities User Moderator | VCP | VCDX

Blogging: http://www.yellow-bricks.com
Twitter: http://www.twitter.com/depping

If you find this information useful, please award points for "correct" or "helpful".

Re: I/O performance of vSphere

dconvery — Fri, 29 May 2009 12:39:47 GMT

Chethank -

I would like a clarification on alignment as well. What I am hearing you say is to create VMFS in the VIC and it would be automatically aligned (I already knew that part). I am not sure I understand if the guest needs anything special to aling from default setting. Are you saying that the guest will be aligned as well? or is there something special I need to do within the guest?

Dave Convery
VMware vExpert 2009
http://www.dailyhypervisor.com

Careful. We don't want to learn from this.
Bill Watterson, "Calvin and Hobbes"

Re: I/O performance of vSphere

depping — Fri, 29 May 2009 06:10:48 GMT

I wrote an article about it a while ago. Especially with the new VMFS Grow feature it makes sense to always use 8MB Blocks. If you do you will always have the option to create the largest file possible. And indeed for Thin Provisioning you will grow in increments of 8MB vs 1MB which should lead to less locking and less growing. The overhead should be minumum. (max = 7MB x total amount of VMs)

http://www.yellow-bricks.com/2009/05/14/block-sizes-and-growing-your-vmfs/

Duncan
VMware Communities User Moderator | VCP | VCDX

Blogging: http://www.yellow-bricks.com
Twitter: http://www.twitter.com/depping

If you find this information useful, please award points for "correct" or "helpful".

Re: I/O performance of vSphere

jasonboche — Fri, 29 May 2009 06:08:45 GMT

chethank wrote:

The virtual disks were created using the VI client, which automatically aligns the vmdk files.

Not in my experience.
The VI Client will align VMFS volumes at creation time, but not the guest OS in the .vmdk.
I've been aligning a lot of .vmdk's lately that were created with VI3.
Would like clarification here.

As far as guest OS is concerned, no file system was created on these virtual disks. For the guest OS (and iometer) they appeared as raw disks.

But, if you were to create a file system for a particular application, my suggestion would be to go with the best practices suggested by Microsoft for particular applications. For e.g., MS recommends 64K NTFS block size for SQL server files.

Hope that helps.

Jason Boche, vExpert
boche.net - VMware Virtualization Evangelist
VMware Communities User Moderator
Minneapolis Area VMware User Group Leader

Re: I/O performance of vSphere

Gabrie — Fri, 29 May 2009 06:02:53 GMT

Thx

http://www.GabesVirtualWorld.com

Re: I/O performance of vSphere

chethank — Fri, 29 May 2009 00:47:06 GMT

There is no reason why you shouldn't use 8MB block size instead of 1MB. In case of 'thin' vmdk format unit of space allocation is the minimum block size used for VMFS. As the file grows space is allocated in chunks of the chosen block size. If the application has periods of bursty writes (which causes file growth) followed by period of inactivity, this may lead to some wastage in space. This may not ber very significant compared to the acual file size, but just something to keep in mind.

Re: I/O performance of vSphere

Gabrie — Wed, 27 May 2009 18:52:05 GMT

chethank wrote:
You choose different block sizes depending on the size of the largest file (vmdks) on the VMFS datastore. For e.g., with 1MB block size, maximum size for a single vmdk is 256GB. The limit increases as you increase the block size.

vScsiStats and esxtop are very popular. You can also use tools provided by storage vendors (such as Navisphere Analyzer from EMC) to complement esxtop data.

I understand about the blocksize and the 256GB limit for a VMDK on a 1MB blocksize VMFS. But why shouldn't I just create only 8MB blocked VMFS in the future? Why is it better to create 1MB block size VMFS when you don't need VMDK bigger then 256GB?

http://www.GabesVirtualWorld.com

Re: I/O performance of vSphere

chethank — Wed, 27 May 2009 17:51:24 GMT

You choose different block sizes depending on the size of the largest file (vmdks) on the VMFS datastore. For e.g., with 1MB block size, maximum size for a single vmdk is 256GB. The limit increases as you increase the block size.

vScsiStats and esxtop are very popular. You can also use tools provided by storage vendors (such as Navisphere Analyzer from EMC) to complement esxtop data.

Re: I/O performance of vSphere

chethank — Wed, 27 May 2009 17:34:13 GMT

The virtual disks were created using the VI client, which automatically aligns the vmdk files. As far as guest OS is concerned, no file system was created on these virtual disks. For the guest OS (and iometer) they appeared as raw disks.

Hope that helps.

Re: I/O performance of vSphere

Gabrie — Wed, 27 May 2009 16:40:16 GMT

Q1: A while ago I saw a discussion on VMFS blocksize. I learned that VMFS blocksize doesnot matter when Windows Guest is reading for example a 64kb block from vmdk on vmfs. So my question is, when does the blocksize matter if it has no influence on performance and why should I choose between 1 or 8MB blocksize?

Q2: What is the best way to track performance from Guest disk through Guest SCSI controller through Host HBA Queue depth to LUN Queue depth? Only vScsiStats and esxtop? Or other tools aswell?

http://www.GabesVirtualWorld.com

Re: I/O performance of vSphere

rDale — Tue, 26 May 2009 21:43:14 GMT

I have a question in order to get tose kinds of IOPS did you do any disk alignment with the vmfs vol or the vmdk files ?

I often hear alot of blah about disk alignment being needed for SQL servers and Exchnage servers.

what block size did you work with on the ntfs partions ?

I/O performance of vSphere

chethank — Tue, 26 May 2009 18:55:57 GMT

I am a senior performance engineer at VMware. I have spent quite a bit of time on performance analysis of ESX storage stack starting from ESX 3.0. We at VMware are constantly working on improving I/O performance of ESX. I have done few experiments to drive extremely high I/O load on single instance of ESX. With 3.5, I obtained 100,000 I/OPs with an I/O load that is most representative of real applications (100,000 I/Ops) until I ran out of hardware. The performance envelope was pushed furthe with vSphere when we achieved 350,000 I/Ops in an experiment done at EMC labs. I wrote a blog highlighting the results with some details on the experiments in VMware's performance blog VROOM.

Having a well configured I/O system is critical for good application performance. Very often I hear questions from customers on storage performance, choice of virtual disk format - VMFS vs RDM, best practices etc., The answer is simple - follow the best practices that you would normally follow in native world when designing an I/O infrastructure for your application. ESX provides excellent I/O performance and can support even extreme I/O demands from applications as the results discussed in the blogs indicate. VMFS or RDM - you can expect similar performance though RDM can help during certain scenarios which are purely non-performance related.

We can discuss more in this thread. Feel free to post your questions, comments on the blogs or any I/O related issues on this thread. I will try my best to respond. May be some one who has already faced a similar situation will jump in with a solution even we at VMware wouldn't have thought of!

Chethan