jrmunday
Commander
Commander

Disk Alignment on guest VM's

Hi All,

We have a fair amount of legacy Windows 2003 servers with misaligned disks in our environment and I would like to resolve this to ensure that we are getting the best out of our storage, and not sufferring unnecessary performance degredation with something that can easily be avoided.

I did some testing to see how much of a difference IO alignment makes, and since the results are pretty significant, I would like to deal with this ASAP.

See results below;

Aligned-IO.png

So my questions are;

  1. What is the best way to programatically go through all of the VM's and check for disk alignment - I don't want any manual process (if possible).
  2. What are the options for aligning the misaligned disks?
  3. What is the impact of each option (ie. How long does it take, Is Downtime required, etc.)
  4. What are the limitations and / or known issues in doing such remediation tasks.

Thanks in advance,

Jon

vExpert 2014 - 2022 | VCP6-DCV | http://www.jonmunday.net | @JonMunday77
0 Kudos
29 Replies

Jon Munday wrote:

Hi All,

We have a fair amount of legacy Windows 2003 servers with misaligned disks in our environment and I would like to resolve this to ensure that we are getting the best out of our storage, and not sufferring unnecessary performance degredation with something that can easily be avoided.

I did some testing to see how much of a difference IO alignment makes, and since the results are pretty significant, I would like to deal with this ASAP.

See results below;

24227_24227.pngAligned-IO.png

So my questions are;

  1. What is the best way to programatically go through all of the VM's and check for disk alignment - I don't want any manual process (if possible).
  2. What are the options for aligning the misaligned disks?
  3. What is the impact of each option (ie. How long does it take, Is Downtime required, etc.)
  4. What are the limitations and / or known issues in doing such remediation tasks.

Thanks in advance,

Jon

1 best way to check woud either be locally like diskpart,  or use a tool that your san has.

Im sure you might be able to script something with the msinfo data

How to check Drive alignment in windows:

  1. run “System Information” (msinfo32.exe in Start-> Run)
  2. and select Components .. Storage .. Disks
  3. scroll to the bottom and you will see the Partition Starting Offset information.
  4. This number needs to be perfectly divisible by 4096.
  5. The default .vmdk, you will see the Partition Starting Offset set to 32,256  (32,256 / 4096 = 7.875)
  6. and thus this file system is not correctly aligned.

http://theregime.wordpress.com/2009/07/20/how-to-align-windows-disks-in-vmware-with-netapp-disk/

2: options are v2v using vmware converter, or valigner.

vmware converter makes a new vm,

vlaigner converts teh actual disk.

The problem with valigner is that if you ever extended the os or have 2 more more patitions on a disk it wont work

3:

since its really a disk conversion byte by byte they all take the same time as a clone

--------------------- Sparrowangelstechnology : Vmware lover http://sparrowangelstechnology.blogspot.com
0 Kudos
mcowger
Immortal
Immortal

Actually, it can be done more efficiently than NetApp's method:

http://nickapedia.com/2011/11/03/straighten-up-with-a-new-uber-tool-presenting-uberalign/

--Matt VCDX #52 blog.cowger.us
0 Kudos
jrmunday
Commander
Commander

Hey Matt, thanks for this ... it look like exactly what I'm after Smiley Happy

I'll test it out and provide feedback.

Cheers,

Jon

vExpert 2014 - 2022 | VCP6-DCV | http://www.jonmunday.net | @JonMunday77
0 Kudos
mcowger
Immortal
Immortal

Hope it helps.  If you have questions, its author (Nick Weaver) is a friend and quite active on twitter as well (@lynxbat).

--Matt VCDX #52 blog.cowger.us
jdptechnc
Expert
Expert

Hi,

For a quick check of starting partition offset values from Windows, I created a Powershell script that queries vCenter for the name of all Windows VM's, then uses WMI to query these servers for their starting offset values. I used 64KB as my multiplier - it flagged it as not correctly aligned if not divisible by 64KB.

Another tool you can use for correcting partition misalignment is VMware Converter Standalone (must at least 5.x version).  You can do a V2V to the same vCenter, and copy to the "destination" VM using volume based copy - there is an option to create an optimized layout which will align everything at 1024KB. This also has the advantage of being able to resize your disks if you are too much or too little space allocated to your windows drives, and you can move from a single vmdk with multiple partitions to multiple vmdk's, and vice versa, if you like.

Whichever option you go with, you will incur downtime as the VM will have to be offline.

Please consider marking as "helpful", if you find this post useful. Thanks!... IT Guy since 12/2000... Virtual since 10/2006... VCAP-DCA #2222
jrmunday
Commander
Commander

Thanks very much, this will save me writing a script Smiley Happy - I will test this tomorrow.

vExpert 2014 - 2022 | VCP6-DCV | http://www.jonmunday.net | @JonMunday77
0 Kudos
RParker
Immortal
Immortal

The bottom line is EACH VM has to be done, it's going to take a LONG time and its very tedious.. extended downtime PER VM.

So you can probably do 2 VM's per LUN (on Netapp) and maybe 4 concurrently on your SAN, but even with the script it copies the information in the VM to a new disk, and it takes around 1 minute per GB (estimated).

So take that into account.  I have done many of these... it's not fun, the best thing for you is to make sure the templates (all of them) are done first, that way ALL new VM's are deployed properly, and existing VM's you can tackle as needed.

I would start with Databases first, since they have the highest IO.  The high IO is where you will see the most dramatic affect.  Linux is higher priority than Windows, because Linux kills in poorly aligned VM's, but if you have Windows, some of that is diminished by caching... also do NOT underesestimate defrag the guests either.  Since they are not aligned, defrag will be even more crucial, but it will still be a factor once they are aligned.

This topic has been discussed to the nth degree, so I will not get into it, but since you are aligning disks, don't discount the important of a good defrag while you are at it.

Since each VM needs to be off.. not in snapshot mode, will not work... you should:

1) Run registry scan / fix on VM's

2) update / patch VM's as needed (including tools upgrade during this cycle)

3) shutdown the VM after steps 1 and 2

4) perform alignment on VM disks

5) defrag each disk / each VM.

Just my suggestion, but that will give you best performance.

0 Kudos
jrmunday
Commander
Commander

Thanks for the tips, I'll definitely factor this in. My first step is to see how many VM's are affected out of the ~650 windows servers and ~250 unix servers and then start planning to do the tedious work.

... I do have 20TB of Flash storage that I can abuse for this job, so can hopefully crunch through it Smiley Happy

vExpert 2014 - 2022 | VCP6-DCV | http://www.jonmunday.net | @JonMunday77
0 Kudos

Rparker,  why is defragging  needed/recommended, especially when its a vmdk on a san? From what I've read usually a defrag isnt needed.

"Typically you are going to have multiple VMs running together on a VMFS  or NFS volume. Therefore the overall I/O to the underlying LUN is going  to be random so defragmenting individual Guest OS'es is not really  going to help performance." -chogan

http://blogs.vmware.com/vsphere/2011/09/should-i-defrag-my-guest-os.html

--------------------- Sparrowangelstechnology : Vmware lover http://sparrowangelstechnology.blogspot.com
0 Kudos
jdptechnc
Expert
Expert

NetApp doesn't recommend defragging the guest at all, because their WAFL feature supposedly places each block in the most optimal location, and defrag can interfere with that.

Also, in general you have to be careful with defrag when you are working with thin provisioning, because you will have to provision more disk in order to execute the defrag. Other than that, I don't know of any other reason why you wouldn't defrag if Windows is reporting excessive fragmentation.  Defrag and the other tips that RParker provided should improve performance, biggest impact being when higher-IO vm's are corrected, of course.

Please consider marking as "helpful", if you find this post useful. Thanks!... IT Guy since 12/2000... Virtual since 10/2006... VCAP-DCA #2222
0 Kudos
meistermn
Expert
Expert

doesn't the vmware converter create a vm and so a new nic, which has a new mac address.  

0 Kudos
jdptechnc
Expert
Expert

Yes. It's a V2V so it would be a new VM object with a unique MAC address.  It tries to keep the same network configuration (static IP address, etc) but is not always successful in doing so.  Usually if you're chaging hardware version or type for virtual NIC being used, from what I have encountered.

You can change the MAC address back to the original MAC address if you have software that is sensitive to that (FlexNET licensing is one example).

Please consider marking as "helpful", if you find this post useful. Thanks!... IT Guy since 12/2000... Virtual since 10/2006... VCAP-DCA #2222
0 Kudos
EdWilts
Expert
Expert

Actually, it can be done more efficiently than NetApp's method:

NetApp has 2 methods.  You can use their mbralign utility which does work but requires the VM to be down.  It does not support any GPT disks so you have to watch for that.  Since all of our storage is NFS, we spun up a guest on the same jumbo-frame network as our datastores, mounted the datastores, and aligned from there.  It's not quick and requires enough free space in the datastore as the size of the VMs you're aligning - it basically reads from one file and creates a new one.  It's not quick but it's cheap and it almost always works.

The 2nd method is in the VSC and only works if you're using iSCSI disk (unless the most recent version addressed that).  What it does is create a purposely misaligned datastore which reversed the offset of the VMDKs.  This should be only used as a last resort since it's guaranteed to bite you later on.  If you put a Windows 2008 VM on this datastore - and they're aligned by default - you'll end up with split I/Os because the datastore is misaligned.

.../Ed (VCP4, VCP5)
0 Kudos
meistermn
Expert
Expert

okay it was some ago I tried with vconverter 5.0 beta , so it seems still no change. Maybe in the next verson of converter.

Here ar some informations about netapp alignment.

1. Top 10 session vmworld 2012

http://download3.vmware.com/vmworld/2012/top10/sto1430.pdf

2.) Netapp VSC  Online Alignment versus Realigment

Instead of realigning the virtual machine, Netapp is creating a new datastore specifically built to align itself with the unaligned virtual machine. These new datastores are considered “Functionally aligned” because they make the vm appear to be aligned.

This is not an alignment of the ntfs partition.

http://theithollow.com/tag/mbralign/

http://theithollow.com/2012/04/10/netapp-vsc4-optimization-and-migration/
http://www.vmadmin.info/2012/10/aligning-vms-via-netapp-vsc-41.html

3.) Keep in mind to take a backup of your vm before you hardcore realign the ntfs partitions. Microsoft does not give support.

Realignment needs downtime.

4.) If you buy a new storage maybe look at modern storage which are eventually not affected by this.

http://www.purestorage.com/blog/say-goodbye-to-vm-alignment-issues-and-poor-performance-with-pure-st...

http://blogs.vmware.com/vsphere/2012/08/interesting-storage-stuff-at-vmworld-2012-us-2.html

5.) powershell scripts to leverage UBERalign for virtual disk alignment

6. Orchstrator session

2049 How OCLC Used vCenter Orchestrator to Automate the Disk Alignment of 1500 + Virtual Machines

http://www.vcoteam.info/newsflash/vmworld-2011-help-to-reveal-vmware-best-kept-secret.html

0 Kudos
mcowger
Immortal
Immortal

MOST array vendors (including NetApp and EMC) don't recommend defrag, because it moves data around the array uncessarily, with no benefit, and can even harm the automatic tiering algorithms.

--Matt VCDX #52 blog.cowger.us
0 Kudos
meistermn
Expert
Expert

Why aren't there partitioning tools on the market ?

http://paragonsoftware.wordpress.com/

The task of migrating the operating system, applications and files from one storage drive to another can be a slow and tedious process. Paragon Migrate OS to SSD 2.0 simplifies the migration process by providing an intuitive wizard-driven interface. Paragon Migrate OS to SSD automatically downsizes the source system volume, if needed; auto-aligns the copied system partition; and provides intelligent selection of specific files and folders for migration — all without rebooting the system.

I do not understand the last part: all without rebooting the system.  Does this mean online realignment from hard disk to ssd. If so why not from vharddisk to vhardisk?

0 Kudos
Lessi001
Enthusiast
Enthusiast

Hi,

just my 2 cents...

We have the same situation - a lot of 2003 Servers which are misaligned...

As maintenance windows are rare using an alignment technology which leads to offline times was not suitable for us.

So we decided to use the NetApp VSC 4  Optimization and Migration Tool.

As "meistermn" wrote this is not a real alignment. How it works you can read here if you want (Part 3):

http://www.running-system.com/netapp-vsc-4-0-optimization-and-migration-tool-and-the-magic-of-optimi...

We have aligned about 50 misaligned VMs in about 8 hours work - without any interruption of our production.

Unfortunately we do not see any improvement of performance from our storage till now...

Regards

Andi

There are 10 types of people in this world. Those who understand binary, and those who do not.
0 Kudos
jdptechnc
Expert
Expert

meistermn wrote:

okay it was some ago I tried with vconverter 5.0 beta , so it seems still no change. Maybe in the next verson of converter.


You have to explicitly use the "select volumes to copy" option in order to expose the option to optimize layout.

We are on NetApp and saw the feature in VSC to create the offset datastore in order to align the I/O and not sustain downtime, but in the end, we decided against that.  It would tie those VM's to this special purpose datastore forever (and create a datastore that no properly aligned VM should use), and it doesn't really solve the problem.

I didn't like mbralign either, because it only aligned the VM at 32K, which is OK for netapp, but if we move these VMs to other vendor storage, 64K at least seems be the recommendation.

Please consider marking as "helpful", if you find this post useful. Thanks!... IT Guy since 12/2000... Virtual since 10/2006... VCAP-DCA #2222
0 Kudos
RParker
Immortal
Immortal

Very basic simple answer, as I said..this topic isn't new, it has been discussed, heated arguments.. for against, no one every wins.. but I will try to break it down.

WAFL is a block.  The VM resides as a block of data... It's no smaller than 4K.. there *ARE* Windows NTFS blocks within that block of data on the SAN.  *THAT's* what we are defragging.. not the SAN.

So Defrag WILL benefit.. Execsoft has proven it, and Netapp depending on the engineer, they agree or disagree depending on mood.  Even Netapp cannot get an agreement as a company.

Defrag ONLY runs when needed.. if your blocks are up to date, no need to KEEP defragging.  The more often you defrag, the less time it takes.  Perfect disk talks to vCenter to ensure that a moderate of VM's are running concurrently and it will not overrun the SAN (this can be adjusted).

So a block may be optimized on the SAN but it's not optimized within the Guest OS File system level.. that's what we are defragging.. FILES not blocks.  For every IOP that is required from Windows less on the SAN, saves more IOPS for the SAN.  So that's my take on defrag.. we are simply making files within the guest more efficient, and therefore less reads.

It's PROVEN benefit, regardless of what your netapp tech says.  I can find a Netapp tech that will disagree with HIM (or her).

0 Kudos