VMware Cloud Community
whynotq
Commander
Commander
Jump to solution

old favorite, ESX disk alignment

I posted this as a response on the Virtual Machine forum but really it belongs here and surely deserves a thread of its own. Let's here your views and interpretations.

OK, just for fun, I've found this in a EMC white paper published August 2006 so theory should be that it's based on ESX3. Seems to contradict the VMware documentation nicely:

Disk partition adjustment for VMWare.

VMware is more complex as there are two cases. When aligning raw disks or Raw Device Mapping (RDM) volumes, the alignment is done at the Virtual Machine (VM) level. For example, on Windows VMs use "diskpart" to perform the alignment.

For VMFS volumes, the alignment will be done at the ESX Server level using "fdisk" as well as at the VM level. This is because both the ESX Server and the clients will put MBRs on the LUNs.

The ESX must align the VMFS volume, and the client systems must align their virtual disks.

To align the ESX Server:

1. On service console, execute “fdisk /dev/sd is the

device on which you would like to create the VMFS

2. Type “n” to create a new partition

3. Type “p” to create a primary partition

4. Type “1” to create partition #1

5. Select the defaults to use the complete disk

6. Type “x” to get into expert mode

7. Type “b” to specify the starting block for partitions

8. Type “1” to select partition #1

9. Type “128” to make partition #1 to align on 32K boundary

10.Type “r” to return to main menu

11.Type “t” to change partition type

12.Type “1” to select partition 1

13.Type “fb” to set type to fb (VMFS volume)

14.Type “w” to write label and the partition information to disk

By declaring the partition type as "fb", ESX Server will recognize the partition as an unformatted VMFS volume. You should be able to put a VMFS file system on it using the MUI or "vmkfstools".

Next, the virtual disks for each VM must be aligned.

For Linux VMs, follow the procedure listed above. For Windows VMs, use the procedure for Windows, above.

so the question seems to be, who do we listen to?

VMware say it's aligned so no need to perform any further alignment.

EMC say it's offset so not only do we need to align to the ESX but also to the VM as well.......

Reply
0 Kudos
1 Solution

Accepted Solutions
bganesha
Enthusiast
Enthusiast
Jump to solution

Hello,

One of my friends in VMware engineering who works on VMFS pointed this to me so I decided to respond. I am part of Symmetrix engineering.

VMware ESX 3.0 will automatically align partitions on 64 KB boundary for all partitions created on SAN attached disks. The procedure that was described earlier in this thread is something that I had listed in my best practices presentation for ESX 2.x. This procedure \*does not* apply to ESX 3.0 environments. The updated EMC solution guides for ESX 3.0 (to be released end of this year or early next year) will list this difference.

Do note that the virtual disks are still created with a geometry of 63 sectors per head. So all the benefits of aligned VMFS will be lost if the partitions created in the virtual disks is not aligned. So we do recommend that you align partitions for virtual disks that would be used as data disks (not boot or applications disks).

I would like to take this opportunity to point out something that was mentioned in this thread about performance, ESX and alignment. The author is correct in his observation that he has not experienced performance problems attributed to alignment.

The best practices recommendation for partition alignment is based on experience. When Microsoft applications were initially deployed on the SAN a few years back customers did not experience any performance problems due to misaligned IOs. In fact, a number of them ignored the recommendation. However, within a short time, that changed due to the growth in the applications, changing criticality of the applications and increased compute power and maturing of the Windows OS. A number of customers experienced a lot of pain migrating from non-aligned partitions to aligned partitions. We believe something similar is very likely in the ESX environment.

A quick comparison of the capabilities of ESX 2.x and ESX 3.0 clearly indicates how rapidly the virtual environment is changing. Coupled this with the fact that a number of customers who have decided to standardize on virtual infrastructure as the de facto standard and are considering deployment of decent size databases on virtual infrastructure, the reason for the best practices recommendation is a lot more obvious. Hope this helps.

Bala

View solution in original post

Reply
0 Kudos
25 Replies
boydd
Champion
Champion
Jump to solution

I think that the parent company of VMware is a little confused here. A VM using RDM would have to (In theory) set the offset. More documentation from EMC states that you only have to do this for Sym and not CX.

Personally - I don't do it. Have never seen any performance issues with ESX and SYM or CX. Windows we do set the offset (Exchange).

DB

DB
Reply
0 Kudos
whynotq
Commander
Commander
Jump to solution

the extract in my post is from the Clariion Flare 22 Best Practices, so it's as up to date as it comes really. I've gotta say, it's mighty confusing.

Perhaps we can get this post to someone who can clarify the "best practices" for once and for all....

Message was edited by:

whynotq

Reply
0 Kudos
admin
Immortal
Immortal
Jump to solution

so the question seems to be, who do we listen to?

VMware say it's aligned so no need to perform any

further alignment.

EMC say it's offset so not only do we need to align

to the ESX but also to the VM as well.......

In this case, listen to VMware. Proof is easy - you can take a look at any VMFS partition that was created using VC (and NOT by ESX installer). You'll notice it is aligned to 64k.

Also dont forget to torture your EMC contact to get the EMC documentation fixed Smiley Wink

Reply
0 Kudos
whynotq
Commander
Commander
Jump to solution

OK, let's see what they have to say. I am working with high-level engineering from EMC this week I'll put it to them as this is their "best practices doc".

watch this space for an update....

Reply
0 Kudos
bganesha
Enthusiast
Enthusiast
Jump to solution

Hello,

One of my friends in VMware engineering who works on VMFS pointed this to me so I decided to respond. I am part of Symmetrix engineering.

VMware ESX 3.0 will automatically align partitions on 64 KB boundary for all partitions created on SAN attached disks. The procedure that was described earlier in this thread is something that I had listed in my best practices presentation for ESX 2.x. This procedure \*does not* apply to ESX 3.0 environments. The updated EMC solution guides for ESX 3.0 (to be released end of this year or early next year) will list this difference.

Do note that the virtual disks are still created with a geometry of 63 sectors per head. So all the benefits of aligned VMFS will be lost if the partitions created in the virtual disks is not aligned. So we do recommend that you align partitions for virtual disks that would be used as data disks (not boot or applications disks).

I would like to take this opportunity to point out something that was mentioned in this thread about performance, ESX and alignment. The author is correct in his observation that he has not experienced performance problems attributed to alignment.

The best practices recommendation for partition alignment is based on experience. When Microsoft applications were initially deployed on the SAN a few years back customers did not experience any performance problems due to misaligned IOs. In fact, a number of them ignored the recommendation. However, within a short time, that changed due to the growth in the applications, changing criticality of the applications and increased compute power and maturing of the Windows OS. A number of customers experienced a lot of pain migrating from non-aligned partitions to aligned partitions. We believe something similar is very likely in the ESX environment.

A quick comparison of the capabilities of ESX 2.x and ESX 3.0 clearly indicates how rapidly the virtual environment is changing. Coupled this with the fact that a number of customers who have decided to standardize on virtual infrastructure as the de facto standard and are considering deployment of decent size databases on virtual infrastructure, the reason for the best practices recommendation is a lot more obvious. Hope this helps.

Bala

Reply
0 Kudos
admin
Immortal
Immortal
Jump to solution

one of your VMware friends, you say? Meet the alter ego.

Reply
0 Kudos
yorkie
Hot Shot
Hot Shot
Jump to solution

If this disk aligning is straight-forward to do, costs nothing, and will not have any effect except an improvement in performance on \_certain workloads_ then just do it and let's all move on.

Smiley Happy

Reply
0 Kudos
bganesha
Enthusiast
Enthusiast
Jump to solution

I agree. It used to be lot more painful with ESX 2.5 (aligning VMS partitions and then doing it again on virtual disks). With 3.0, it is much more easier. You will essentially be doing the same thing that you would have done in the physical world.

Reply
0 Kudos
bganesha
Enthusiast
Enthusiast
Jump to solution

There is small addendum to the discussion here. I failed to point it out specifically and that has raised some questions. The aligned partitions (as SV already pointed out) are created when using VC. If one uses vmkfstools to create partitions, then one does need to align the partitions.

Having said that, the procedure that was listed at the begining of the thread was something that is needed only when one needs to create aligned partitions on ESX 2.x. VMware (and EMC) strongly recommends using VC to create datastores in VI3 (thus negating the requirement for the process used in 2.x). Hope this helps.

Reply
0 Kudos
lroderic
VMware Employee
VMware Employee
Jump to solution

You can find a comprehensive tech note on this topic at http://www.vmware.com/vmtn/resources/608. You'll want to align the VMFS partition as well as the guest partitions (diskpar.exe for Windows, fdisk for Linux). Use VC 2.0 and later or VI Client 3.0 or later to create your VMFS partitions since it automatically aligns the partitions for you.

Reply
0 Kudos
joepje
Enthusiast
Enthusiast
Jump to solution

clear story...

but,

what about environments in which you want to virtualize current physical servers. If there a way to re-allign the disk partition of a virtualized vm (which is converted to virtual with p2v) ?

Reply
0 Kudos
lroderic
VMware Employee
VMware Employee
Jump to solution

Not that I can find. However, depending on the OS, the new VMware Converter (beta) might be able to help with this. Its forum is at http://www.vmware.com/community/forum.jspa?forumID=357.

Reply
0 Kudos
joepje
Enthusiast
Enthusiast
Jump to solution

should we worry about vm's that are not aligned ?

Reply
0 Kudos
VirtualKenneth
Virtuoso
Virtuoso
Jump to solution

What they told me on the VMware training was that it could make only a slight difference when the SAN if fully loaded. Consequently you won't notice it in normal loaded environments.

Reply
0 Kudos
GAustin
Contributor
Contributor
Jump to solution

The only way to correct the disk allignment during a P2V would be to create the LUN on the disk array with an Allignment Offset.

If you are using remote mirroring solutions use caution with this- While this may give you your intended results in terms of performance increases, it may also decrease effeciency surrounding array based replication. A 5% improvement in production performance may not be worth a 20% degredation in remote mirroring performance. Often remote mirroring performs optimally if the allignment is handled at the host level and not the array level.

Reply
0 Kudos
lroderic
VMware Employee
VMware Employee
Jump to solution

Your VMware trainer is correct. Migrating from unaligned to aligned is disruptive (today) since it requires a shutdown then backing up/restoring the VM's data. If your host resources aren't maxed out, it might not be worth the effort. Again, this is discussed in the tech note at http://www.vmware.com/vmtn/resources/608.

I talked with the VMware Converter team, and for P2V on Windows, you could 1) run Converter to do an online (hot) migration from physical to virtual, 2) create an aligned partition, then 3) run the Converter again against the virtual to do a file-based conversion to the aligned partition. You can get more info in the Converter forum at http://www.vmware.com/community/forum.jspa?forumID=357.

Reply
0 Kudos
dbkelly
Contributor
Contributor
Jump to solution

So, what are the consequences of aligning the LUN to the offset of the VM instead of the VMFS?

I have data that shows "partial writes" occurring when only the VMFS is aligned. Based on what has been said above, this is the expected behavior. Personally this doesn’t make sense to me. If the VMFS block size is 1MB and the VMFS is aligned, it should mask non-aligned blocks in the VM, as the storage unit would only see 1MB writes/reads.

But, if I align my LUN to the offset of the VM, and rerun the same test. All my I/O from the VM is aligned on 4K blocks and my "partial write" counters are all zero!

Can someone explain this to me?

Reply
0 Kudos
jdvcp
Enthusiast
Enthusiast
Jump to solution

How can I verify the alignment? What commands and what to look for?

thx!

Reply
0 Kudos
bowulf
Enthusiast
Enthusiast
Jump to solution

Type this command:

fdisk -lu /dev/sdX

The start of the partition should be on 128 not 63.

Reply
0 Kudos