Skip navigation
VMware

JustinTurver

2 Posts
0

After the excitement of a large non-Platespin P2V project has settled down, or in hindsight when VI admins arrive on the scene to find a previous wave of Windows template or script based VM builds have resulted in hundreds (or more) VDIs or servers with misaligned partitions, the following questions get asked:

 

  1. Should I bother to retrospectively align partitions on my Win 2K3 or XP VMs?
  2. Is the mis-alignment actually having an impact on the storage array or VM performance?
  3. Is it relevant to align Windows OS partitions?

 

Couple of pre-reqs to understand if you are a newcomer to the scene:

  • Of course, thankfully in the new era of Windows 2008 and Windows 7 we no longer need to worry about partition alignment as both diskpart and Disk Management tools do it for you (default to 1MB offset)
  • Version 5 of VMware Converter has the option to "Optimize disk layout" (aka align) partitions as part of the conversion or V2V - NICE

 

So we come back to the original 3 questions. My personal views are as follows:

 

A1:

If you have misaligned VM partitions on NetApp storage (FC or NFS), then my answer to all 3 of the above is YES

NetApp WAFL filesystem uses a 4KB block size which for Windows default 4KB NTFS cluster size means most reads / writes in the guest OS will result in additional IO overhead on the storage array

NetApp also has a little documented threshold for "concurrent partial writes" (writes that result in split IOs at the WAFL FS level due to misaligned partitions). When this threshold is reached, the array will suspend IO until the outstanding "partial writes" are cleared - not nice. Am not sure whether there is an equivalent limitation with other arrays (?)

If you google - 'NetApp partition alignment partial writes' - you will get a host of related info from the likes of Duncan Epping and others.

The impact for NetApp NFS volumes (vs FC) is apparently greater still, and operations like caching and data de-duplication can have additional impact - or less benefit - in cases where a large number of misaligned partitions reside on the array in question.

Due to larger stripe sizes on EMC and other vendors arrays (usually 64K +), the impact of misaligned partitions may be significantly less to the point that a large retrospective partition alignment project may not be worthwhile, but you may want to target some of the high IO VMs, particularly if database or log partitions are misaligned.

Keep in mind that alignment of any Windows Dynamic disks/volumes should never be attempted. Most alignment tools will skip these but I've come across instances when that is not the case and we've had to revert to backups or snapshots in order to restore corrupted partition data after a failed alignment attempt.

A2:

This is something you can quantify by measuring the IOPS on all misaligned partitions over time, and giving this info to your storage vendor. The best tool for doing this is vSCSIStats because it details the number of IOPS and the IO size. If this number is large, you have NetApp storage, and vScsistats shows you the IO profile is largely made up of 4KB reads and writes (which it will generally be with Windows - at least on the OS partitions), then that's a bad combination that should definitely be addressed

A3:

In most environments the OS partition contains the Windows pagefile. If your VMs are low on memory, or run MS SQL or Java with over-ambitious memory reservations set, you may see peaks of sometimes hundreds or even over 1000 IOPS on these partitions. In these scenarios aligning your OS (or pagefile) partition is more important.

The same applies if you run bulk VM reboots or patching runs - during startup and patch installs etc you may see significant IOPS on the OS partition

 

Having summarised my view above, I've recently had the "opportunity" to perform some retrospective bulk alignments

 

We used the following tools:

  • NetApp MBRAlign: cost effective (free with NetApp support), but only runs on NetApp and is more difficult to automate / script / schedule
  • Quest vOptimizer: worked well but is relatively slow, and requires WMI access to Windows guest. Good for production / critical systems because it performs a full VM backup before alignment starts, and automatically rolls back to this backup if the alignment fails. So - provided you have capacity on your storage for the backups - and the IO load during alignment doesn't upset anything - you can safely schedule alignments during outage windows with some confidence that the worst case scenario will be a VM that is back up and running fine in the morning, but where the alignment did not actually complete and you'll have to run it again. Downside is that it's relatively costly (it does other stuff in addition to partition alignments such as free space zeroing for reducing the size of thin disks on the SAN), and currently there is no way to select which VMDK files get backed up which becomes a problem when you just want to align a 30GB OS partition, but have a 500GB data partition that is already aligned.
  • Paragon PAT Cli (Partition Alignment Tool via command line): cost effective, overall faster than vOptimizer, and - using the WinPE embedded command line version in an ISO - can be automated using powershell scripts and does not require any interaction with the guest OS (e.g. no Windows auth required). Lacks the nice GUI and scheduling functionality in vOptimizer, but this can be worked around.

 

The attached sample script shows how this can be done (but with no automated rollback).
Only tested on vSphere 4.0x to date via Powershell 2.0 and PowerCLI 4.1
Use at your own risk, and not without fully understanding the logic

Pre-reqs:

  • Purchase and upload the PAT Cli WinPE ISO to an appropriate shared datastore.
    This should have a PAT Cli batch file embedded that will run the correct command to automatically align any mis-aligned partitons on the VM (skipping Dynamic volumes), and then reboot the VM back into the OS
  • Create the required input text file with one VM name per line
  • Currently the script assumes that all VMs will have a CDROM drive device available. It would be easy to modify the script to add one for the duration if you have none in your environment, and remove it at the end
  • Irrespective of the tool we use, we generally sVMotion a batch of VMs to a LUN from a dedicated disk subsystem in advance of an alignment run. If you don't have something similar, be carefull about how many alignments you run concurrently in terms of potential disruption to other systems using that storage.

Script:

  • Connects to a specified vCenter instance
  • Reads a batch of VM names from the text file
  • Processes only 4 Vms concurrently to limit the impact of additonal IO on storage during alignment. Easily modfiied.
  • For each VM:
  • Connects the previously configured PAT Cli ISO image to the VM
  • Updates the VM BIOS to boot from CD first
  • Attempts to "shut down guest"
  • If this has not completed after 3 minutes, powers off the VM
  • Takes a snapshot of the VM
    This provides a valid rollback in case the alignment is interrupted. Paragon has it's own recovery process which has worked reliably when I've deliberately interrupted the alignment, but having another rollback option for production systems is still vital.
    Note that during testing in a single emvironment, we found the snapshots grew about 4GB per 40GB of partition being aligned
  • Powers the VM on - it will boot into the WinPE image and run the automated PAT Cli script to perform the alignments. Yes this script can be customised to align certain partitions only if required.
  • While the alignment is running, reconfigures the VM BIOS back to the default boot order, and set's the VMs CD back to pass-through
  • Once the alignment has completed, PATCli will reboot the VM automatically
  • All going well, the VM will boot back into a nicely aligned set of partitions

 

The sample script does not automatically check that the OS has come back online after the alignment, nor does it delete the snapshot. Currently that's a manual process. For high IO VMs, it would be useful (or required) to to have this functionality in order to avoid excessive snapshot growth - assuming you can reliably verify the VM is fully functional before the snapshot is removed. If that's of concern, and you still want to automate / schedule alignments to run unattended, then it might be advisable to purchase a tool like vOptimizer.

581 Views 0 Comments Permalink Tags: performance, partition, virtual, machine, vm, netapp, to, how, alignment, align
27

The default VCB scripts do not enable backup of a select disk or disks attached to a VM - vcbMounter.exe will snapshot and backup all disks attached to a VM unless a disk is placed in "Independent" mode meaning it cannot be placed in snapshot mode.

 

In many situations you may have a VM with a system disk (say 10GB) and an application or data disk (say 50GB) - and you only want to do a full-image backup of the system disk to (and maybe an incremental of the data disk).

 

This has always frustrated me so I set out to write a script to enable me to export specific VMDKs only.

 

A description of the process follows.

 

Attached, you will find a vb script that I have written to perfrom the process from your VCB proxy server. The VBScript takes 2 arguments as shown below:

 

cscript.exe export_select_disks.vbs <FQDN_OF_VM> <COMMA-SEPARATED_LIST_SCSI_BUS_IDs_OF_DISKS_TO_BACKUP>

 

 

e.g.

cscript.exe *export_select_disks.vbs *DEVSQL01.domain.local scsi0.0,scsi0.1

 

 

You will need to modify the constants at the top of the script to suit your environment (default mount point, vcb proxy user and password, whether to delete any existing vcb snapshot for the VM etc).

 

 

Hope this helps someone!

 

 

(if you have trouble reading the below - see attached word doc)

 

 

====================================================

FULL IMAGE BACKUP OF INDIVIDUAL DISK USING VCB:

====================================================

 

NOTE: to create a usable script that implements the following process, will need to be able to feed in relevant command line args to CommVault "Pre-backup" script:

 

 

 

 

 

 

 

 

 

 

  • FQDN of VM to backup

  • COMMA-SEPARATED LIST oF SCSI ID/s oF DISK/s to export (while ignoring other disks)

 

Process is as follows:

 

1. GET "VmId" OF VM:

 

=> vcbVmName -h <vc_server> -u <vc_logon_account> -p <vc_password> -s ipaddr:<FQDN_or_IP_OF_VM>

e.g.

vcbVmName -h myvcserver.domain.local -u vcbproxyuser -p vcbproxypass -s ipaddr:vcbtest2.domain.local

 

Output (note VM ID in bold):

 

Found VM:

moref:vm-14168

name:VCBTEST2

uuid:5026fe86-c802-f99d-b1db-f3d6e9a5e732

ipaddr:10.96.64.136

 

2. CREATE SNAPSHOT (RETURNS SNAPSHOT ID FOR USE IN SUBSEQUENT COMMANDS):

 

 

SYNTAX: vcbSnapshot -h <vc_server> -u <vc_logon_account> -p <vc_password> -c <VmId> <SNAPSHOT_NAME>

 

=> vcbSnapshot -h myvcserver.domain.local -u vcbproxyuser -p vcbproxypass -c moref:vm-14168 VCB-BACKUP

 

 

Output (note snapshot ID in bold):

 

2007-10-03 19:06:27.901 'CreateSnapshot' 4316 info Creating snapshot

SsId:snapshot-14259

 

 

3. GET LIST OF DISKS IN SNAPSHOT (I.E. ENABLES IDENTIFICATION OF DISK/s TO EXPORT & BACK UP):

 

 

=> vcbSnapshot -h myvcserver.domain.local -u vcbproxyuser -p vcbproxypass -l moref:vm-14168 SsId:snapshot-14259

 

(NOTE: "ssid" and "vmid" values comes from output of previous commands)

 

 

Output (note SCSI IDs in bold):

 

 

scsi0.0:[DEVSANNET08] VCBTEST2/VCBTEST2_1.vmdk

scsi0.1:[DEVSANNET08] VCBTEST2/VCBTEST2.vmdk

vmx:[DEVSANNET08] VCBTEST2/VCBTEST2.vmx

logdir:[DEVSANNET08] VCBTEST2/

suspenddir:[DEVSANNET08] VCBTEST2/

config0:VCBTEST2.vmsd

config1:VCBTEST2.vmxf

config2:VCBTEST2.nvram

log0:vmware-7.log

log1:vmware-8.log

log2:vmware-9.log

log3:vmware-10.log

log4:vmware-5.log

log5:vmware-6.log

log6:vmware.log

 

 

4. EXPORT THE DESIRED VIRTUAL DISK/s (SCRIPT GETS DISK PATH FROM OUTPUT OF PREVIOUS COMMAND):

 

 

cd D:\mnt

md VCBTEST2.emapdev.net-CustomDiskBackup

 

 

SYNTAX: vcbExport -M 1 -d VCBTEST2_1.vmdk -s "blklst://<snapshot_id><VMDK_PATH>@<vc_servert>?<vc_user>/<vc_pass>"

 

 

NOTE: sytax must be precise!

 

E.G.:

=> vcbExport -M 1 -d D:\mnt\VCBTEST2_1.vmdk -s "blklst://snapshot-14259[DEVSANNET08] VCBTEST2/VCBTEST2_1.vmdk@lonman01.emap.net? vcbproxyuser/vcbproxypass"

 

 

Output (over SAN not NIC):

 

 

Converting "D:\mnt\VCBTEST2_1.vmdk" (compact file):

0%=====================50%=====================100%

**********************************************************************************

 

 

Does block-level export in consolidated format to a single .VMDK file.

Repeat for additional disks

 

 

5. POST-DISK-EXPORT COMMAND - COMMIT SNAPSHOT AS PER NORMAL vcbMounter POST-EXPORT PROCESS:

 

 

SYNTAX: vcbSnapshot -h <vc_server> -u <vc_logon_account> -p <vc_password> -d <VM_ID> <SNAPSHOT_ID>

 

 

E.G.

=> vcbSnapshot -h myvcserver.domain.local -u vcbproxyuser -p vcbproxypass -d moref:*vm-14168 SsId:snapshot-14259*

 

 

Then run commands to delete local temp directory holding previously exported vmdk's

 

 

6. *SAMPLE COMMAND LINE FOR COMMVAULT PREBACKUP SCRIPT:*_

 

 

SYNTAX: cscript.exe export_select_disks.vbs <VM_FQDN> <SCSI_ID1,SCSI_ID2,....>

 

 

E.G. (this is what we would specify in "Pre-scan" command line of our backup software (CommVault in our case)):

cscript.exe export_select_disks.vbs DEVSQL01.domain.local scsi0.0,scsi0.1

 

 

(this will back up VM disks SCSI(0:0) and SCSI(0:1) on the VM - each to a consolidated .vmdk file)

 

 

7. MOUNTING A DISK EXPORTED (BACKED UP) USING vcbExport (LOTS OF OPTIONS - THIS IS ONLY AN EG OF SYNTAX):

 

 

(Use for file extraction or as quick test of backup integrity)

 

SYNTAX: mountvm -cycleId -d <full_path_to_exported_disk_file> <directory_path_for_mount>

 

 

E.G.

=> mountvm -cycleId -d D:\mnt\DEVSQL01_DISK_SCSI_0.1.vmdk D:\mnt\temp

 

 

Unmount:

SYNTAX: mountvm -u <directory_path_for_mount>

=> mountvm -u D:\mnt\temp

 

 

8. HOW TO RESTORE A DISK EXPORTED (BACKED UP) USING vcbExport:

 

 

Provided "-M 1" option is used during vcbExport backup, then restore is simple process:

 

 

 

 

 

 

 

 

 

 

  • Copy the VMDK to a temp drive on the target ESX server

  • Import the vmdk file: vmkfstools -i <SOURCE_VMDK_THE_ONE_FROM_vcbExport> <DESTINATION_VMDK>
    (NOTE: This will expand the disk to it's original size!)

  • Now just add the virtual disk to either a new VM, or to original VM (at same SCSI BUS ID as per original disk you are replacing)

 

See attached files and comments in the VB Script file

14,937 Views 27 Comments Permalink


Communities