VMware Communities
dan0001
Contributor
Contributor

Will I Have Issues With The New Advanced Format Drives On A WinXP Guest?

Hi,

I run workstation 7.1 on a windows 7 64 bit host with a number of windows xp sp3 (and windows server 2003) guests - I am looking to get one of the large capacity 2.5 inch drives that are about to be released (either Seagate or WD 750Gb) but I have read that windows xp will have problems with the way these drives (and apparently all new drives from now on) have changed the size of their sectors from 512 bytes to 4k, something called Advanced Format.

I don't pretend to understand too much about the new format but I do know that they will all come with a solution to the problem so that the drives won't have any issues on windows xp. However, from what I understand the solution works for either the new operating systems or the old ones, but not both. So I have no idea how that will affect my vm setup - i.e. do I install the fix and knacker the host or not install it and then knacker the guests?

Can anyone advise if there is a way for the host and guests to be unaffected?

Or am I worrying needlessly and for technical reasons I don't know about, this setup won't be affected by the new format?

I have posted here rather than in Seagate/WD forums as I think this scenario is something they would not know how to advise on and it may even be that the way the vm disks are constructed, this may not even be an issue (although that is more me being optimistic than anything!).

Cheers
Dan

0 Kudos
7 Replies
DanielARK
Contributor
Contributor

You don't mention if you're on a laptop or desktop, but if you're on a laptop, it might be worth considering the Seagate 750 GB with hardware encryption built-in. You set a password in the BIOS, and your entire drive will be AES encrypted with no performance hit. Good for when someone steals your puter Smiley Wink

Anyway, the issue with the new 4k sectors, as you also write, is specific to Windows XP and previous Windowses. The problem here is two-fold: You cannot boot Windows XP from a disk partitioned using the newer GPT scheme, you have to use DOS-style partitions. And, the real culprit: the boot partitions for Windows XP additionally *need* to be aligned in a way that causes them to not start where a sector starts on these new 4k drives.

That means that each block on a new 4k drive that has been partitioned by, or for, Windows XP, will have to read and write two sectors each time you're trying to read or write one.

Because you're using Windows 7, this is no longer an issue, as Windows 7 installs and runs just fine on GPT partitioned disks with properly aligned partitions. But, if you're dual-booting with Windows XP, for sure your partitions are misaligned.

Don't quote me on this, but I even think that Windows 7, if you let it partition the disk itself, does it right the first time. Re-check with Google to make sure.

Finally, some vendors IIRC, have claimed to have solved the issue internally in hardware, so it may be a non-issue altogether. Btw, this technology is here to support LBA on really large drives, so I don't really get why they'd even use it on a 750 GB drive...

Now, about whether or not this will affect your VMs, then I guess, yes and no. Once your physical disk has been properly partitioned and formatted and Windows 7 installed on it, then virtual disk images themselves will also be properly aligned to these 4k sectors. (Btw, now, more than ever would be a good time for anyone to stop using 2k, 1k and 512-byte sizes when formatting NTFS, as this will seriously impact performance since the drives are no longer able to read or write anything less than 4k at a time).

The "yes" would be due to the fact that the virtual disk image will be formatted internally, and so when the guest is XP (and most likely 2003, too), you'll end up with the exact same issue. HOWEVER! And this is just me guessing here based on what seems logical - this would always be an issue for virtual disks with XP guest OSes on them, as NTFS (and most other filesystems) is by default always formatted with 4k sectors. And as the NTFS filesystem on the host is by functional comparison equal to "physical hardware" with respect to the virtual disk image, it in itself imposes the same problem that the new 4k drives do. So, I'm guessing this issue would have been addressed in VMware already? Someone with insight on this?

My two cents, anyway Smiley Wink

0 Kudos
dan0001
Contributor
Contributor

Many thanks for the reply, DanielARK!

I am a bit confused about the sector size thing now though as I decided to try and find out where all my disks were at. I ran fsutil fsinfo ntfsinfo C: at the command prompt on my host and 2 vms and got exactly the same results for all 3:-

Host - OS = Win7x64

Guest1 - OS = Win7x64

Guest2 - OS = WinXPSP3

Bytes Per Sector  :               512
Bytes Per Cluster :               4096

It seems that even my win7 machines (host and guest) are running at 512 bytes per sector anyway! So I am guessing that the win7 installation was the culprit here as I let it full format with ntfs the disks during setup.

So maybe this at least explains why the problem hasn't manifested itself more commonly on todays setup with xp guests on win7 hosts as, at least in my case, both are running with 512 bytes per sector. I agree though that a fair few people would have formatted with 4k sectors so it is likely this has been an issue and been resolved before - could anyone from vmware help shed any light on this?

PS yep, its a laptop Smiley Happy

0 Kudos
DanielARK
Contributor
Contributor

Hmm, well, I have to admit I may once again be mistaken here, but as far as my brain capacity allows me to deduct, the sector size referred to there is the physical sector size. But one of the points of NTFS is to avoid dependencies on hardware sectors sizes, as is the case with earlier MS filesystems. Check this out: http://support.microsoft.com/kb/100108

The cluster size in NTFS determines the smallest allocateable chunk of data possible, with the exception of files so small that NTFS will chose to put them directly into the Master File Table (MFT) rather than write them to a data block. Try this: Open a folder, right-click and chose "New" > "Text Document". This file, if you right-click and chose Properties, will be zero bytes, but more importantly, the "Size on disk" property, i.e. the actual amount of data that has been allocated in order to store it on disk, is also zero bytes. This is because NTFS determines that the extremely small size of it + the overhead of keeping its name and statistics in the MFT + data pointers about where the actual file contents (there are no contents) are on disk + having to seek to it, isn't worth it. It just keeps whatever tiny data packed into the MFT together with the files statistics.

Actually, now that I'm checking it, it seems only zero-byte files and folders are packed directly into the NTFS MFT. I'm mostly a Linux guy myself, and here you can often choose (depending on FS) to have small bits of file data packed into the FS metadata rather than "waste" space having to use e.g. a 4k block (/cluster) to store a 12-byte piece of data. It's called "tailing", to indicate that the small "tails" of data at the end of files that don't take up an entire allocation unit, get collected in the metadata instead.

Anyway, once you write just a single letter into that new .txt file and save it, you'll see that the "Size on disk" property jumps to 4k even though the actual size is just 1 byte. That's because you can't allocate less than an entire cluster (sorry, btw., that I've been using these terms (sector, block, cluster, etc.) interchangably, I guess I'm just damaged Smiley Wink ).

The reason that 4k is the default cluster size for NTFS is because of two things:

1) Each time you halve the cluster size, you double the number of clusters to keep track of. And so, when the space to format increases, you soon run out of address space (either 32 or 64 bits, not sure) and you have to increase the cluster size to be able to "fit" the number of resulting clusters to not exceed the largest number that the cluster address space can handle. A partition doesn't have to be very large by these days' standards before even 2k clusters become impossible to use.

2) In order to use NTFS compression your cluster size cannot be larger than 4k. So, even if you'd desire larger clusters because it makes sustained reads and writes faster (the system has to handle every cluster, so it goes naturally that for a 100 MB file on a 4k partition, you'd have 25,600 clusters to handle individually, while if you formatted that with 32k clusters, only 3,200 clusters would have to be handled - the trade-off being bye-bye compression and smallest allocateable piece of data would be 32k, i.e. more wasted space. That's fine for video editing, kinda sucks for a system drive), you'd have to use 4k in case you wanted compression, too.

Anyway, I should stop rambling now :smileygrin: Just think it's a really interesting topic. Oh yeah, the 512-byte sector size reported by fsutil! It's the same here, btw, albeit on a Windows XP x64/Kubuntu dual-boot system.

Now that is pretty interesting. And I'm wondering what it's doing there. I mean, it's true that your drive is using 512-byte sectors (and mine), but is NTFS using this knowledge for anything, and how does it know this? If it's just guessing, and it would guess the same for a new 4k drive, and it's also using it to align something internally, that could be bad. But then again, I really don't think anything in NTFS is aligned outside of the FS itself - e.g. when you mount your NTFS drive in Linux you suddenly get to see that the metadata itself is files and folders on the drive, such as "System Volume Information", "$RECYCLER", "$MFT$", etc. Although these files might just be accessors to get at the internal metadata. Hmmm... We really need someone with more expertise on this Smiley Wink

Okay, so anyway, if Windows/NTFS is correctly picking up the sector size from the drive itself, then this would in general be good, except for those drives that choose - for compatibility reasons - to present themselves as 512-byte sector drives even though they're really 4k. I read a post recently about a guy who had a drive like that. He had to be very careful in correctly partitioning the drive to avoid the possibility of misaligned FSes. That solved his problem on Linux, but if NTFS is assuming the drive to be reporting the correct sector size, and it can use this knowledge internally for something, a drive misrepresenting itself could be a problem.

Hang on, rather than ramble on like crazy, I'll try to find the link to a really interesting post I read about the 4k issue and Linux. It doesn't apply to Windows, but it explains a lot of things, and I'm sure you'll find it interesting...

Hmm... Can't find the exact article, but a couple are clearly based on it:

http://lwn.net/Articles/322777/

http://www.osnews.com/story/22872/Linux_Not_Fully_Prepared_for_4096-Byte_Sector_Hard_Drives

Here's also one that talks about the issue of misaligned partitions (the LBA 63 address being the default, to be compatible with older MS systems, including XP) and misrepresentation of sector size from the drive:

http://bugs.gentoo.org/show_bug.cgi?id=304727

What bugs me is that I myself have four 2TB drives, two of them being these WD ones, and I never thought one second about this when I partitioned them the old way Smiley Sad

0 Kudos
dan0001
Contributor
Contributor

Wow, seriously detailed reply! I won't pretend that I understood all of it, but I definitely know a lot more than I did - thanks danielARK!!

I guess there are still too many unknowns though, especially with the mis-representation, and certainly pertaining to how the vm's internally handle their disks. We haven't had an answer as yet so this probably means no-one (or at the very least, not many) knows so I guess it's up to us to experiment with existing disks to see if there is any clever stuff built in to handle it, then onto the new disks to see the effect there.

0 Kudos
DanielARK
Contributor
Contributor

You're very welcome Smiley Happy

With the risk of sounding like a Seagate employee (which I'm not), here's the drive I was considering as an upgrade for the ASUS U36Jc I've ordered. It's a 750GB 7200 RPM Seagate Momentus with encryption and this "Seagate SmartAlign" feature, which supposedly handles partitions being misaligned without performance degradation:

http://www.seagate.com/ww/v/index.jsp?name=st9750420as-momentus-7200-sata-3gbs-750-gb-hd&vgnextoid=c...

A Google search for "Seagate SmartAlign" seems to confirm that the technology works ad advertised.

Seems like a pretty good deal, especially if one wants to still be able to use, e.g. Windows XP, on a dual-boot system. Plus, you get 7200 RPM and built-in encryption without draining the CPU and battery.

I opted not to buy it, though, but only because I decided to go ca$$$h out bigtime and get an OCZ Vertex 2 SSD instead. I just got it this morning, now I just have to wait for the laptop Smiley Sad

0 Kudos
DanielARK
Contributor
Contributor

Okay, one last stab at this, just to clear out some mistakes I made, and to sum up how to get around this issue.

Here's a couple of good links, the first one explaining the problem (with relation to Linux), the other describing how to properly align partitions using fdisk on Linux:

http://lwn.net/Articles/377895/

http://www.ocztechnologyforum.com/forum/showthread.php?77769-A-Simple-How-To-on-Partitioning-and-Ali...

It seems that XP and previous Windowses don't require misaligned partitions, they "just" misalign partitions when creating them. I was repeating misinformation.

The problem is, of course, that you cannot install Windows XP and partition the disk using the setup tool without getting misaligned partitions. You also cannot create new well-aligned partitions later using the disk manager in Windows XP.

You can, however, partition the disks beforehand, using different approaches. Here's a couple of rule-of-thumbs:

1) If you have Windows 7, using it to partition the disk beforehand will always give you properly aligned partitions. (Not sure about Windows Vista)

2) Installing Windows 7 directly onto a blank disk and using the setup tool's partitioner will always give you properly aligned partitions.

3) Installing Ubuntu/Kubuntu/etc. 10.10 Maverick onto a drive will always give you properly aligned partitions (including extended/logical ones, which can be tricky to align manually using the howto I link to above).

4) You can check if your partitions are aligned using a live linux cd, such as Ubuntu or Kubuntu. Open up a terminal and write:

------

daniel@kubuntu:~$ sudo fdisk -lu /dev/sdk
Disk /dev/sdk: 60.0 GB, 60022480896 bytes
255 heads, 63 sectors/track, 7297 cylinders, total 117231408 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0008be95
   Device Boot      Start         End      Blocks   Id  System
/dev/sdk1   *        2048    52430847    26214400    7  HPFS/NTFS
/dev/sdk2        52430848    69208063     8388608   83  Linux
/dev/sdk3        69208064   111151103    20971520   83  Linux
/dev/sdk4       111151104   117231407     3040152   82  Linux swap / Solaris

------

/dev/sdk just happens to be my boot disk here, it would be sda for a one-harddrive system such as a laptop, or perhaps sdb, whatever, try it, it's read-only Smiley Wink The thing to look for here is the "Start" value. Does this, when divided by 512, give no remainder? If so, then your partitions are properly aligned. The disk you're looking at there is an SSD dual-booting Kubuntu and Windows XP x64. XP installed just fine on the properly aligned partition.

5) You can always use the HOWTO linked to above to manually partition your drive using a Linux livecd.

I've just installed three systems, the first one I set up manually - it's the one I gave output for above. The second one is in my laptop, installed to and partitioned using a Kubuntu 10.10 "alternate" cd, i.e. not the livecd. All partitions are properly aligned. The third one was installed to and partitioned using the Windows 7 installation DVD. All partitions are properly aligned.

If you're installing Windows 7 onto the drive and want to be able to use BitLocker's full-disk encryption (and perhaps other features, not sure), you're best off letting the installer partition it automatically, as that weird 100MB partition it puts at the very front of the drive is required for that to work.

Btw, the issue with some manufacturers making drives which try to compensate for XP's misalignment of partitions, seems to not be so widespread. As they say, "few have been seen in the wild". And it would be logical to assume they were produced before Windows 7, and that no-one in their right mind would make one of those today.

I think that sums up my experiences with this. For the record, all of my other drives are misaligned, with partitions starting at 63. In a stroke of luck, all of them are 512-byte sector drives. Except for a couple of Seagate drives, which they claim have the SmartAlign feature that allows you to misalign partitions on 4k drives without performance penalty, while at the same time, the manual states that they have 512-byte sectors. Go figure Smiley Happy

0 Kudos
dan0001
Contributor
Contributor

I just ordered a seagate 750gb (with smartalign) and a western digital scorpio black 750gb (with wdalign) and they should turn up in 1 to 2 weeks. I will give them both a go and compare to my current drive (seagate 500gb 7200.4) and will report back with my findings Smiley Happy

Cheers
Dan

0 Kudos