Unable to create snapshots of VM's with large vmdk's (2 GB++)

[sharzas]

Hi

I have a virtual machine - I hadn't really been creating snapshots for this one, but one day I needed to, only to find out I couldn't. It throws the error:

"One of the parameters supplied is invalid"

So after testing a bit around, I realise several things:

1. This seems to be a general issue - tested on 2 separate PC's
2. It is not VM specific - it happens to VM's with very large VMDKs - problem begins between 1.5TB - 2TB (Snapshots create succesfully up to 1.5TB, but fails at 2TB and anything larger - I haven't tested values in between 1.5 - 2TB)
3. It happens to VM's with HW version 10,11,12
4. It happens on Workstation 10, 11 and 12
5. Seems to happen on VMDKs which has allocated the disk space. I haven't tested this on a thin disk, actually growed to a size greater than 1.5TB, however snapshotting a 4TB thin VMDK, not yet allocated beyond 1.5TB works fine.
6. It happens wether the VM is powered off or on when attempting the snapshot.
7. Nothing is logged.

This puzzles me - is there any hitherto unknown snapshot limitation in Workstation, around very large VMDKs??

If anyone can shed some light on this, I'd be grateful.

Regards,

        Sharza

[admin]

What is your host OS?

Even though you say that nothing is logged, could you reproduce the problem with a powered-off VM and then provide the UI log (its location is given in the Help > About dialog)?

[sharzas]

Ahh I knew there was something I forgot.

Tested on the following OSes:

• Windows Server 2012 R2
• Windows 8.1 x64

Attached the relevant portion of the UI log - unfortunately there is no mention of exactly what parameter is considered wrong, in what context it is specified, and where it is specified, which could be nice to know. I assume its an internal parameter.

Tracing with Process Monitor shows the snapshot proceeding nicely along, until the large file is encountered, at which point vmware.exe only queries for the existence of the -000001.vmdk file, and then rolls the snapshot back, as opposed to creating the snapshot vmdk file and proceeding.

[bluefirestorm]

Considering that the different VMware products such as ESXi, Workstation, and Fusion share common technologies; VMDKs and snapshots is probably one of them; it would not be surprising that limitations that apply to ESXi will also apply to Workstation and Fusion.

If the enterprise ESXi has this problem in taking a 2TB snapshot, not surprised that Workstation would, too.

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10123...

If you look at the KB, there is an overhead to creating snapshots, and that it is recommended that VMDK is limited to 2032GB.

[sharzas]

I would not be surprised too. However that specific issue is with VMFS3 or VMFS5 with a certain block size.

In my current infrastrastructure vSphere environments, I have VMDK files well in excess of 2TB, across ESX 5.5, 6.0 and 6.5 hosts, on VMFS5/6 datastores, that snapshot with no issues.

I would expect Workstation 12 would share some of the newer technologies (at least ESX 5.5+), and wouldn't be limited by anything but the operational filesystem, in this case NTFS, which allows very large files. There is in the time of testing, way more free space than the total size of the VMDK and potential overhead, on the disk with the VM - times 10 in fact, so that should not be an issue.

I cannot find anywhere official, where this is stated as a limitation in Workstation, which is why I wonder about it. I would at least expect it to be noted with the configuration maximums - that even tho' 8 TB vmdk's are supported on newer hardware versions, snapshots will be unavailable above a certain size etc., but I cannot seem to find that information - so my current thinking, is it should be both possible and supported. If no one here has any such experience, I'll open a support case - where I (hopefully) should be able to get a definitive answer, however any feedback from you guys is much appreciated!

What really puzzles me though - is the error message. It looks more to me, that there is somewhere in the code, a variable not able to hold the size of the file, or similar, which makes workstation bail out with an invalid parameter message - like they increased the maximum limits, but somewhere there is some straggling code, that never really got updated, to the possibility that someone would actually ever use such large files, on a desktop product, and even attempt to create a snapshot.

[bluefirestorm]

If you look at this KB,

Support for virtual machine disks larger than 2 TB in VMware ESXi 5.5.x and 6.0.x (2058287) | VMware...

Snapshots taken on VMDKs larger than 2 TB are now in Space Efficient Virtual Disk (SESPARSE) format. No user interaction is required. The redo logs are automatically created as SESPARSE instead of VMFSSPARSE (delta) when the base flat VMDK is larger than 2 TB.

While you look at the log you attached, in the creation of the snapshot VMDK

2017-05-26T08:10:53.557+02:00| vmui| I125: DISKLIB-LIB_CREATE   : CREATE CHILD: "R:\Vmware\VMs\W8-PRON-X64-TEM\2TB-000001.vmdk" -- monolithicSparse grainSize=128 policy=''

I would suspect the SESPARSE format is currently not supported (yet) by Workstation and it is monolithicSparse that is the likely invalid parameter.

[sharzas]

It seems like a sound suggestion. I will reach out to support, when I get around to it - these days are busy days

Thats most likely the route to go, in terms of getting a definitive answer to that question.

[continuum]

WS 12 running on Windows 2012 can create "twoGbMaxExtentSparse" snapshots for vmdks  larger than 2tb.
I would rather check if you use an "alien" filesystem.
And for a  Windows 2012 host everything else but plain UNCOMPRESSED NTFS should be considered as alien. That includes Fat32, ReFS , exFat and even compressed NTFS.

Please attach a COMPLETE vmware.log to your next reply.

[sharzas]

continuum​ - I would gladly attach a complete vmware.log, the thing is - there is none. The machine is not powered on, the snapshot fails with zero information logged to vmware.log - the only log to find, is the text in the ui.log, as provided in an earlier reply.

However - for the sake of this troubleshooting - I powered the VM on before attempting a new snapshot - vmware.log for that session has been attached.

As for the file system, see attached text files.

We're talking a vanilla NTFS 3.1 volume. The only thing on this volume is the VM in question. Created and formatted for the sole purpose of testing this. There is no compressed files or directories on the volume.

[continuum]

Interesting.
Kenneth - I would like to examine this case in a remote session.
If you can arrange a Teamviewer 10 session for early next week I would like to have a closer look.
I am interested why WS wants to create a delta named blabla-001-000001.vmdk.
Anyway - if you are interested - call me via skype - see signature
Ulli

[sharzas]

I am very interested, and can definetely arrange a TeamViewer session - albeit it'll be a v12 session.

Don't be fooled by the 001-000001 suffix - it is actually me having a general habit, of naming my disks -001, -002, -003 etc, which is where the -001 comes from, its not WS induced.

I used an existing VM to test this on, only adding additional vmdk's of increasing size, and to keep it simple, I named them just the size of the disk - leaving out the -XXX numbering suffix.

The -001 is the existing OS disk.

What time frame is the best to catch you btw?

- Kenneth

[sharzas]

Update with solution:

So after a quite cheerful few hours together with forum member continuum, on TeamViewer/Skype - whenever my internet connection having a life of its own apparently, decided to play nice - a solution to the issue was found.

The culprit was identified to having nothing with the total size of the vmdk to do at all. The issue is related to the extent size.

Whenever a single .vmdk disk extent exceeds 4261412864 sectors, which equals 2032GB - just a bit under 2TB, any attempt to make a snapshot fails instantly. We tried to work around this using various types of disks and createtypes, only being rewarded with various other vague and less than descriptive error messages.

The solution is to not use extents with more than 4261412864 segments - plain and simple. You can use whatever number of extents tho' just keep below that number of segments, and make sure the geometry matches.

I will update this question with further details - I do however have a few more tests to run, which involves letting workstation create disks from scratch, in various sizes and types - which is a lenghty affair in 2 - 4 TB+ scale. Whenever done, I will dutifully return with the results.

Until then - thanks for all the feedback, and thanks for a quite enjoyable session Ulli - you truly take Vmware geekism to whole new level

- Kenneth

[sharzas · Accepted solution]

As promised an update to this question, with a few further details.

I wanted to know, if there for some reason, was a difference between using the GUI and the command line tools (vmware-vdiskmanager), when creating a new disk. Basically I'm pretty sure the GUI calls vmware-vdiskmanager, using the correct parameters, as specified by the user - so there SHOULD not be any difference in behavior.

Quite as expected - there is no difference technically between creating a disk using the GUI, and using vmware-vdiskmanager. There is however one thing that works out a bit different. When using the GUI, there are some sanity checks, which will amongst other things, make sure you stay within the supported limits, and most importantly force you to create a pre-allocated disk as a split disk, as soon as it exceeds 2032GB in size, which according to the tests carried out, is the maximum extent size capable of being snapshotted. In other words, if you use the GUI to create your disk - you'll be just fine, as it prevents you from repeating my mistake.

When you create a disk using vmware-vdiskmanager, there is no such check, and if you request a non-split pre-allocated disk, you get it - which results in only one -flat.vmdk extent being created, regardless of the size specified. So if you decide to create a non-split pre-allocated disk, with a size  of 3.5 TB, you will get just one extent - and it will cause an error if you attempt to create a snapshot of the VM. It will run just fine in any other aspects, you just won't be able to create a snapshot of it.

The very simple solution to this issue, is to always use the type "preallocated virtual disk split in 2GB files" - which is type 3 using vmware-vdiskmanager. Don't be fooled by the "split in 2GB files". I did and it was what caused all this trouble for me in the first place. I kinda thought - hell no I want 3.5 TB split into 2GB files. In reality Vmware Workstation will split the file, at maximum allowable extent size, which is 2032GB - someone in the development team apparantly didn't find it important to update the help description as well.

Workstation will let you do just about anything with your pre-allocated vmdk, as long as your vmdk descriptor file is accurate in terms of geometry, and your -flat.vmdk extents matches the extent definition in the descriptor file. Just for the sake of pushing the limits, and for my own curiosity ignited by the session with continuum, I made a 11.9TB - or more precisely a 12192GB pre-allocated disk, consisting of 6 extents exactly 2032GB large. As most know - the supported limits for Vmware workstation 12 is 8GB, and its the maximum size you're allowed to create using conventional weapons. However if you calculate your geometry and creates the extents manually (either by copying existing ones, or using fsutil), you can go way beyond the supported limit, it is - of course - obviously not supported, and not recommended either. The disk worked flawlessly, initiated, created partitions, stored files - wrote data to offsets past the 8TB limit, and read them just fine - even snapshots worked fine, however committing the snapshot to the running (idle) vm, was a lenghty affair, which might end up being a neverending task on a active vm. Again - this is only for experimenting, and I urge you not to do it for any data you intend to keep.

I have attached the .vmdk descriptor files, used in these various experiments, to this post for anyone interested. They are named like this:

3.5TB-SINGLE-VIA-CMDLINE.vmdk - 3.5TB non-split pre-allocated disk created using vmware-vdiskmanager

4TB-SPLIT-VIA-CMDLINE.vmdk - 4TB split pre-allocated disk created using vmware-vdiskmanager

4TB-SPLIT-VIA-GUI.vmdk - 4TB split pre-allocated disk created using GUI

4064GB.vmdk - 4064GB split pre-allocated disk created to illustrate how it consist of exactly 2 extents, exactly 2032GB in size.

12192GB.vmdk - 12192GB split pre-allocated disk manually created to illustrate what is possible with vmdk's

So thats it - issue resolved, knowledge gained, and a quite amusing session with a highly respected vmware dude richer.

I hope that someone else, may have use of this information, and a big thanks to continuum for providing the missing pieces!

Cheers,

       Kenneth