Hello,
some days ago i set up virtual machine with 2 Hard disks and installed Windows Server 2019.
one 200GB for the Windows Server 2019
and a 2.1TB for Data mapped to Drive F: currently 200MB Data and 1,8TB Free diskspace
Today i got a Error Message in web ui (in german...i switched to englisch for the next time this error pops up...so i have to translate..:-| )
"These is not enough disk space for the virtual Harddisk 'vmfs/volumes/........./Windows Sever 2019...vdmk) availabe.
and the Windows Server makes trouble (long response times and so on...) and the ESXi OS becomes non responisve
when i look at the cli a df -h shows
...
VMFS-6 2.2T 2.2T 9.0M 100% /vmfs/volumes/Daten
...
and the following files in /vmfs/volumes/Daten
-rw------- 1 root root 340 Mar 4 12:53 Windows Server 2019_2-000001.vmdk
-rw------- 1 root root 461 Feb 10 08:38 Windows Server 2019_1.vmdk
-rw------- 1 root root 465 Feb 20 08:49 Windows Server 2019_2.vmdk
-rw------- 1 root root 10.4G Mar 4 13:20 Windows Server 2019_2-000001-sesparse.vmdk
-rw------- 1 root root 40.0G Feb 10 08:38 Windows Server 2019_1-flat.vmdk
-rw------- 1 root root 2.1T Feb 20 15:26 Windows Server 2019_2-flat.vmdk
the ....sesparse.vmdk seems to bee the snapshot i created.
what causes this enourmous flat.vdmk files? What is the Problem?
The "problem" seems to be that the snapshot file sesparse.vmdk has filled your datastore.
The flat.vmdk contains the data of your virtual disk, up to the point that you took the snapshot.
Since taking the snapshot, no changes have been written to the flat.vmdk, instead the writes have been to the sesparse.vmdk
I would suggest that you need to delete the snapshot.
That was my first that was my first suspicion, so i deleted the VM snapshot in the snapshot manager. Currently i see
---------------------------------------------------------------------
> Windows Server 2019
> O You are here
---------------------------------------------------------------------
Nothing selectable for "Delete Snapshot"-Button -- I'm afraid of the enabled "Delete all" Button 😉
how do i get ahead now? The huge flat.vdmk files are still present even after ESXi restart.
The huge flat.vmdk is your virtual disk at the point in time that you took the snapshot.
What your VM is currently using as the virtual disk is that flat.vmdk PLUS the sesparse.vmdk - at a block level all writes go to the sesparse.vmdk file, while reads are taken from the sesparse.vmdk OR flat.vmdk depending on whether the particular block is present in the sesparse.vmdk file (ie. the block has been written to since you took the snapshot)
When you "delete all" to remove the snapshot the blocks of the sesparse.vmdk file will be merged into the correct locations of the flat.vmdk file, the sesparse.vmdk file will then be removed.
So, it's actually the sesparse.vmdk file that you need to "lose" - definitely NOT the flat.vmdk - and it's the action of deleting the snapshot which will do that.
I would strongly suggest reading this article: Snapshot Formats on VMFS
The "state" of the virtual disk is the flat.vmdk file, while the "delta" or "child" disk is your sesparse.vmdk file
I'm guessing the 200GB vDisk is thin provisioned and the 2.1TB is thick provisioned. That seems a bit tight for a 2.2TB datastore, did you mean to do that?
Good point.
Please run ls -lisa in the VM's folder and post the output. This will show the provisioned space as well as the currently used disk space on the datastore.
André
Both Harddisks are created with the default option "Thick provisioned, lazily zeroed".
Except the Size of the virtual disk i left all Disk parameters on default.
i pushed the "delete all" Snapshot-manager-button but that seems to have no effect.
is the 100% Use even a problem?
----- Settings of the VM -----------------------------------------------------------------------------------------------------------------
CPU 4
Memory: 24576
Hard disk 1: 200 GB
Maximum Size: 97.69 GB
Type: Thick provisioned, lazily zeroed
Disk File: [datastore1] Windows Server 2019/Windows Server 2019-000001.vmdk
Shares: Normal 1000
Limit - IOPs Unlimited
Controller location: SCSI Controller 0, SCSI(0:0)
Disk mode: Dependent
Sharing: None
Hard disk 2: 2.139999999664724 TB
Maximum Size: 9 MB
Type: Thick provisioned, lazily zeroed
Disk File: [Daten] Windows Server 2019/Windows Server 2019_2-000001.vmdk
Shares: Normal 1000
Limit - IOPs: Unlimited
Controller location: SCSI Controller 0, SCSI(0:1)
Disk mode: Depending
Sharing: None
--------------------------------------------------------------------------------------------------------------------------------
here the 2 results from df -h / ls -lisah, 15min after "delete all"-action:
[root@localhost:/vmfs/volumes/5e41149f-1347ee90-0b6e-441ea152bf0c/Windows Server 2019] df -h
Filesystem Size Used Available Use% Mounted on
VMFS-6 551.2G 429.5G 121.8G 78% /vmfs/volumes/datastore1
VMFS-6 2.2T 2.2T 9.0M 100% /vmfs/volumes/Daten
vfat 249.7M 147.6M 102.1M 59% /vmfs/volumes/ffa8bd7a-382cd6a0-0507-f52a8a2f7a22
vfat 249.7M 4.0K 249.7M 0% /vmfs/volumes/acd8f37f-57778c59-dcc7-bd9db2929a5f
vfat 285.8M 195.8M 90.0M 68% /vmfs/volumes/5e3982ff-257818f6-a261-441ea152bf0c
vfat 4.0G 16.2M 4.0G 0% /vmfs/volumes/5e39830c-37397436-6bde-441ea152bf0c
[root@localhost:/vmfs/volumes/5e41149f-1347ee90-0b6e-441ea152bf0c/Windows Server 2019] ls -lisah
total 2342572160
2180 128 drwxr-xr-x 1 root root 72.0K Mar 4 14:35 .
4 1024 drwxr-xr-t 1 root root 72.0K Feb 10 08:38 ..
1668 41943040 -rw------- 1 root root 40.0G Feb 10 08:38 Windows Server 2019_1-flat.vmdk
4195972 0 -rw------- 1 root root 461 Feb 10 08:38 Windows Server 2019_1.vmdk
16778884 2820096 -rw------- 1 root root 10.4G Mar 4 16:27 Windows Server 2019_2-000001-sesparse.vmdk
20973188 0 -rw------- 1 root root 340 Mar 4 14:35 Windows Server 2019_2-000001.vmdk
8390276 2297807872 -rw------- 1 root root 2.1T Feb 20 15:26 Windows Server 2019_2-flat.vmdk
12584580 0 -rw------- 1 root root 465 Feb 20 08:49 Windows Server 2019_2.vmdk
I can't figure out what "Maximum Size" exactly means. Even after reading the documentation for it. 9MB for a 2TB Disk doesn't sound right for me.
You have 9MB space left on the VMFS Datastore so the max. value your 2TB vDisk can be increased is 9MB which gives you a "Maximum Size" of ~2057GB.
Filling up a Datastore to 99.9999% is not a good idea and i suggest that you delete the 2TB vDisk and create a smaller one... and maybe you select "thin provisioning" this time which means that the vDisk starts small and than grows with the inner Windows NTFS when it filles up with data.
There is a option to tell the VM to store the Snapsthots on a different location(Datastore), but i dont prefer this solution and you should avoid this method because its not "standard".
Regards,
Joerg