CentOS 7 VMware fusion machine hangs, cannot get it is start

[gxclark]

I'm running VMware Fusion 11.5.1 on Mac OS Sierra 10.12.6

I have a PostgreSQL database, and I think it reached over 1 terabyte.

Originally I got an error and the VM would not start. Now, after some troubleshooting, the VM boot sequence just hangs.

I've used the ISO image to boot and then mount the system (chroot /mnt/sysimage) so I can browse, and I deleted some big CSV files.

I've also added this VMDK to a brand new VM of the same OS version, but it still will not boot, just hangs.

After I logged in an freed up about 20GB+ of extra space, from the host, here are some commands that I've tried, and their results:

$ sudo /Applications/VMware\ Fusion.app/Contents/Library/vmware-vdiskmanager -R /_vm/_test/pgsql_hana.vmwarevm/Virtual\ Disk.vmdk
Password:
The virtual disk, '/_vm/_test/pgsql_hana.vmwarevm/Virtual Disk.vmdk', was corrupted and has been successfully repaired.

$ /Applications/VMware\ Fusion.app/Contents/Library/vmware-vdiskmanager -d /_vm/_test/pgsql_hana.vmwarevm/Virtual\ Disk.vmdk
Failed to defragment: An error occurred while writing a file; the disk is full. Data has not been saved. Free some disk space and try again (0x8).

$ /Applications/VMware\ Fusion.app/Contents/Library/vmware-vdiskmanager -k /_vm/_test/pgsql_hana.vmwarevm/Virtual\ Disk.vmdk
Failed to shrink the disk '/_vm/_test/pgsql_hana.vmwarevm/Virtual Disk.vmdk' : An error occurred while writing a file; the disk is full. Data has not been saved. Free some disk space and try again (0x8).

This is the log section that I think is applicable.

2022-10-05T22:16:49.771-08:00| vcpu-3| I125: DISKUTIL: scsi0:0 : capacity=3145728000 logical sector size=512
2022-10-05T22:16:49.771-08:00| vcpu-3| I125: SCSI DEVICE (scsi0:0): MODE SENSE(6) for unsupported page 0x8
2022-10-05T22:16:49.772-08:00| vcpu-3| I125: DISKUTIL: scsi0:0 : capacity=3145728000 logical sector size=512
2022-10-05T22:16:49.772-08:00| vcpu-3| I125: SCSI DEVICE (scsi0:0): MODE SENSE(6) for unsupported page 0x8
2022-10-05T22:16:49.773-08:00| vcpu-3| I125: DISKUTIL: scsi0:0 : capacity=3145728000 logical sector size=512
2022-10-05T22:16:49.773-08:00| vcpu-3| I125: SCSI DEVICE (scsi0:0): MODE SENSE(6) for unsupported page 0x8
2022-10-05T22:16:49.783-08:00| vcpu-0| I125: CDROM ide1:0: CMD 0x52 (*UNKNOWN (0x52)*) FAILED (key 0x5 asc 0x20 ascq 0)
2022-10-05T22:16:50.168-08:00| vcpu-0| I125: Tools: Running status rpc handler: 0 => 1.
2022-10-05T22:16:50.168-08:00| vcpu-0| I125: Tools: Changing running status: 0 => 1.
2022-10-05T22:16:50.168-08:00| vcpu-0| I125: Tools: [RunningStatus] Last heartbeat value 1 (last received 0s ago)
2022-10-05T22:16:50.168-08:00| vcpu-0| I125: Tools: Removing Tools inactivity timer.
2022-10-05T22:16:50.223-08:00| vthread-192155020| I125: VTHREAD 123145399185408 "vthread-192155020" tid 192155020
2022-10-05T22:16:50.224-08:00| vthread-192155021| I125: VTHREAD 123145403977728 "vthread-192155021" tid 192155021
2022-10-05T22:16:50.224-08:00| vthread-192155022| I125: VTHREAD 123145411452928 "vthread-192155022" tid 192155022
2022-10-05T22:16:50.237-08:00| vcpu-2| I125: DDB: "longContentID" = "7266c3dcef095c91ad6f30e262ccb47f" (was "5b4531943b8a3d5f11de5f6efb829d6e")
2022-10-05T22:16:50.238-08:00| vthread-192155023| I125: VTHREAD 123145411989504 "vthread-192155023" tid 192155023
2022-10-05T22:17:10.169-08:00| vcpu-0| I125: Tools: Tools heartbeat timeout.
2022-10-05T22:17:10.169-08:00| vcpu-0| I125: Tools: Running status rpc handler: 1 => 0.
2022-10-05T22:17:10.169-08:00| vcpu-0| I125: Tools: Changing running status: 1 => 0.

I would like to at least save some TXT and CSV files on the file system, or better yet start PostgreSQL and write some backup files to disk, and then save them.

Please assist!

Regards,
Geoffrey Clark

[Technogeezer]

Just a note. Fusion 11.5.1 is not supported on Sierra 10.12. It requires High Sierra 10.13, per https://kb.vmware.com/s/article/2088571 Have you tried using Fusion 11.1.1, which is the last version supported on Sierra?

I notice that you're trying to shrink your virtual disk, and that it's reporting insufficient space to complete the operation. How is the virtual disk configured in Fusion (is it a single file, or is is split into multiple pieces).

You also say that you deleted some files in the VM. Unless you defrag the VM and then write zeros to the remainder of the virtual disk from within the guest, a shrink operation will not reclaim space on the Mac.

What is the configuration of your Mac mini? (memory and hard disk size)?

The log section doesn't really say anything out of the ordinary. To me, the "Tools heartbeat timeout" isn't unexpected, and I see that on my running Linux VMs. It may be consistent with not having VMware Tools installed, but using open-vm-tools. Open-vm-tools does not report back to the host that it's installed like VMware Tools does.

My immediate thought is that you are still running out of space on the Mac. Have you tried moving the VM to an external drive? (You should also make a copy of the VM to another device before doing anything further to preserve it as we try to figure out what's going on). 

[gxclark]

Hey, thanks for the reply.

Not aware of 11.5.1 not supported on Sierra 10.12 ... I upgraded a while back, this machine has been running Fusion since ~v7.  The mac itself has 16 GB RAM, and 2.1 TB SSD, of which 1 TB is free.  I've built a new VM as part of this troubleshooting with the same CentOS 7 ISO, it runs OK, no issues with lack of disk space on the Mac.

The VMDK is a single file. After I clear space within linux, then I try to defrag from the host using vmware-vdiskmanager -d ... but that complains "Failed to defragment: An error occurred while writing a file; the disk is full. Data has not been saved. Free some disk space and try again (0x8)."  How do I get this space recognized as free?

Thanks a lot.

[gxclark]

It looks like I used to have VMware-Fusion-11.0.2-10952296.dmg installed (~2019-02-17), and I upgraded to 11.5.1-15019442 ~2019-12-05.  But I've been running 11.5.1 since then without any issues.  I can fall back if needed, but I don't think this is the root of the problem

[Technogeezer]

Since you have a monolithic (not sliced) VMDK file, you will need enough space to hold the original file plus the new file created by the vmware-vdiskmanager utility until the process is completed.. if there is not enough space on the Mac, to do so, you will get this out of space warning. The new file could be up to the same size os the old, depending on how much space can be recovered  in your case, the new file seems to be consuming all of the available space, hence the error  

IMO the defrag operation of VMware-vdiskmanager is useless for Linux.  Deleting files in the guest is not sufficient in many cases to reclaim space on the host.  IMO unless you have both defragmented within the guest and written zeros to fill up the file system ( example: dd if= /dev/zero of=/zerofile; rm /zerofile) a shrink operation may not get you back as much as you’d think.

How big of a VMDK file did you configure with Fusion, and what is its current size in the VM bundle?

If all you want is to get files off that VM, why not create a new VM with its own boot disk and add the old virtual disk as a second hard drive? 

I’m thinking that you should shift into Linux recovery thinking. In my mind there’s something wrong within that VM if it’s failing to boot, Did you run a full fsck on that VM’s file system from the installation ISO? And if it won’t boot, have you checked out that disk’s /boot partition to see if it’s OK. 

Any event come to mind that immediately preceded the inability of this VM to boot? 

[gxclark]

OK, thanks for the explanation, it helps a lot.

The Virtual Disk is set to 1500 GB.  The VM bundle is just at 1.01 TB.

What I was doing right before the crash was loading data from a SQLite DB into Postgres ... I think this pushed me over some limit for a linux VM of 1 TB ...

[root@pgsql-hana _oob]# pgloader sqlite:///opt/_data/_mil/_oob/soviet_oob.db pgsql://postgres:pwd@localhost/lucidata

2022-09-23T05:37:51.072000+01:00 LOG pgloader version "3.6.2"

2022-09-23T05:37:51.206000+01:00 LOG Migrating from #<SQLITE-CONNECTION sqlite:///opt/_data/_mil/_oob/soviet_oob.db {10059EC9E3}>

2022-09-23T05:37:51.206000+01:00 LOG Migrating into #<PGSQL-CONNECTION pgsql://postgres@localhost:5432/lucidata {1005C2FDD3}>

client_loop: send disconnect: Broken pipe

I do need to switch over to linux recovery mode.  I did boot into linux from the ISO and try to recover things.

I did not yet try try a full fsck, but I will research that and try it next.

Thanks again.

[RDPetruska]

There's your problem.  Your virtual disk is a single file disk with a max defined size of 1.5TB.  In order to do ANY virtual disk operations on it (defragment, compact/shrink, convert to different type, etc.) you will need at least 1.5 TB free on your host disk (the max defined size plus some additional for overhead).