inconsistent state of guest files

adamwms · ‎09-12-2017

Hi all,

On 9 September at 13:41 Thinware vBackup job failed with the error below.

Irrelevant info skipped for brevity.

Version: 4.0.1.2747 (Version 4.0, Update 1)

Process Started: 09/09/2017 14:41:26

Execution Arguments:

- Virtual Machine = guest1

- Server = host1:22

- Job Type = Backup-Image-SSH

- Backup Root = D:\

- Disk Exclusion = None

- Backups to Keep = 1

- Quiesce Guest File System = True

- Snapshot Virtual Machine Memory = False

- Create Snapshot Task Timeout = 15

- Remove Snapshot Task Timeout = 15

- Encrypt Temporary Password File = True

- Compression Level = Advanced

- Maximum Log Size = 1024

- Backup Type = Image-Full

Server Platform: VMware ESXi 6.0.0 3620759

Virtual Machine Guest OS: Microsoft Windows 7 (64-bit)

License:

- Type: Perpetual

- Status: Active

- Edition in Use: Professional

Backup Folder: D:\\guest1

Backup: 170909_144141

No existing temporary snapshots found.

Temporary Snapshot: vBackup-TMP-170909_144141

VMware Virtual Disk Manager: VMware Virtual Disk Manager - build 2248791

VMware Virtual Disk Manager response for Hard disk 1: [datastore2-SATA] guest1/guest1.vmdk:

SSL_VerifyCbHelper: Certificate verification is disabled, so connection will proceed despite the error

[NFC ERROR] NfcNet_Recv: requested 264, recevied only 0 bytes

[NFC ERROR] NfcFile_Receive: Failed to get message

[NFC ERROR] NFC_NETWORK_ERROR

Failed to convert disk: Operation completes asynchronously (0x300004650).

E Unable to complete backup. Unable to verify successful transfer of virtual machine virtual disk file D:\\guest1\170909_144141\guest1.vmdk.

e Unable to remove temporary snapshot vBackup-TMP-170909_144141. Unable to retrieve snapshot ID. Error while creating 'snapshot' file. Failed to login: Operation timed out.

Temporary files successfully removed.

Backup data folder deleted in cleaning up after job failure.

Process Completed: 09/09/2017 16:17:19

Elapsed Time: 1 hour, 35 minutes and 53 seconds

This has left me with the following files on host1 (VMware ESXi, 6.0.0, 3620759):

drwxr-xr-x 1 root root 2660 Sep 12 04:20 .

drwxr-xr-t 1 root root 3780 Apr 19 09:47 ..

-rw------- 1 root root 31391404032 Sep 12 14:37 guest1-000001-delta.vmdk

-rw------- 1 root root 329 Sep 9 13:41 guest1-000001.vmdk

-rw------- 1 root root 12884901888 Apr 19 19:57 guest1-3d58c3bf.vswp

-rw------- 1 root root 20346 Sep 9 13:41 guest1-Snapshot19.vmsn

-rw-r--r-- 1 root root 131 Sep 9 13:41 guest1-aux.xml

-rw------- 1 root root 644245094400 Sep 9 13:41 guest1-flat.vmdk

-rwx------ 1 root root 398 Sep 9 13:41 guest1-vss_manifests19.zip

-rw------- 1 root root 503 Sep 2 19:25 guest1.vmdk

-rw-r--r-- 1 root root 583 Sep 9 13:41 guest1.vmsd

-rw-r--r-- 1 root root 3324 Sep 9 13:41 guest1.vmx

-rw------- 1 root root 0 Apr 19 19:57 guest1.vmx.lck

-rw-r--r-- 1 root root 3317 Sep 9 13:41 guest1.vmx~

-rw------- 1 root root 8684 Sep 9 13:42 old-guest1.nvram

-rw------- 1 root root 3855 Sep 12 04:20 old-guest1.vmxf

-rw-r--r-- 1 root root 1252233 Jun 24 14:06 vmware-1.log

-rw-r--r-- 1 root root 725229 Sep 12 12:10 vmware.log

-rw------- 1 root root 1047527424 Apr 19 19:57 vmx-guest1-1029227455-1.vswp

Host1 appears to be running fine but this is clearly a messy situation.

I'm concert to touch it before resolving this.

I don't know what happens if I:

- restart the guest or the host

- take a snapshot

- rerun the backup

Even though the vmdk file is pretty big disk space should not be an issue:

host1 - 3.2TB free on datastore

guest1 OS - 85GB free

backup destination media 😧 - 3.4TB free

But I've noticed the following in vSphere:

Provisioned Storage: 1.18TB
Not-shared Storage: 642.21GB
Used Storage: 642.21GB

Is it possible that it runs out of space during snapshotting / imagining?

I would like to know what the safest recovery procedure from this situation is.

And what the root cause of me ended up here was.

Thanks

Adam

All

inconsistent state of guest files