Grain is orphaned errors with VMWare Workstation

[EzTechAK]

Not sure if this is the right place for this or not but here it goes.

I have a client that us using very old software that isn't even in business anymore but their infrastructure needed to be upgraded. I was able to virtualize their old hardware/software with the stipulation that they would migrate as soon as they can.

Fast forward a few years later after a restart of their virtual server the file size goes down to a few mb and a bunch of error in the log file says Grain is orphaned.

I have tried to do a recovery of the vmdk via built in command line and a few software downloads with no luck. I also ran a disk check with no luck.

Luck would have it that this corruption must of been around for some time since even though the backups have succeeded the backups also seem to be corrupt.

Despite my pleads over the years and several warnings to the owner we are now in this situation and I am simply trying my due diligence to get the data back.

A reddit post suggested that I post here. I also tried to repair the vmdk with version 7.1 of vmware workstation and it simply errored out.

Any suggestions would be appreciative.

[wila]

Hi,

Yes, that was me telling you to post your problem here. This is the correct forum.

Welcome and sorry to hear that using the vmware-vdiskmanager from WS7.1 didn't do the trick.

Please follow the steps that a.p.​ lays out in this thread here: VMDK corrupted  so that either Ulli or Andre can help you.

--

Wil

[EzTechAK]

02/12/2019  11:20 AM             8,684 nvram

02/12/2019  11:20 AM        52,262,631 vmware.log

02/12/2019  11:03 AM            68,673 vmware-0.log

02/12/2019  11:01 AM            68,423 vmware-1.log

02/12/2019  11:02 AM    <DIR>          EDSERVER.EDS.Local.vmdk

02/12/2019  11:00 AM    <DIR>          EDSERVER.EDS.Local.vmx.

02/12/2019  11:02 AM    <DIR>          EDSERVER.EDS.Local-0.vm

02/12/2019  11:20 AM         9,961,472 EDSERVER.EDS.Local.vmdk

02/12/2019  11:20 AM         9,895,936 EDSERVER.EDS.Local-0.vm

02/12/2019  11:00 AM                 0 EDSERVER.EDS.Local.vmsd

02/12/2019  11:20 AM             2,368 EDSERVER.EDS.Local.vmx

02/12/2019  11:00 AM               273 EDSERVER.EDS.Local.vmxf

[a_p_]

In this case it looks like there's been a file, or file system issue which truncated the .vmdk file(s).

Since the VM is stored on a disk/partition (F:) other than the system drive, there may be a chance to recover some data, but that is definitely something that Ulli (continuum​) needs to take a look at.

André

[EzTechAK]

That is what happened after I tried to open it, I also have an intact version from backup that I did not try to open that has the original size but if I try to open it results in the same thing.

[a_p_]

Maybe that will help.

As a first step, please download dsfok.zip from http://faq.sanbarrow.com/index.php?action=artikel&cat=47&id=111&artlang=en, extract the executables, run the below mentioned command in the backup folder, then compress/zip the .bin files, and attach the .zip file to a reply post.

for %i in (*.vmdk) do @dsfo.exe "%i" 0 1536 "xxx-%~ni.bin"

What I'm confused about is that there are files, and folder with the same names in your listing!? Can you explain that? What's in the sub-directories?

What's also strange (but could be a copy&paste issue) is that the second virtual disk has a ".vm" extension (rather than .vmdk)!?

André

[EzTechAK]

I am not sure why they would have the .VM extension either. That only happens after I tried to run the VM after reboot.

[a_p_]

I also have an intact version from backup that I did not try to open that has the original size ...

The files you posted seem to be from the corrupted files, rather than from the backup files.

Please run the bleow command for the backup .vmdk files (the ones with the original sizes), and attach a .zip archive with the .bin files to a reply post

for %i in (*.vmdk) do @dsfo.exe "%i" 0 9841664 "xxx-%~ni.bin"

This will extract the metadata, i.e. no user data.

Also post the exact size (in Bytes) for both original .vmdk files.

André

[EzTechAK]

So those are the backed up files which were simply a shadow copy of the data that was made daily.

02/21/2019  09:52 AM    <DIR>          .

02/21/2019  09:52 AM    <DIR>          ..

02/21/2019  09:51 AM             1,536 xxx-EDSERVER.EDS.Local.bin

02/21/2019  09:51 AM             1,536 xxx-EDSERVER.EDS.Local-0.bin

02/20/2019  01:55 PM    64,647,135,232 EDSERVER.EDS.Local.vmdk

02/20/2019  01:56 PM    23,325,573,120 EDSERVER.EDS.Local-0.vmdk

09/15/2014  02:56 PM             1,292 EDSERVER.EDS.Local.vmx

02/21/2019  09:52 AM             1,113 Bin Files.zip

               6 File(s) 87,972,713,829 bytes

               2 Dir(s)  573,988,352,000 bytes free

[EzTechAK]

For the .VM I think I may not had grabbed the whole file name when I grabbed the text.

[a_p_]

I'm afraid that there's nothing I can do to restore the metadata. It has somehow been initialized, that is, all grain tables (including the redundant ones) except for 2 have been zeroed out. Restoring the grain table - which is basically a block of pointers to the user data locations within the file - in this case can be compared to a puzzle with more than a million pieces.

You could however try to find and extract data from within the file using a hex editor, or a tool which is able to detect files based on their characteristics. A huge number of grains are usually adjacent, so that chances to extract at least some of the data exist.

As mentioned in my first reply, this may be something where continuum​ might be able to help, or give some hints.

André

[wila]

Andre,

He mentioned using shadow copies for making backups.

Would there not be a chance that if he looks at multiple copies of his backups (which I understood from an earlier reply he has more backups) that he might be able to find a copy which does not have its grain tables wiped out?

Grain tables should not change that much over time I think? At least not for existing files that don't grow.

For the rest.. yes agreed that Ulli would be the best person to ask for anything involving recovery.

--

Wil

[a_p_]

Good point Wil.

EzTechAK If there are other shadow copies available, I can can take a look at their metadata too.

André

[EzTechAK]

So the backup service just uses the shadow copy itself to make the backup i believe and the back up that I am running these tasks off of is the oldest that we have.

Would you have any suggestions of software that may be able to work at that level to try to get data back. I am not sure what we would be able to do with the data even if we were to get what we are looking for since they don't make the software anymore for the program they used the VM for. But I guess anything is better than nothing.

I thank you for the help you were able to provide.

[wila]

Hi,

Sorry I should have been more clear.

There is no guarantee that the oldest backup is the best copy. Yes it might be, but it depends.

If your backup was taken when the VM was shut down then yes, the oldest copy is the best.

If however you took your shadow copy backups with the VM running then that assumption might well be false.

The grain tables are pointers to the actual data blocks in your vmdk files. If you allocate new data blocks those grain tables are getting updated.

As such there is a time when the grain table data area in the vmdk file does not match with what the actual grain tables are.

Apparently there are even times when parts of those grain tables are blanked out.

So there is an actual chance that another later backup has most -if not all- of the grain tables intact.

This is one of the reasons why you should not take backups that way from a running VM, but that is another topic.

What matters now is your client getting their data back.

--

Wil

[EzTechAK]

I take a daily, weekly, and monthly backup of everything. Right now the oldest I have is the monthly which is what we have been working on, I have no older backups or backups that were made when the machine was shut down unfortunately.

[EzTechAK]

Is there any way to tell why/how this happened to prevent this in the future. Besides backing up in by other means, this was never meant to be a permanent virtual machine.

[a_p_]

I can't unfortunately tell you what caused the issue, but it may have been related to the way the VM has been backed up (powered on, using shadow copies). What I think is that VMware Workstation doesn't respond to VSS, i.e. does not flush data from memory/cache to disk when the shadow copy is created.

The most secure way to backup a Workstation VM is to do this with the VM powered off.

If this is not possible for whatever reasons, it may be an option to use the vmware-vdiskmanager command line utility, and convert the virtual disks to a "Preallocated ESX-type virtual disk" (Type 4). This way the virtual disks will consume the fully provisioned disk space on the host (which, according to the file listing you posted shouldn't be an issue), but will not contain metadata (grain direcories, and grain tables). Backing up the VM with this type of virtual disks will result in a crash consistent backup.

André

[EzTechAK]

Thank you for that response.

Are you aware of any companies that may be able to repair the VMs or get the data off?

Also is there a place I can go to buy you a coffee for your time?

[a_p_]

Are you aware of any companies that may be able to repair the VMs or get the data off?

As mentioned earlier, you may try to contact continuum​. He may be able to help, or give you some hints on how to proceed.

Also is there a place I can go to buy you a coffee for your time?

You are welcome. Once you visit southern Germany, let me know. Maybe we can have a coffee (or a beer) together 😉

André