VMware Cloud Community
carl78
Contributor
Contributor

VMFS datastore data recovery

Hello,

​I would like to advice with you.

​I had datastore on RAID 0. I lost it after re-installation of my ESXi hypervisor. I made a mistake and recreated a new datastore on the place of old datastore. No data was not written on this datastore. I can say that I see it in ls vmfs/volumes..but it is empty.

​Do you think that I have lost data definitely?

​​​thank you very much, Olda

Reply
0 Kudos
40 Replies
a_p_
Leadership
Leadership

Try to contact continuum​, who may be able to help.


André

Reply
0 Kudos
adgate
Enthusiast
Enthusiast

Hi,

Check out this blogpost regards recovering a VMFS datastore.

https://vmwaremine.com/2013/09/20/recover-vmfs-datastore-with-all-data/#sthash.RsrXGDl8.dpbs

Reply
0 Kudos
continuum
Immortal
Immortal

Do not follow the advice from adgate !!!!

By formatting the datastore you very likely overwrote the existing VMFS metadata for the vmdks stored on that volume.

It still maybe possible to recover files - read my instructions for creating a VMFS header-dump:

Create a VMFS-Header-dump using an ESXi-Host in production | VM-Sickbay

If the VMFS metadata has been overwritten only thick provisioned vmdks that were written in one piece can be recovered.

Anyway - with a headerdump I can give you a better evaluation ....

Call me via skype  ...

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
adgate
Enthusiast
Enthusiast

continuum​ what is wrong with my advice to recover VMFS3 datastore?

Am I assuming right that the VMware Knowledge Base article​ would not help to recover a lost partition table on a VMFS volume?

Reply
0 Kudos
a_p_
Leadership
Leadership

adgate

In this case the datastore has been re-formatted, which means that the VMFS metadata within the partition has been overwritten, so it's not actually a matter of recovering a partition table.


André

Reply
0 Kudos
carl78
Contributor
Contributor

Hello Ulli,

I prepared header-dump for you https://www.dropbox.com/s/igekeq9zlf9wer9/array.2048.gz?dl=0

When I checked it via trialware of VMFS recovery utils, I saw my data (VMs).

thank you very much for help, Olda

Reply
0 Kudos
carl78
Contributor
Contributor

Hello @

Reply
0 Kudos
continuum
Immortal
Immortal

Hi Olda

just downloaded your dump - unfortunately it is truncated: only 1.21 gb instead of the expected 2 GB.

But even with the truncated dump it looks promising. I found several vmdks and I expect to be able to restore full VMs including latest snapshots.

Did you watch the command execution while you created the dump ?

Looks like you received a "short write" message while using gzip ...

Anyway - I suggest to either buy the full version or call me via Skype.

With a full dump I am sure I can help you

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
carl78
Contributor
Contributor

Hi Ulli,

yes, i see in log "short write":

[root@HPG8:/dev/disks] ls -lisa | grep -v vml

total 13187275106

      4      0 drwxr-xr-x    2 root     root           512 Jun 14 20:53 .

      1      0 drwxr-xr-x   13 root     root           512 Jun 14 20:53 ..

    198 3922432 -rw-------    1 root     root     4016570368 Jun 14 20:53 mpx.vmhba32:C0:T0:L0

    186   4064 -rw-------    1 root     root       4161536 Jun 14 20:53 mpx.vmhba32:C0:T0:L0:1

    188 255984 -rw-------    1 root     root     262127616 Jun 14 20:53 mpx.vmhba32:C0:T0:L0:5

    190 255984 -rw-------    1 root     root     262127616 Jun 14 20:53 mpx.vmhba32:C0:T0:L0:6

    192 112624 -rw-------    1 root     root     115326976 Jun 14 20:53 mpx.vmhba32:C0:T0:L0:7

    194 292848 -rw-------    1 root     root     299876352 Jun 14 20:53 mpx.vmhba32:C0:T0:L0:8

    196 2621440 -rw-------    1 root     root     2684354560 Jun 14 20:53 mpx.vmhba32:C0:T0:L0:9

    183 262144 -rw-------    1 root     root     268435456 Jun 14 20:53 mpx.vmhba32:C0:T0:L1

    181 257008 -rw-------    1 root     root     263176704 Jun 14 20:53 mpx.vmhba32:C0:T0:L1:1

    178 143338560 -rw-------    1 root     root     146778685440 Jun 14 20:53 naa.600508b1001c3a79a4f4a61826ef8e55

    174 358400 -rw-------    1 root     root     367001600 Jun 14 20:53 naa.600508b1001c3a79a4f4a61826ef8e55:1

    176 142978048 -rw-------    1 root     root     146409521152 Jun 14 20:53 naa.600508b1001c3a79a4f4a61826ef8e55:2

    164 2930141144 -rw-------    1 root     root     3000464531456 Jun 14 20:53 naa.600508b1001c5afa370b08de8580cd9c

    162 2930138112 -rw-------    1 root     root     3000461427200 Jun 14 20:53 naa.600508b1001c5afa370b08de8580cd9c:1

    171 3516169176 -rw-------    1 root     root     3600557236224 Jun 14 20:53 naa.600508b1001cfd5e14e8b8e5264c73eb

    167   4064 -rw-------    1 root     root       4161536 Jun 14 20:53 naa.600508b1001cfd5e14e8b8e5264c73eb:1

    169 3516163072 -rw-------    1 root     root     3600550985728 Jun 14 20:53 naa.600508b1001cfd5e14e8b8e5264c73eb:2

[root@HPG8:/dev/disks] esxcfg-scsidevs -m

naa.600508b1001c5afa370b08de8580cd9c:1                           /vmfs/devices/disks/naa.600508b1001c5afa370b08de8580cd9c:1 5edcea68-9140a3f7-5a0f-d89d672d7                                                                                 a30  0  IBM_Array_01_OLD

[root@HPG8:/dev/disks] dd if=/dev/disks/Device:1 bs=1M count=2048 of=/tmp/replace with your name.2048

[root@HPG8:/dev/disks] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=2048 of=/tmp/replace.2048

dd: writing '/tmp/replace.2048': No space left on device

259+0 records in

258+0 records out

[root@HPG8:/dev/disks] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=2048 | gzip -c > /tmp/array.2048.gz

gzip: short write

[root@HPG8:/dev/disks] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=2048 | gzip -c > /tmp/array.2048.gz

gzip: short write

[root@HPG8:/dev/disks]

My partition is 2700 GB large (5 x 600Gb SAS hard drives). ESXi is on 4GB SD card.

How can I do full dump? I used your guide...

thank you, Olda

PS: it is my a private server, unfortuantely it doesnt make sence buy license for almost 1000 dolars for me 😞

Reply
0 Kudos
carl78
Contributor
Contributor

Dump file is necessary to recover data or only for an analyses?

Reply
0 Kudos
continuum
Immortal
Immortal

Do you have another - unaffected datastore ?

If yes - store the dump there.

If not :

- clean up /tmp

- then create dumps in smaller pieces like for example

dd if=DEVICE of=/tmp/DUMP_1.600 bs=1M count=600 skip=0

dd if=DEVICE of=/tmp/DUMP_2.600 bs=1M count=600 skip=600

dd if=DEVICE of=/tmp/DUMP_3.600 bs=1M count=600 skip=1200


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
carl78
Contributor
Contributor

Hello continuum,

I have added USB drive to ESXi. Path to drive is /vmfs/devices/disks/mpx.vmhba33:C0:T0:L0 but I am not able to store dump to this drive. I didnt find out how to do it. Do you help me?

I modified command

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=2048 of=/vmfs/devices/disks/mpx.vmhba33:C0:T0:L0/array.2048

but it doesn work. thank you Olda

Reply
0 Kudos
continuum
Immortal
Immortal

That will not work like that - and it is not worth the effort.

Instead create a partial dump.

Connect to the ESXi with WinSCP

Create an empty directory on your adminhost - call it Olda-dump

Clean up /tmp

Then run the following commands via putty one by one.

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=0    | gzip -c > /tmp/partdump500-0.gz

Download generated file into Olda-dump and clean up /tmp

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=500  | gzip -c > /tmp/partdump500-1.gz

Download generated file into Olda-dump and clean up /tmp

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=1000 | gzip -c > /tmp/partdump500-2.gz

Download generated file into Olda-dump and clean up /tmp

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=1500 | gzip -c > /tmp/partdump500-3.gz

Download generated file into Olda-dump and clean up /tmp

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=2000 | gzip -c > /tmp/partdump500-4.gz

Download generated file into Olda-dump and clean up /tmp

You now should have 5 gz-files in Olda-dump

Zip the complete directory Olda-dump and provide a download link to the zip-file.


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
carl78
Contributor
Contributor

Hello continuum

I did it and there is link: Dropbox - Dump.7z - Simplify your life

The file has over 800 MB, but there was "short write" in log too :

[root@HPG8:~] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=0|

gzip -c > /tmp/partdump500-0.gz

500+0 records in

500+0 records out

[root@HPG8:~] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=500  |

gzip -c > /tmp/partdump500-1.gz

500+0 records in

500+0 records out

[root@HPG8:~] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=1000 |

gzip -c > /tmp/partdump500-2.gz

gzip: short write

[root@HPG8:~] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=1000 |

gzip -c > /tmp/partdump500-2.gz

gzip: short write

[root@HPG8:~] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=1500 |

gzip -c > /tmp/partdump500-3.gz

gzip: short write

[root@HPG8:~] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=2000 |

gzip -c > /tmp/partdump500-4.gz

gzip: short write

thank you, Olda

BTW: for modification USB drive I used this guide:

VMware Knowledge Base

Unfortuanately it didnt work.

Reply
0 Kudos
continuum
Immortal
Immortal

Did you really clean up /tmp after each dd-command ?

So far we only have dumped the first 1000mb - thats not enough.


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
carl78
Contributor
Contributor

continuum There are only a couple of 1-2 KB files in temp directory. It doesnt help to delete them. When I delete it, i save 50-60 KB 🙂

pastedImage_0.png

Reply
0 Kudos
carl78
Contributor
Contributor

There is probably around 260 MB free space on SD card. I dont know If I can clean up other directories

Reply
0 Kudos
continuum
Immortal
Immortal

So we must make sure that none of the dump-parts ever gets larger than 250 MB ...

This should do ...

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1  bs=1M count=250 skip=0    | gzip -c > /tmp/partdump-0.gz           

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1  bs=1M count=250 skip=250  | gzip -c > /tmp/partdump-250.gz         

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1  bs=1M count=250 skip=500  | gzip -c > /tmp/partdump-500.gz         

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1  bs=1M count=250 skip=750  | gzip -c > /tmp/partdump-750.gz         

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1  bs=1M count=250 skip=1000  | gzip -c > /tmp/partdump-1000.gz        

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1  bs=1M count=250 skip=1250  | gzip -c > /tmp/partdump-1250.gz        

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1  bs=1M count=250 skip=1500  | gzip -c > /tmp/partdump-1500.gz        

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1  bs=1M count=250 skip=1750  | gzip -c > /tmp/partdump-1750.gz        

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1  bs=1M count=250 skip=2000  | gzip -c > /tmp/partdump-2000.gz         

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1  bs=1M count=250 skip=2250  | gzip -c > /tmp/partdump-2250.gz        

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1  bs=1M count=250 skip=2500  | gzip -c > /tmp/partdump-2500.gz        

Do the same as before - but please give the archive you upload a new name so that I immediatly see which is the newer one.


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
continuum
Immortal
Immortal

Wait a minute ...

Splitting the dump in several parts is so anoying ... lets try a workaround I have not used before : lets try to store the dump in a ramdisk.

Again clean up /tmp first.

Then connect via putty and run the following commands:

mkdir /moa

esxcli system visorfs ramdisk add -m 1000 -M 2000 -n moa  -t /moa  -p 0755

dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=2048 | gzip -c > /moa/olda.gz

If that works without error message you can download a one-piece dump from the /moa directory.


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos