1 2 3 Previous Next 40 Replies Latest reply on Jun 23, 2020 1:53 AM by carl78

    VMFS datastore data recovery

    carl78 Novice

      Hello,

      I would like to advice with you.

      I had datastore on RAID 0. I lost it after re-installation of my ESXi hypervisor. I made a mistake and recreated a new datastore on the place of old datastore. No data was not written on this datastore. I can say that I see it in ls vmfs/volumes..but it is empty.

      Do you think that I have lost data definitely?

      thank you very much, Olda

        • 1. Re: VMFS datastore data recovery
          a.p. Guru
          User ModeratorsCommunity WarriorsvExpert

          Try to contact continuum, who may be able to help.


          André

          • 3. Re: VMFS datastore data recovery
            continuum Guru
            Community WarriorsvExpert

            Do not follow the advice from adgate !!!!

            By formatting the datastore you very likely overwrote the existing VMFS metadata for the vmdks stored on that volume.

            It still maybe possible to recover files - read my instructions for creating a VMFS header-dump:

            Create a VMFS-Header-dump using an ESXi-Host in production | VM-Sickbay

            If the VMFS metadata has been overwritten only thick provisioned vmdks that were written in one piece can be recovered.

            Anyway - with a headerdump I can give you a better evaluation ....

            Call me via skype  ...

            Ulli

            • 4. Re: VMFS datastore data recovery
              adgate Enthusiast

              continuum what is wrong with my advice to recover VMFS3 datastore?

              Am I assuming right that the VMware Knowledge Base article would not help to recover a lost partition table on a VMFS volume?

              • 5. Re: VMFS datastore data recovery
                a.p. Guru
                vExpertCommunity WarriorsUser Moderators

                adgate

                In this case the datastore has been re-formatted, which means that the VMFS metadata within the partition has been overwritten, so it's not actually a matter of recovering a partition table.


                André

                • 6. Re: VMFS datastore data recovery
                  carl78 Novice

                  Hello Ulli,

                   

                  I prepared header-dump for you https://www.dropbox.com/s/igekeq9zlf9wer9/array.2048.gz?dl=0

                  When I checked it via trialware of VMFS recovery utils, I saw my data (VMs).

                   

                  thank you very much for help, Olda

                  • 7. Re: VMFS datastore data recovery
                    carl78 Novice

                    Hello @

                    • 8. Re: VMFS datastore data recovery
                      continuum Guru
                      vExpertCommunity Warriors

                      Hi Olda

                      just downloaded your dump - unfortunately it is truncated: only 1.21 gb instead of the expected 2 GB.

                      But even with the truncated dump it looks promising. I found several vmdks and I expect to be able to restore full VMs including latest snapshots.

                      Did you watch the command execution while you created the dump ?

                      Looks like you received a "short write" message while using gzip ...

                      Anyway - I suggest to either buy the full version or call me via Skype.

                      With a full dump I am sure I can help you

                      Ulli

                      • 9. Re: VMFS datastore data recovery
                        carl78 Novice

                        Hi Ulli,

                        yes, i see in log "short write":

                         

                         

                        [root@HPG8:/dev/disks] ls -lisa | grep -v vml

                        total 13187275106

                              4      0 drwxr-xr-x    2 root     root           512 Jun 14 20:53 .

                              1      0 drwxr-xr-x   13 root     root           512 Jun 14 20:53 ..

                            198 3922432 -rw-------    1 root     root     4016570368 Jun 14 20:53 mpx.vmhba32:C0:T0:L0

                            186   4064 -rw-------    1 root     root       4161536 Jun 14 20:53 mpx.vmhba32:C0:T0:L0:1

                            188 255984 -rw-------    1 root     root     262127616 Jun 14 20:53 mpx.vmhba32:C0:T0:L0:5

                            190 255984 -rw-------    1 root     root     262127616 Jun 14 20:53 mpx.vmhba32:C0:T0:L0:6

                            192 112624 -rw-------    1 root     root     115326976 Jun 14 20:53 mpx.vmhba32:C0:T0:L0:7

                            194 292848 -rw-------    1 root     root     299876352 Jun 14 20:53 mpx.vmhba32:C0:T0:L0:8

                            196 2621440 -rw-------    1 root     root     2684354560 Jun 14 20:53 mpx.vmhba32:C0:T0:L0:9

                            183 262144 -rw-------    1 root     root     268435456 Jun 14 20:53 mpx.vmhba32:C0:T0:L1

                            181 257008 -rw-------    1 root     root     263176704 Jun 14 20:53 mpx.vmhba32:C0:T0:L1:1

                            178 143338560 -rw-------    1 root     root     146778685440 Jun 14 20:53 naa.600508b1001c3a79a4f4a61826ef8e55

                            174 358400 -rw-------    1 root     root     367001600 Jun 14 20:53 naa.600508b1001c3a79a4f4a61826ef8e55:1

                            176 142978048 -rw-------    1 root     root     146409521152 Jun 14 20:53 naa.600508b1001c3a79a4f4a61826ef8e55:2

                            164 2930141144 -rw-------    1 root     root     3000464531456 Jun 14 20:53 naa.600508b1001c5afa370b08de8580cd9c

                            162 2930138112 -rw-------    1 root     root     3000461427200 Jun 14 20:53 naa.600508b1001c5afa370b08de8580cd9c:1

                            171 3516169176 -rw-------    1 root     root     3600557236224 Jun 14 20:53 naa.600508b1001cfd5e14e8b8e5264c73eb

                            167   4064 -rw-------    1 root     root       4161536 Jun 14 20:53 naa.600508b1001cfd5e14e8b8e5264c73eb:1

                            169 3516163072 -rw-------    1 root     root     3600550985728 Jun 14 20:53 naa.600508b1001cfd5e14e8b8e5264c73eb:2

                        [root@HPG8:/dev/disks] esxcfg-scsidevs -m

                        naa.600508b1001c5afa370b08de8580cd9c:1                           /vmfs/devices/disks/naa.600508b1001c5afa370b08de8580cd9c:1 5edcea68-9140a3f7-5a0f-d89d672d7                                                                                 a30  0  IBM_Array_01_OLD

                        [root@HPG8:/dev/disks] dd if=/dev/disks/Device:1 bs=1M count=2048 of=/tmp/replace with your name.2048

                        [root@HPG8:/dev/disks] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=2048 of=/tmp/replace.2048

                        dd: writing '/tmp/replace.2048': No space left on device

                        259+0 records in

                        258+0 records out

                        [root@HPG8:/dev/disks] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=2048 | gzip -c > /tmp/array.2048.gz

                        gzip: short write

                        [root@HPG8:/dev/disks] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=2048 | gzip -c > /tmp/array.2048.gz

                        gzip: short write

                        [root@HPG8:/dev/disks]

                         

                        My partition is 2700 GB large (5 x 600Gb SAS hard drives). ESXi is on 4GB SD card.

                        How can I do full dump? I used your guide...

                         

                        thank you, Olda

                         

                        PS: it is my a private server, unfortuantely it doesnt make sence buy license for almost 1000 dolars for me :-(

                        • 10. Re: VMFS datastore data recovery
                          carl78 Novice

                          Dump file is necessary to recover data or only for an analyses?

                          • 11. Re: VMFS datastore data recovery
                            continuum Guru
                            Community WarriorsvExpert

                            Do you have another - unaffected datastore ?

                            If yes - store the dump there.

                            If not :

                            - clean up /tmp

                            - then create dumps in smaller pieces like for example

                            dd if=DEVICE of=/tmp/DUMP_1.600 bs=1M count=600 skip=0

                            dd if=DEVICE of=/tmp/DUMP_2.600 bs=1M count=600 skip=600

                            dd if=DEVICE of=/tmp/DUMP_3.600 bs=1M count=600 skip=1200

                            • 12. Re: VMFS datastore data recovery
                              carl78 Novice

                              Hello continuum,

                              I have added USB drive to ESXi. Path to drive is /vmfs/devices/disks/mpx.vmhba33:C0:T0:L0 but I am not able to store dump to this drive. I didnt find out how to do it. Do you help me?

                               

                              I modified command

                              dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=2048 of=/vmfs/devices/disks/mpx.vmhba33:C0:T0:L0/array.2048

                               

                              but it doesn work. thank you Olda

                              • 13. Re: VMFS datastore data recovery
                                continuum Guru
                                Community WarriorsvExpert

                                That will not work like that - and it is not worth the effort.

                                Instead create a partial dump.

                                Connect to the ESXi with WinSCP

                                Create an empty directory on your adminhost - call it Olda-dump

                                Clean up /tmp

                                Then run the following commands via putty one by one.

                                dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=0    | gzip -c > /tmp/partdump500-0.gz

                                Download generated file into Olda-dump and clean up /tmp

                                dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=500  | gzip -c > /tmp/partdump500-1.gz

                                Download generated file into Olda-dump and clean up /tmp

                                dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=1000 | gzip -c > /tmp/partdump500-2.gz

                                Download generated file into Olda-dump and clean up /tmp

                                dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=1500 | gzip -c > /tmp/partdump500-3.gz

                                Download generated file into Olda-dump and clean up /tmp

                                dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=2000 | gzip -c > /tmp/partdump500-4.gz

                                Download generated file into Olda-dump and clean up /tmp

                                You now should have 5 gz-files in Olda-dump

                                Zip the complete directory Olda-dump and provide a download link to the zip-file.

                                • 14. Re: VMFS datastore data recovery
                                  carl78 Novice

                                  Hello continuum

                                   

                                  I did it and there is link: Dropbox - Dump.7z - Simplify your life

                                   

                                  The file has over 800 MB, but there was "short write" in log too :

                                   

                                  [root@HPG8:~] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=0|

                                  gzip -c > /tmp/partdump500-0.gz

                                  500+0 records in

                                  500+0 records out

                                  [root@HPG8:~] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=500  |

                                  gzip -c > /tmp/partdump500-1.gz

                                  500+0 records in

                                  500+0 records out

                                  [root@HPG8:~] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=1000 |

                                  gzip -c > /tmp/partdump500-2.gz

                                  gzip: short write

                                  [root@HPG8:~] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=1000 |

                                  gzip -c > /tmp/partdump500-2.gz

                                  gzip: short write

                                  [root@HPG8:~] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=1500 |

                                  gzip -c > /tmp/partdump500-3.gz

                                  gzip: short write

                                  [root@HPG8:~] dd if=/dev/disks/naa.600508b1001c5afa370b08de8580cd9c:1 bs=1M count=500 skip=2000 |

                                  gzip -c > /tmp/partdump500-4.gz

                                  gzip: short write

                                   

                                  thank you, Olda

                                   

                                   

                                  BTW: for modification USB drive I used this guide:

                                  VMware Knowledge Base

                                   

                                  Unfortuanately it didnt work.

                                  1 2 3 Previous Next