13 Replies Latest reply on May 15, 2019 4:32 PM by continuum

    Esxi 6.7 No more Datastore after Raid 1+0 rebuild  (HP micro server Gen 8)

    sebaheretic Novice

      Hi Everybody ,

      i need help to connect again my datastore at ESXI 6.7

      After a HDD fail in my Raid 1+0  in my (HP micro server Gen 8)  i have no more datastore connected i can look at harddrives and i see the the drive with partition in web interface and also in

      CLI

       

      [root@xxxxx:~] esxcfg-scsidevs -m

       

      [root@xxxxxx:~] cd /dev/disks

      [root@xxxxx:/dev/disks] ls -lisa | grep -v vml

      total 3918219898

       

          171 1953458904 -rw-------    1 root     root     2000341917696 May 13 17:54 naa.600508b100xxxxxx55d9cdfxxxxxxxxx

          169 1953456128 -rw-------    1 root     root     2000339075584 May 13 17:54 naa.600508b100xxxxxx55d9cdfxxxxxxxxx:1

       

       

      how can i fix this  ?

       

      Thank you all  in advance !

        • 1. Re: Esxi 6.7 No more Datastore after Raid 1+0 rebuild  (HP micro server Gen 8)
          Sureshkumar M Expert
          vExpert

          What does vmkernel log say ? Are you getting any error in vmkernel logs related to the lun id (naa.xxxxxxx)

          • 2. Re: Esxi 6.7 No more Datastore after Raid 1+0 rebuild  (HP micro server Gen 8)
            continuum Guru
            vExpertCommunity WarriorsUser Moderators

            First of all check wether the VMFS-volume is detected as a snapshot.

            If yes - follow this knowledgebase article:

            VMware Knowledge Base

            If that does not help - read
            Create a VMFS-Header-dump using an ESXi-Host in production | VM-Sickbay

            If you provide sucha dump I maybe able to help you recover your VMs.

            But first check the KB - if you are lucky that should help you to mount the datastore again.

            Ulli

            • 3. Re: Esxi 6.7 No more Datastore after Raid 1+0 rebuild  (HP micro server Gen 8)
              sebaheretic Novice

              thats is all i have in vmkernel.log about naa

               

              Z cpu0:2097463)ScsiDeviceIO: 9297: Get VPD 86 Inquiry for device "naa.60xxxxxxxxxxxxxxxxxxxxb80f" from Plugin "NMP" failed. Not supported

              2019-05-13T17:51:55.122Z cpu0:2097463)ScsiDeviceIO: 7998: QErr is correctly set to 0x0 for device naa.6xxxxxxxxxxxxxxxxxxxxxxxxf.

              2019-05-13T17:51:55.146Z cpu0:2097463)ScsiDeviceIO: 8495: Could not detect setting of sitpua for device naa.60xxxxxxxxxxxxxxxxxxxxxxxxx0f. Error Failure.

              2019-05-13T17:51:55.313Z cpu0:2097463)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 40, Parameter: 0x4302b4db7180, Registered!

              2019-05-13T17:51:55.313Z cpu0:2097463)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 200, Parameter: 0x4302b4db7180, Registered!

              2019-05-13T17:51:55.313Z cpu0:2097463)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 800, Parameter: 0x4302b4db7180, Registered!

              2019-05-13T17:51:55.313Z cpu0:2097463)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 400, Parameter: 0x4302b4db7180, Registered!

              2019-05-13T17:51:55.313Z cpu0:2097463)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 8, Parameter: 0x4302b4db7180, Registered!

              2019-05-13T17:51:55.313Z cpu0:2097463)ScsiDevice: 4944: Successfully registered device "naa.60xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx80f" from plugin "NMP" of type 0

              2019-05-13T17:51:55.313Z cpu0:2097463)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 180, Parameter: 0x430869010010, Registered!

              2019-05-13T17:51:55.400Z cpu0:2097463)ScsiUid: 276: Path 'vmhba32:C0:T0:L0' does not support VPD Device Id page.

               

               

              Z cpu0:2097668)ScsiDeviceIO: 3015: Cmd(0x459a40bf28c0) 0x85, CmdSN 0x0 from world 2098977 to dev "naa.60xxxxxxxxxxxxxxxxxxxxxx0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

               

              thank you

              • 4. Re: Esxi 6.7 No more Datastore after Raid 1+0 rebuild  (HP micro server Gen 8)
                sebaheretic Novice

                I'm sorry about my lack of knowledge  ,

                how can i check if there is a snapshot about my harddrive ?

                 

                naa.6xxxxxxxxxxxxxxxxxxxxxx                         vml.0xxx

                naa.6xxxxxxxxxxxxxxxxxxxf:1                        vml.0xxx:1

                 

                 

                thank you

                 

                • 5. Re: Esxi 6.7 No more Datastore after Raid 1+0 rebuild  (HP micro server Gen 8)
                  sebaheretic Novice

                  Ps ,

                  when Esxi is booted  i can notice a really heavy hard drives activity but i don't know why .

                  HDD led is  always blinking

                   

                  thank you

                  • 6. Re: Esxi 6.7 No more Datastore after Raid 1+0 rebuild  (HP micro server Gen 8)
                    continuum Guru
                    User ModeratorsvExpertCommunity Warriors

                    > how can i check if there is a snapshot about my harddrive ?
                    read the knowledgebase article I linked.

                     

                    The esxcli command is used on the command line. To list the volumes detected as snapshots, run this command:

                     

                    # esxcli storage vmfs snapshot list

                    • 7. Re: Esxi 6.7 No more Datastore after Raid 1+0 rebuild  (HP micro server Gen 8)
                      sebaheretic Novice

                      sadly no output with that command

                      +

                      [root@:~] esxcli storage vmfs snapshot list

                      [root@dae:~]

                       

                       

                      thank you

                      • 9. Re: Esxi 6.7 No more Datastore after Raid 1+0 rebuild  (HP micro server Gen 8)
                        sebaheretic Novice

                        thank you , i was like i imagine .

                        Now i have another problem, if i create the dump with DD i just have a 256Mb file 100% due to no space left in the filesystem ( i can say that because if i run dd twice the second time creat almost 0kb file casenamexx )

                         

                        how can i save the dd command in another path like an usb  or network path ?

                         

                        thank you

                        • 10. Re: Esxi 6.7 No more Datastore after Raid 1+0 rebuild  (HP micro server Gen 8)
                          continuum Guru
                          Community WarriorsUser ModeratorsvExpert

                          first do

                          cd /tmp

                          rm *.1536

                          then use the option with gzip

                          dd if=/dev/disks/Device:1 bs=1M count=1536 | gzip -c >  /tmp/sebastiano.1536.gz

                          or use another datastore if available

                          dd if=/dev/disks/Device:1 bs=1M count=1536 | gzip -c >  /vmfs/volumes/unaffected-datastore/sebastiano.1536.gz

                          • 11. Re: Esxi 6.7 No more Datastore after Raid 1+0 rebuild  (HP micro server Gen 8)
                            sebaheretic Novice

                            thank you i will connect another disk in the internal sata slot and i will create a new datastore to save the DD

                             

                            just 2 quick question before i prepare the temp datastore

                             

                            1) why the activity of the disks is so high ? i  see (and i hear ) the disk reads/writing all the time , also led hd activity is alway blinking ? (i have only the raid 10 and the micro sd with esxi installed )

                             

                            2)what could be the cause of this snapshot missing and datastore missing situation ? maybe a boot esxi before the raid rebuild ?

                             

                            thank you

                            • 12. Re: Esxi 6.7 No more Datastore after Raid 1+0 rebuild  (HP micro server Gen 8)
                              sebaheretic Novice

                              Hi continuum,

                              i have the dd dump but inside there a lot of data and information in  clear txt that i can not share  like password ecc.

                               

                              which part of the file you need ? or can i recreate the snap my my own ?

                               

                              sorry fo the inconvenient

                               

                              thank you

                               

                              ps i can just read the wmfs partition and backup my data al re-create the datastore   ?

                              • 13. Re: Esxi 6.7 No more Datastore after Raid 1+0 rebuild  (HP micro server Gen 8)
                                continuum Guru
                                User ModeratorsCommunity WarriorsvExpert

                                > sorry fo the inconvenient

                                No need to aplogize - for me this just means that I do not  spend / waste 1 - 2 hours while analysing your chances to recover your data.
                                Maybe I even manage to go to bed before 3 or 4 tonight.


                                > which part of the file you need ?
                                All of it.


                                Datastores that suddenly disappear usually are caused by:

                                - power failure destroys the VMFS metadata which is used to store the location of all vmdk-fragments stored on that datastore

                                - misbehaving RAID-controllers

                                - bad decisions of local admins trying to deal with Raid-controllers error messages or warnings

                                 

                                I use the dump-file to

                                - recover vmx-files and vmdk-descriptorfiles

                                - check if there is any vmfs-metadata left to find the locations of vmdk-fragments (this is required to recover flat, delta and sesparse vmdk - all the big files)


                                If I am successful I provide vmx-files and vmdk-descriptorfiles and create sh-scripts
                                that will extract the lost vmdk-fragments and assembles them to new VMDKs on another datastore.

                                 

                                 

                                Your other options are:

                                - call vmware support - they will tell you that you have a recovery-request and that they do not help with recovery-requests

                                - try commercial recovery tools such as Diskinternals and UFSexplorer (this is the complete list of useful tools - both charge 500 - 1000 dollars and both will have poor results for the first-time user )

                                - call Ontrack - and prepare for a 10.000 dollar bill

                                - call HP-support - may help to fix the current problems but will probably be too late to fix the VMFS-volume.

                                - try freeware recovery tools - none of them is designed for this job so you can only recover files by signatures - typical result when searching for database files: 10.000 false detections, 20.000 corrupt files , handful of good files - but all too old ,,,,

                                - try to google for instructions - well this is probably the worst thing you can do: 99,9 % chance to fail during the first 10 attempts

                                - go to your boss and report that no files have been shared with some freak in germany and that also nobody else will ever be able to read that data as it is finally gone forever.
                                You see - there really is no need to apologize - your decision is only inconvenient for you.

                                 

                                I recommend that you call HP-support next.

                                 

                                Call me via skype if you have further questions.
                                Good night and good luck


                                Ulli