14 Replies Latest reply on May 27, 2020 10:45 AM by GOBrian01

    ESXi Datastore Inaccessible

    GOBrian01 Novice

      Had issues with a server running ESXi.  I believe it was 5.5 but not 100% sure.  Reloaded 6.5 on the esxi partition and all went smooth.  However, I can not see our datastore after booting into esxi.  I decided to load 5.5 on the disk.  Again, same thing.

       

      What I can see is that a partition exists.  This was stored on a RAID6 array of 4 ~1TB disks.  The usable size was 1.82 TB to vmware.  I know the datastore had been utilized 100%.  For this reason, I slapped another drive into the array and reconstructed.  Hoping maybe the full disk was causing an issue.

       

      I now have a 2.73TB array and show the 1.82TB partition.  I know without expanding it, it may not have helped yet, but this is where we start to get where I am unsure and scared to lose the datastore. I did read several articles and eventually found I was suffering from this error.  I ran the partedutil fix command to resolve.  But my datastore is still not showing up   Im stuck and now reaching out for any advice.  A little wary to just start trying things as I may make it worse!    Thank you anyone that may be able to advise me.

       

      1. partedUtil getptbl /vmfs/devices/disks/[disk_ID]

      Error: The backup GPT table is not at the end of the disk, as it should be.  This might mean that another operating system believes the disk is smaller.  Fix, by moving the backup to the end (and removing the old backup)?
      Warning: Not all of the space available to /dev/disks/naa.6a4badb021e1ae001eea2a790847ba7d appears to be used, you can fix the GPT to use all of the space (an extra 584843264 blocks) or continue with the current setting? This will also move the backup table at the end if is is not at the end already. diskSize (1754529792) AlternateLBA (1169686527) LastUsableLBA (1169686494) NewLastUsableLBA (1754529758)

       

       

        • 1. Re: ESXi Datastore Inaccessible
          a.p. Guru
          User ModeratorsvExpertCommunity Warriors

          Welcome to the Community,

           

          does the error that you've posted still occur after you ran the partedUtil fixGpt "/vmfs/devices/disks/<Device>" command?

           

          Please post (as text please) the output for the following commands:

          1. ls -lisa /vmfs/devices/disks
          2. esxcli storage vmfs snapshot list
          3. the offset= ...; done  command from Step 1 at https://kb.vmware.com/s/article/2046610
          4. partedUtil getptbl "/vmfs/devices/disks/<Device>" (if the output is different after the partedUtil fixgpt command)
          5. partedUtil getUsableSectors "/vmfs/devices/disks/<Device>"

           

          What you may also want to do is to run vmkfstools -V (note the upper-case "-V"), and then look for issues reported in /var/log/vmkernel.log.

           

          André

           

          PS: You wrote "Had issues with a server running ESXi." . Can you please provide some details about this issue?

          1 person found this helpful
          • 2. Re: ESXi Datastore Inaccessible
            GOBrian01 Novice

            Thank you André for the reply.

             

            After I ran the partedUtil fix command, I now get the output that should occur.

             

            gpt

            364602 255 63 5857345536

            1 2048 3904896990 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

             

            As for your other questions.

            1.

            total 4900023185

                  4      0 drwxr-xr-x    1 root     root           512 May 24 09:12 .

                  1      0 drwxr-xr-x    1 root     root           512 May 24 09:12 ..

                140 15360000 -rw-------    1 root     root     15728640000 May 24 09:12 mpx.vmhba32:C0:T0:L0

                128   4064 -rw-------    1 root     root       4161536 May 24 09:12 mpx.vmhba32:C0:T0:L0:1

                130 255984 -rw-------    1 root     root     262127616 May 24 09:12 mpx.vmhba32:C0:T0:L0:5

                132 255984 -rw-------    1 root     root     262127616 May 24 09:12 mpx.vmhba32:C0:T0:L0:6

                134 112624 -rw-------    1 root     root     115326976 May 24 09:12 mpx.vmhba32:C0:T0:L0:7

                136 292848 -rw-------    1 root     root     299876352 May 24 09:12 mpx.vmhba32:C0:T0:L0:8

                138 2621440 -rw-------    1 root     root     2684354560 May 24 09:12 mpx.vmhba32:C0:T0:L0:9

                125 2928672768 -rw-------    1 root     root     2998960914432 May 24 09:12 naa.600605b005c598201face2e020d41d61

                123 1952447471 -rw-------    1 root     root     1999306210816 May 24 09:12 naa.600605b005c598201face2e020d41d61:1

                141      0 lrwxrwxrwx    1 root     root            20 May 24 09:12 vml.0000000000766d68626133323a303a30 -> mpx.vmhba32:C0:T0:L0

                129      0 lrwxrwxrwx    1 root     root            22 May 24 09:12 vml.0000000000766d68626133323a303a30:1 -> mpx.vmhba32:C0:T0:L0:1

                131      0 lrwxrwxrwx    1 root     root            22 May 24 09:12 vml.0000000000766d68626133323a303a30:5 -> mpx.vmhba32:C0:T0:L0:5

                133      0 lrwxrwxrwx    1 root     root            22 May 24 09:12 vml.0000000000766d68626133323a303a30:6 -> mpx.vmhba32:C0:T0:L0:6

                135      0 lrwxrwxrwx    1 root     root            22 May 24 09:12 vml.0000000000766d68626133323a303a30:7 -> mpx.vmhba32:C0:T0:L0:7

                137      0 lrwxrwxrwx    1 root     root            22 May 24 09:12 vml.0000000000766d68626133323a303a30:8 -> mpx.vmhba32:C0:T0:L0:8

                139      0 lrwxrwxrwx    1 root     root            22 May 24 09:12 vml.0000000000766d68626133323a303a30:9 -> mpx.vmhba32:C0:T0:L0:9

                126      0 lrwxrwxrwx    1 root     root            36 May 24 09:12 vml.0200000000600605b005c598201face2e020d41d614d5239323630 -> naa.600605b005c598201face2e020d41d61

                124      0 lrwxrwxrwx    1 root     root            38 May 24 09:12 vml.0200000000600605b005c598201face2e020d41d614d5239323630:1 -> naa.600605b005c598201face2e020d41d61:1


            2.

            esxcli storage vmfs snapshot list does not output anything

             

            3.

            /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0

            gpt

            1912 255 63 30720000

            1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128

            5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

            6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

            7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0

            8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

            9 1843200 7086079 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0

            ---------------------

            /vmfs/devices/disks/naa.600605b005c598201face2e020d41d61

            gpt

            364602 255 63 5857345536

            1 2048 3904896990 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

            Checking offset found at 2048:

            0200000 d00d c001

            0200004

            1400000 f15e 2fab

            1400004

            0140001d  4d 69 6c 66 6f 72 64 44  61 74 61 73 74 6f 72 65  |MilfordDatastore|

            0140002d  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

             

            4.

            gpt

            364602 255 63 5857345536

            1 2048 3904896990 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

             

            5.

            34 5857345502

             

            I did actually also try the vmkfstools -V without any output.   Seemed to just take.  Downloading and viewing in a text editor, it didn't show anything that stood out to me as unusual, but I admittedly, I don't have a ton of experience outside of normal working operation of this stuff.  I attached the file as well.

             

            Lastly, you asked what issues prompted all this to begin with.  Simply, ESXi was hanging at boot.  It was booting off a completely different drive aside of the array.  I have dealt with this recovery in the past.  Simply reloading over that and then importing the inventory after reload from the datastore.  I have had to install drivers before to get the array to show up at all, but if I recall correctly, as long as the device and array shows, I am good.  I dont remember having to deal with the drivers after a reload unless the array is not seen at all.

             

             

            Thank you again for the reply.  Hopefully some of this may shed some light and help determine the issue.  Very grateful for your assistance!

             

            Brian

            • 3. Re: ESXi Datastore Inaccessible
              GOBrian01 Novice

              I am so sorry.. I just saw you asked me to post the stuff in text.. I apologize.. I will edit the post to be so now

              • 4. Re: ESXi Datastore Inaccessible
                a.p. Guru
                Community WarriorsvExpertUser Moderators

                Does esxcli storage filesystem list produce an output for the VMFS LUN?

                 

                André

                • 5. Re: ESXi Datastore Inaccessible
                  GOBrian01 Novice

                  Ughhh..

                   

                  Error getting data for filesystem on '/vmfs/volumes/581a2c28-24aba208-abd5-0025908813ae': Unable to get FS Attrs for /vmfs/volumes/581a2c28-24aba208-abd5-0025908813ae, skipping.

                   

                  This is what I get to that command.

                  • 6. Re: ESXi Datastore Inaccessible
                    a.p. Guru
                    Community WarriorsUser ModeratorsvExpert

                    It may be related to the VMFS version and the fact that you are currently running ESXi 5.5 U1, but I can't tell you for sure. It could however as well be a driver, or RAID controller issue!?

                    So at this point - instead of making thing even worse with trial&error - I'd recommend that you try to contact continuum who may be able to help.

                     

                    One command, which may help with troublshooting the issue is:

                    hexdump -C /dev/disks/naa.600605b005c598201face2e020d41d61:1 -n 16636 -s 0x100200 | grep -C 2 '[a-zA-Z0-9]\{4\}-[a-zA-Z0-9]'

                     

                    André

                    • 7. Re: ESXi Datastore Inaccessible
                      GOBrian01 Novice

                      Thank you for the help.  I will reach out to him.  I've seen him pop up quite a bit in my searches before posting myself.  I appreciate your help.

                      • 8. Re: ESXi Datastore Inaccessible
                        GOBrian01 Novice

                        Unfortunately I have no heard back.  If anyone has any advice or suggestions, it would be greatly appreciated!

                        • 9. Re: ESXi Datastore Inaccessible
                          continuum Guru
                          Community WarriorsvExpert

                          > Unfortunately I have no heard back.

                          I have not seen you in skype ... ?

                          Anyway - if you want help read Create a VMFS-Header-dump using an ESXi-Host in production | VM-Sickbay

                          If you provide a dump like that I may be able to help.

                          Ulli

                          • 10. Re: ESXi Datastore Inaccessible
                            GOBrian01 Novice

                            I apologize.  I have skype, but didnt know to reach out to you that way.  I was attempting to use the forum messaging system.

                             

                            Unfortunately, i didnt get very far with this.  I do not have another datastore available but can put a drive in and make one.  This is remote though so it would take some time to get out there and do so.  Is this the error i would be getting if I have not enough space.  I have space it appears.  Thank you very much for your response.

                             

                            Ramdisk                   Size      Used Available Use% Mounted on

                            root                       32M      508K       31M   1% --

                            etc                        28M      124K       27M   0% --

                            tmp                       192M        2M      189M   1% --

                            vsantraces                300M       24K      299M   0% --

                            hostdstats                413M        1M      411M   0% --

                             

                            /dev/disks # ls -lisa | grep -v vml

                            total 4900023185

                                  4      0 drwxr-xr-x    1 root     root           512 May 26 17:46 .

                                  1      0 drwxr-xr-x    1 root     root           512 May 26 17:46 ..

                                140 15360000 -rw-------    1 root     root     15728640000 May 26 17:46 mpx.vmhba32:C0:T0:L0

                                128   4064 -rw-------    1 root     root       4161536 May 26 17:46 mpx.vmhba32:C0:T0:L0:1

                                130 255984 -rw-------    1 root     root     262127616 May 26 17:46 mpx.vmhba32:C0:T0:L0:5

                                132 255984 -rw-------    1 root     root     262127616 May 26 17:46 mpx.vmhba32:C0:T0:L0:6

                                134 112624 -rw-------    1 root     root     115326976 May 26 17:46 mpx.vmhba32:C0:T0:L0:7

                                136 292848 -rw-------    1 root     root     299876352 May 26 17:46 mpx.vmhba32:C0:T0:L0:8

                                138 2621440 -rw-------    1 root     root     2684354560 May 26 17:46 mpx.vmhba32:C0:T0:L0:9

                                125 2928672768 -rw-------    1 root     root     2998960914432 May 26 17:46 naa.600605b005c598201face2e020d41d61

                                123 1952447471 -rw-------    1 root     root     1999306210816 May 26 17:46 naa.600605b005c598201face2e020d41d61:1

                            /dev/disks # dd if=/dev/disks/naa.600605b005c598201face2e020d41d61:1 bs=1M count=1536 of=/tmp/gobrian01.1536

                            dd: /dev/disks/naa.600605b005c598201face2e020d41d61:1: Input/output error

                            /dev/disks # dd if=/dev/disks/naa.600605b005c598201face2e020d41d61:1 bs=1M count=1536 | gzip -c > /tmp/gobrian01.1536

                            dd: /dev/disks/naa.600605b005c598201face2e020d41d61:1: Input/output error

                            /dev/disks #

                            • 11. Re: ESXi Datastore Inaccessible
                              continuum Guru
                              Community WarriorsvExpert

                              You do not get the I/O error because of insufficient amount of free space to write to.

                              You get the I/O error because of problems on the source drive.

                              Do you remember after how many copied MBs the error occured ?
                              Anyway - we will very likely need to boot the system into Linux - as ESXi alone can not handle I/O errors as good as Linux.

                              If you are forced to work on a remote system to access the ESXi I would suggest that we discuss the next steps via skype so that we can avoid unnecessary actions.

                              You should be able to find me in skype by searching for user "sanbarrow"

                              Ulli

                              • 12. Re: ESXi Datastore Inaccessible
                                GOBrian01 Novice

                                In order to boot into a separate linux boot, I assume I will need to visit the site and add a device whether it be flash drive, cd drive (usb), etc to boot.. I can get this prepared.  What version of Linux should I obtain and prepare for booting into?  I also just sent you a skype contact request.  Thank you so much for taking the time to reply and assist.  I apologize for the slow responses as I am juggling other stuff as well..

                                 

                                Thanks again!

                                • 13. Re: ESXi Datastore Inaccessible
                                  continuum Guru
                                  Community WarriorsvExpert

                                  Hi

                                  I have not seen your skype contact request yet - maybe send me your skype-name in a PM ?

                                  For first analysis download this iso: Live-CD fur Notfallrettung verlorener Daten

                                  Ulli

                                  • 14. Re: ESXi Datastore Inaccessible
                                    GOBrian01 Novice

                                    I am using skype for business not sure if that matters.. I will get this ISO burned and in the server so it is something we can boot off of.  I will do that tomorrow at some point.  Thanks!  In the meantime, I will reach out w my personal skype account as well.. Maybe since I am using skype for business, something is hanging up.