1 2 3 Previous Next 35 Replies Latest reply on Aug 30, 2018 3:09 PM by continuum Go to original post
      • 15. Re: The best way how to swap two hard drives
        Mufimufin Novice

        It finally finished after additional 4 hours (14,5 hours in total for 1,82 Tb), but with an error:

         

        Clone: 100% done.Failed to clone disk: Input/output error (327689).

         

        Here is the vmkernel log output:

        https://paste.ee/p/iG8uI

         

        Our new hard drive is:

        5000cca273d760d2 (can be found in the vmkernel log above)

         

        Old hard drive is:

        5000c5004f06c948

         

        What I have found on the internet is that I should try to export into the OVF template, but in terms of the web vSphere GUI this means that I will have to download it to my local computer first and I do not have that much of a free hard drive space available there. Another tips were that I should use something like ghettoVCB.sh - Free alternative for backing up VM's for ESX(i) 3.5, 4.x & 5.x   , but these are not meant for ESXi 6.7.

         

        Any ideas what should I try to do now?

        • 16. Re: The best way how to swap two hard drives
          a.p. Guru
          Community WarriorsUser ModeratorsvExpert

          I/O errors may explain why it took so long.

          I'd definitely try any possible way to backup, or export the VM at this point. How do you usually backup the VMs?

           

          André

          • 17. Re: The best way how to swap two hard drives
            Mufimufin Novice

            I do not backup this particular VM as it does not contain anything sensitive in terms of the data, but it will be a pain to set it up again (lots of services). I would therefore rather do what I can do to move it on another datastore first.


            I will try the data store transfer through the datastore browser and MicroFocus VMExplorer now unless you have any other recommendations?

            • 18. Re: The best way how to swap two hard drives
              a.p. Guru
              vExpertCommunity WarriorsUser Moderators

              You may use any backup tool you want, as long as it helps.

              If the backup also fails, try to contact continuum. He's an expert for such cases.

               

              André

              • 19. Re: The best way how to swap two hard drives
                Mufimufin Novice

                Thanks a.p. .

                 

                Well I do not know about any other backup solution that works with ESXi free tier and all sh scripts I have found so far are outdated and not compatible with ESXi 6.7.

                 

                So right now:

                1) Data store browser - copy button (large file first)

                2) MicroFocus VMExplorer

                3) I am currious whether it is possible to make OVF file and move it directly to the new data store without getting my local Windows computer involved as a middle man (its hard drive). Something like this https://www.virtuallyghetto.com/2012/06/how-to-copy-vms-directly-between-esxi.html  just not between 2 ESXi servers, but 2 data stores on the same ESXi host.

                4) Anything else I will find or that will be suggested here, but hopefuly I will not get into this stage...

                 

                Perhaps continuum will know what to do.

                • 20. Re: The best way how to swap two hard drives
                  continuum Guru
                  Community WarriorsvExpertUser Moderators

                  I do not have the full background here ...
                  Are you trying to copy a VMDK from one datastore on host A to another datastore on host B wile the vmdk fights back and reports I/O errors ?
                  If yes do this:
                  create a small Linux VM one one of those hosts - for example using this ISO:: http://sanbarrow.com/livecds/moa64-nogui/MOA64-nogui-incl-src-111014-efi.iso

                  then boot into it and create 2 directories:
                  mkdir /vmfs-in
                  mkdir /vmfs-out
                  Then connect to the source ESXi like this:
                  sshfs -o ro root@esxi-A:/vmfs/volumes/source-datastore /vmfs-in
                  Connect to the target ESXi like this:
                  sshfs root@esxi-B:/vmfs/volumes/target-datastore /vmfs-out
                  Once that is done you can use ddrescue like this
                  ddrescue /vmfs-in/source-directory/name-flat.vmdk /vmfs-out/target-directory/name-flat.vmdk   /vmfs-out/target-directory/name-copy.log
                  This way you will have readonly access on the source and can write to the target.
                  ddrescue will help to skip I/O errors - make sure to write the log-file ...
                  If you only have one host then use the same approach: mount one datastore readonly and the target datastore in writeable mode.
                  This is NOT the fastest approach.
                  But if it fails now due to I/O errors it is the easiest way to workaround I/O errors.
                  Feel free to call via skype if you can not handle this ....
                  Ulli

                   

                  By the way - the OVA/OVF workaround will fail if there is only one small error in the source-file.
                  ddrescue should not fail even with multiple errors

                  • 21. Re: The best way how to swap two hard drives
                    Mufimufin Novice

                    Thank you for the answer continuum .

                     

                    Are you trying to copy a VMDK from one datastore on host A to another datastore on host B wile the vmdk fights back and reports I/O errors ?

                    As mentioned in my first post I am trying to move one VM from host A and datastore A to host A and datastore B. In other words I need to move my VM from one hard drive to a new hard drive on the same computer with ESXi 6.7. I totally understand that there is lots of content in this thread now, so I hope that this will clear things up for you.

                     

                    Anyway it does not matter now if that will be VDMK or OVF or any other format if that solution is going to work.

                     

                    If you only have one host then use the same approach: mount one datastore readonly and the target datastore in writeable mode.

                    This is NOT the fastest approach.

                    But if it fails now due to I/O errors it is the easiest way to workaround I/O errors.

                    So if I understand it correctly:

                     

                    1) I will connect to ESXi using SSH and run this command in the terminal:

                    ddrescue /source-data-store/source-directory-VM/name-flat.vmdk /target-data-store/target-directory-VM/name-flat.vmdk   /target-data-store/target-directory-VM/name-copy.log

                    2) After that I will copy the remaining files from the old data store (e.g. via data store browser or Winscp) to the new data store.

                    3) Rename the VM on the old data store (not sure if this is even necessary, but it was mentioned everywhere).

                    4) Register the VM from the new data store.

                     

                    Please let me know if I have missed something.

                     

                    Thank you - I will try it as soon as the current copy through the data store browser fails/finishes.

                    • 22. Re: The best way how to swap two hard drives
                      continuum Guru
                      vExpertCommunity WarriorsUser Moderators

                      Keep in mind that ddrescue does not exist in ESXi.
                      And if you really have to deal with VMDKs with I/O errors it is essential to mount the source datastore in readonly mode.
                      I use ddrescue for flat and delta vmdks. All other related files of a VM are so small that you can use cp.
                      For a quicktest if a flat.vmdk or delta.vmdk is healthy simply run
                      vmkfstools -p 0 name-flat.vmdk > mapping.txt
                      If that works you can assume that there are no I/O errors in that file.
                      If a vmkfstools -i command hangs at 100 % wait upto one or two hours.
                      Before you give up and kill the command first assume that it already did the job and check the output vmdk.
                      Try to dump the last 10 MBs to a new file and inspect that with hexdump -C.
                      If you see all zeroes then the vmkfstools -i command probably failed too early.

                      • 23. Re: The best way how to swap two hard drives
                        Mufimufin Novice

                        If it does not exist on ESXi (not sure if it can be installed) then I will connect from the inner Linux VM to ESXi as mentioned by you above with a difference that both folders will be mapped on one host instead of two. I believe that this should do the job.

                         

                        I will also try that flat/delta check once the current browser datastore copy process attempt will be finished.

                         

                        Thanks.

                        • 24. Re: The best way how to swap two hard drives
                          Mufimufin Novice

                           

                                   I will continue with the recovery procedure above now.

                          • 25. Re: The best way how to swap two hard drives
                            Mufimufin Novice

                             

                            I know that you have mentioned this not being the fastest option, but is there any way how to speed things up? Waiting 2 days seems extreme to me given to the fact that this is just 1,82 Tb vmdk file.

                             

                            Thanks.

                            • 26. Re: The best way how to swap two hard drives
                              Mufimufin Novice

                              Here is the mapping.txt file output:

                              [           0:         517] --> [VMFS -- LVID:50eeddd7-bf71cab4-d5f7-8c89a57ce0bc/50eeddd7-ae2abe54-5a98-8c89a57ce0bc/1:(    54821888 -->     54822405)]

                               

                              I have used the following command:

                              vmkfstools -p 0 Flawless\ Server.vmdk > mapping.txt

                               

                              It was quite fast (2 seconds).

                              • 27. Re: The best way how to swap two hard drives
                                continuum Guru
                                User ModeratorsvExpertCommunity Warriors

                                vmkfstools -p 0 Flawless\ Server.vmdk > mapping.txt
                                When you use that command against the descriptorfile it will only display a single line.
                                This command is basically useless as we already know the content of the descriptor file.
                                We expect to have the I/O error inside the flat.vmdk - not the descriptor.
                                So use the command against the flat.vmdk
                                vmkfstools -p 0 Flawless\ Server-flat.vmdk > mapping.txt
                                This will result in a mapping.txt with lots of lines - I have seen upto 1 million lines for a heavily fragmented thin provisioned flat.vmdk.
                                Regarding the poor performance ....
                                2 days for a 2TB flat.vmdk is really very slow.
                                I normally would expect about 12 - 24 hours for a 2TB file.
                                But that probably means that the source file has several I/O errors.
                                ddrescue can slow down to a crawl around the location of I/O errors - but if that is the only option to clone the file at all you probably have to live with that.
                                I started to write a blog post about issues like this - see
                                VMFS-volumes with I/O-errors | VM-Sickbay

                                • 28. Re: The best way how to swap two hard drives
                                  Mufimufin Novice

                                  You are right. I have completely missed that as I was thinking that the correct larger file is being checked. See annex for the new output (mapping.txt). I am not sure what I should be looking at in that file.

                                   

                                  Regarding the slow performance - here is the current progress:

                                  I do not see any errors so far.

                                   

                                  Copy log:

                                   

                                   

                                  However when I see almost 2 days of remaining time I am starting to think whether it would not be easier to simply ditch the current VM, install new one and move files through SSH (server x server).

                                  • 29. Re: The best way how to swap two hard drives
                                    Mufimufin Novice

                                    This i crazy slow. Yesterday the ETA was 2 + days and today its again 2 + days.

                                     

                                    I am starting to think whether this is not going through the local network since I have mapped those folders to local IP netwrok addresses on the same machine. Its however in the same pool so it should be going directly. But if that would be going through the network I may cancel, disconnect the powerline adapter and connect directly to the router.

                                     

                                    2018-08-29_10-14-25.png