VMware Cloud Community
Mufimufin
Contributor
Contributor

The best way how to swap two hard drives

Dear all,

first of all I do not have any RAID on my home server and I plan to add it later when I will switch to some other solution (matter of years).

One of the non-system hard drives (datastore2) is showing symptomps of a faulty drive. I have also made a few tests and confirmed that (Hirenboot, Linux commands).

I plan to do the following:

a) connect another 10 Tb WD Red NAS hard drive to my ESXi 6.7 server (its already connected to my MSI H61MA-E35 (B3) MB, but ESXi refuses to detect it - that is however another problem I need to solve first)

b) copy the datastore2 content to datastore 3 (the new drive) using the Datastore browser option

c) rename datastore2 to some other name (e.g. goodbye-my-old-hard-drive)

d) rename datastore3 (new hard drive) to datastore2

e) disconnect old failing hard drive

I do not see any vMotion options in my ESXi 6.7 web interface (vSphere) because I do not have any Virtual Center installed.

I therefore believe that I will have to remove the VM from my invetory and re-add it once moved to make ESXi know that it was moved on another hard drive.

Potential important information is that there is another datastorage which is mounted to the same VM as the one on datastore2 and I would like to keep it that way as it contains some data. Does this represent a problem? I think that I will have to re-add it as well once the VM will be removed from the inventory, right?

Thank you.

Reply
0 Kudos
35 Replies
a_p_
Leadership
Leadership

b) The datastore browser may not be the best option for this. I'd go ahead and either export/import as OFV, backup/restore, or copy the VM's files (at least the .vmdk files) manually from the command line using the vmkfstools command line utility. Please be careful if the VM has active snapshot to avoid possible data loss!

In any case, use the "copy" option rather than "move". In case something goes wrong, you will still have the original files. After copying the files, rename the source folder!

c + e) Please take a look at https://kb.vmware.com/s/article/2004605​ for how to properly disconnect a LUN/disk from ESXi.

I therefore believe that I will have to remove the VM from my invetory and re-add it once moved to make ESXi know that it was moved on another hard drive.

Yes, remove the VMs from the inventory prior to copying the files over to the new datstore, and then register the VMs again from the new location.

Btw. datastore names are not that important, they are basically just "friendly" names. Internally, ESXi uses the UUIDs instead, so that you can rename datastores without impacts to the VMs. In case you are moving/copying files which are not located in the VM's home folder, you will need to adjust the VM's settings (absolute paths).

André

Reply
0 Kudos
Mufimufin
Contributor
Contributor

b) The datastore browser may not be the best option for this

I have started the migration yesterday by a simple copy of the VM through the data store explorer. Its just 2 Tb HDD, the procedure is running from 2 AM until now = 10,75 h and its only made 43% progress. This seems extremely slow. I am going to stop that. Hopefuly export/import or vdmk option will be faster than this.

I'd go ahead and either export/import as OFV, backup/restore, or copy the VM's files (at least the .vmdk files) manually from the command line using the vmkfstools command line utility. Please be careful if the VM has active snapshot to avoid possible data loss!

Export OVF option in the web interface seems to download files on my computer and I do not have enough space for such large files at the moment. It also ended on the network error a few seconds after the start. This is therefore not an option.

I have not found any other backup option in the web vSphere interface. Did you meant Veeam backup solution (free option)? This does not seems to work with free ESXi tier that I use. Perhaps I should use Trilead's VM Explorer?


Anyway using vmkfstools command line method seems to be also extremely slow, but at least its working. I am waiting until its finished and will copy the remaining portion of files as described in this thread Move VMware ESXi VM to new datastore - preserve thin-provisioning - Server Fault  .  Not sure why it has to be so cumbersome and why ESXi web interface does not offer anything more user friendly.

In any case, use the "copy" option rather than "move". In case something goes wrong, you will still have the original files. After copying the files, rename the source folder!

That is what I did given to the fact that the original hard drive might be faulty.

c + e) Please take a look at https://kb.vmware.com/s/article/2004605 for how to properly disconnect a LUN/disk from ESXi.

I will use the unmount option in the ESXi vSphere web interface and the physically detach the hard drive once finished.

Yes, remove the VMs from the inventory prior to copying the files over to the new datstore, and then register the VMs again from the new location.

Its unregistered and I will re-register once moved.

Thank you.

Reply
0 Kudos
Mufimufin
Contributor
Contributor

I have moved the VM using vmkfstools and registered through the data store browser. It appeared in the web Vsphere interface.

The problem I face now is that it does not start (Linux) after the power-on. ESXi shows green dot which means that there are most likely no obvious issues on the virtualization part.

By does not start I mean that its stucked on the OS logo being shown after I choose that it should boot normally.

Here is the vmware.log output:

https://hastebin.com/anoqalenid.makefile

Here is the vmkernel.log output:

https://hastebin.com/wocipafiga.sql

When I choose the recovery boot option in GRUB the output looks like this:

2018-08-26_18-55-17.png

I do not really know what I should do now.

Reply
0 Kudos
a_p_
Leadership
Leadership

I have moved the VM using vmkfstools and registered through the data store browser.

The vmkfstools utility only copies the virtual disk file(s). Did you also copy the other files (e.g. the .vmx,...) over to a new folder on the target datastore, and rename the old folder prior to registering the VM from the target location, and powering it on?

André

Reply
0 Kudos
Mufimufin
Contributor
Contributor

The vmkfstools utility only copies the virtual disk file(s). Did you also copy the other files (e.g. the .vmx,...) over to a new folder on the target datastore

Thanks for the answer. Yes, I did using the following command:

  • find "/vmfs/volumes/source_datastore/Some VM" -maxdepth 1 -type f | grep -v ".vmdk" | while read file; do cp "$file" "/vmfs/volumes/destination_datastore/Some VM"; done

and then this one:

  • find "/vmfs/volumes/source_datastore/Some VM" -maxdepth 1 -type f | grep [0123456789][0123456789][0123456789][0123456789][0123456789][0123456789] | grep ".vmdk" | while read file; do cp "$file" "/vmfs/volumes/destination_datastore/Some VM"; done

After that the VM got registered. I had another problem with the file lock but that was caused by vmkfstools clone process being stucked on 100% in another putty instance and I had to kill it. The data size however matches the cloned file (bytes), so it was moved as whole 1,62 Tb file.

and rename the old folder prior to registering the VM from the target location, and powering it on?

No, I have not renamed the old VM folder as it is on another data storage (datastore2 - OLD instead of datastore3 - NEW).

I have did that now:

mv /home/user/oldname /home/user/newname

and tried to re-register the VM on my new data store without copying the files again. Unfortunately the problem still persists.

Was it screwed up just because I have not renamed the folder and should I copy the files again?

Reply
0 Kudos
Mufimufin
Contributor
Contributor

Here is a comparison of the old folder and the new folder.

2018-08-26_19-24-53.png

2018-08-26_19-24-14.png

Some files have different data size, so perhaps I can try to owerwrite them manually?

Also I have clicked on "I moved the VM" and not on "I have made a copy of the VM".

Reply
0 Kudos
Mufimufin
Contributor
Contributor

I have deleted all log files and tried to move all files except 2 vmdk files again (using Winscp) by owerwriting the old files. ESXi asked me whether I have moved or made a copy so this time I choosed the copy option as my answer. When I power up the VM it still does not work.

I am out of ideas and thinking that I might try another option like Trilead's VM Explorer and start over again.

Reply
0 Kudos
a_p_
Leadership
Leadership

From what you write, I think it's likely an issue with the copied virtual disk files.

The "cp" command may work, but is not what you want to use for large files on VMFS datastores.

"vmkfstools" is build to copy virtual disk files (including metadata, and user data) in an optimized way. The target file will be created up front with it's provisioned size, and then the contents will be copied over block by block to avoid the need to resize the file (as cp does it), i.e. modify VMFS metadata. So even if the file sizes matche, the content may not be consistent if vmkfstools was killed.

André

Reply
0 Kudos
Mufimufin
Contributor
Contributor

I have compared the content of the smaller vdmk file (Flawless Server.vmdk) and noticed that there is an additional row in the new one.

ddb.thinProvisioned = "1"

The previous vmdk file did not had such row. Its most likely caused by the command that I have used for the initial file copy:

  • vmkfstools -i "/vmfs/volumes/source_datastore/Some VM/Some VM.vmdk" -d thin "/vmfs/volumes/destination_datastore/Some VM/Some VM.vmdk"

Should I try to delete it or move files again without the thin parameter?

Reply
0 Kudos
Mufimufin
Contributor
Contributor

The "cp" command may work, but is not what you want to use for large files on VMFS datastores.

Ok, so should I use another command instead?

"vmkfstools" is build to copy virtual disk files (including metadata, and user data) in an optimized way. The target file will be created up front with it's provisioned size, and then the contents will be copied over block by block to avoid the need to resize the file (as cp does it), i.e. modify VMFS metadata. So even if the file sizes matche, the content may not be consistent if vmkfstools was killed.

Understood, but what should I do if its stucked on cloning 100% done for an hour? I can do the vmkfstools copy round no. 2.

Do you recommend this step by step guide Revisions to Move VMware ESXi VM to new datastore - preserve thin-provisioning - Server Fault  or I should rather follow something else?

Thank you a lot for your ongoing support. Its greately appreciated.

Reply
0 Kudos
a_p_
Leadership
Leadership

Which target format (thin, or thick) do you want/need?

In case you want to clone the virtual disk as is, omit "-d thin" from the command line.

There are a couple of ways to find out whether a virtual disk has been thin, or thick provisioned. An easy way is to run ls -lisa, and compare the provisioned size (the column left to the time stamp) with the real size in kB (2nd column). If the sizes match, the virtual disk has most likely been thick provisioned.

Understood, but what should I do if its stucked on cloning 100% done for an hour? I can do the vmkfstools copy round no. 2.

I'm not 100% sure about this, but I'd give it a second chance. Maybe it's a result of converting a thick disk to a thin disk!?

André

Reply
0 Kudos
Mufimufin
Contributor
Contributor

Which target format (thin, or thick) do you want/need?
In case you want to clone the virtual disk as is, omit "-d thin" from the command line.

I would rather stick with the original format. It looks that I have added thin just because it was in the step by step guide. I will therefore omit "-d thin" as mentioned above.

There are a couple of ways to find out whether a virtual disk has been thin, or thick provisioned. An easy way is to run ls -lisa, and compare the provisioned size (the column left to the time stamp) with the real size in kB (2nd column). If the sizes match, the virtual disk has most likely been thick provisioned.

Here is the output of ls -lisa (original VM folder):

2018-08-26_20-09-50.png

It looks that the size does not match and it therefore was most likely using the thin format of provisioning.

I'm not 100% sure about this, but I'd give it a second chance. Maybe it's a result of converting a thick disk to a thin disk!?

I will give it one more chance and if that will not work I will most likely try that VM Explorer with a hope that it will allow me to move from one datastore to another without getting my local Windows computer involved in terms of its hard drive.

Reply
0 Kudos
a_p_
Leadership
Leadership

It looks that the size does not match ...

Column 2 is in kB, whereas the provisioned size is in Bytes. Both values show a size of 1,853 GB. So to me it looks like a thick provisioned virtual disk.

André

Reply
0 Kudos
Mufimufin
Contributor
Contributor

It seems to be stucked at 100% again.

pastedImage_0.png

I am not sure if it actually still does something or not.

pastedImage_5.png

It might still be running as there is nothing else on that new hard drive at the moment.

pastedImage_1.png

Unless I cancel this the ESXi web vSphjere VM power-up will complain about the locked file. There are no *.lck files, just this hanging copy process in its 100% stage.

Can this be caused by differences in the data store VMFS versions (5.58 for the old data store and 6.82 for the new one)? I will rather wait for some more time.

Edit: 11:52 (4 more hours) = still running. At least properly working progress would be nice :-/. Its running for about 10,5 hours now (1,82 Tb).

Reply
0 Kudos
Mufimufin
Contributor
Contributor

It finally finished after additional 4 hours (14,5 hours in total for 1,82 Tb), but with an error:

Clone: 100% done.Failed to clone disk: Input/output error (327689).

Here is the vmkernel log output:

https://paste.ee/p/iG8uI

Our new hard drive is:

5000cca273d760d2 (can be found in the vmkernel log above)

Old hard drive is:

5000c5004f06c948

What I have found on the internet is that I should try to export into the OVF template, but in terms of the web vSphere GUI this means that I will have to download it to my local computer first and I do not have that much of a free hard drive space available there. Another tips were that I should use something like ghettoVCB.sh - Free alternative for backing up VM's for ESX(i) 3.5, 4.x & 5.x   , but these are not meant for ESXi 6.7.

Any ideas what should I try to do now?

Reply
0 Kudos
a_p_
Leadership
Leadership

I/O errors may explain why it took so long.

I'd definitely try any possible way to backup, or export the VM at this point. How do you usually backup the VMs?

André

Reply
0 Kudos
Mufimufin
Contributor
Contributor

I do not backup this particular VM as it does not contain anything sensitive in terms of the data, but it will be a pain to set it up again (lots of services). I would therefore rather do what I can do to move it on another datastore first.


I will try the data store transfer through the datastore browser and MicroFocus VMExplorer now unless you have any other recommendations?

Reply
0 Kudos
a_p_
Leadership
Leadership

You may use any backup tool you want, as long as it helps.

If the backup also fails, try to contact continuum​. He's an expert for such cases.

André

Reply
0 Kudos
Mufimufin
Contributor
Contributor

Thanks a.p. .

Well I do not know about any other backup solution that works with ESXi free tier and all sh scripts I have found so far are outdated and not compatible with ESXi 6.7.

So right now:

1) Data store browser - copy button (large file first)

2) MicroFocus VMExplorer

3) I am currious whether it is possible to make OVF file and move it directly to the new data store without getting my local Windows computer involved as a middle man (its hard drive). Something like this https://www.virtuallyghetto.com/2012/06/how-to-copy-vms-directly-between-esxi.html  just not between 2 ESXi servers, but 2 data stores on the same ESXi host.

4) Anything else I will find or that will be suggested here, but hopefuly I will not get into this stage...

Perhaps continuum​ will know what to do.

Reply
0 Kudos