ESXi 5.0 scp error when copying vmdk files from ESXi 4.0 server

[loligator]

Hey guys,

This one has me stumped.  I built an ESXi 5.0 server and need to migrate all of my VMs off of my old ESXi 4.0 server.  The vmdk files I've tried to copy with scp will copy for about an hour before they fail with an Input/output error.  All of the other files in the VMs folder on the ESXi 4 server copied successfully; it's just the vmdk files that won't copy.  I've tried serveral VMs.  One had a 20 GB vmdk while the third has an 8 GB.  My ESXi server is running build 515841.   

Does anyone have any ideas of what could be wrong?

Thanks

[Troy_Clavell]

I don't have a direct answer.  However you may try using VMware vCenter Converter Standalone instead.

  Also, maybe give FastSCP a try, if you aren't already using it.

[loligator]

Hi Troy,

Thanks for the suggestions.  Maybe I'm wrong, but it doesn't look like the VMware vCenter Convertor Standalone does what I need.  From what I saw, it only makes physical machines into virtual.  I downloaded Veeam's FastSCP and it did pretty much the same thing... it ran for about an hour until it errored out saying the connection timed out.  Maybe the SSH session has a timeout value and even though scp (or FastSCP) is running, the connection still times out?  Is this even possible?  I'm a Windows Admin by trade so I know only know enough about unix to be dangerous. 

Thanks,

Tim

[psplnalin123]

VMware converter is also doing V2V conversion.  As you said that copying of vmdk is taking too long hours might it is possible that the VM is power on either you could down the VM and copy the vmdk or do V2V

There are other ways to do also.

1. Take a backup using VCB http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100297... and restore the VM using vmware converter to your new esxi box

2. If you have a vCenter server if not you can download the evaluation version of vCenter Server 5.0 and add both hosts  4.0 anbd 5.0 in vCenter Server. Create vmkernel port and power off the VM and migrate the VM.

[loligator]

I plan to give this a go today and will post on how it plays out.

Thanks

[loligator]

Unfortunately, VMware vCenter Converter Standalone failed as well with the error: 'The operation experienced a network error'.  Where do I go from here?  Are there network logs on the ESXi host that I can take a look at to see why all these operations are failing with some sort of a network error?

[Troy_Clavell]

sounds to me there are some networking issues you need to work out.  I don't believe this is a VMware related problem.

[continuum]

> The vmdk files I've tried to copy with scp will copy for about an hour before they fail with an Input/output error.

do you see AIOMGR error type 3 in the vmware.log of that VM ?

[loligator]

I'll have to check the logs.  What would that error being present mean? 

I've spent the past few days troubleshooting different network components and settings.  I'm now using a new switch and ethernet cables.  I am still unable to copy any vmdk files as scp, FastSCP, and the VMware tool recommended all still error out with the same errors as before. I haven't tried replacing the NIC yet but maybe that will be next.  I've also messed with the duplex settings, changing it from Full 100 Mbps to half but this did not help.  Does anyone else know where I could possibly go with this?  It basically seems like the connection is being reset while copying large files.  The copy job will get to about 50% (roughly 9 to 10 GB copied) before erroring out.  

[continuum]

if you see that error you have a real problem

[loligator]

So I successfully got the vmdk file to copy over the network!  My ESXi server has two internal disks, therefore having two datastores.  I was attempting to copy the vmdk to datastore2, the one that houses all VMs.  Once I copied it to datastore1, scp completed without error.  I then tried to mv the vmdk file to datastore2 and the dreaded input/output error came back.  It's looking like there may be an issue with the disk datastore2 is on.  This hard drive was originally in my ESXi 4.0 server but I moved it to the new ESXi 5.0 server.  I didn't reformat it; ESXi 5.0 was able to import it and the VMs already on this disk (there's five) were all added to inventory and work without issue.  Literally up until the day I moved it I had a backup script running that copied vmdk files off of the disk and I had no issues with scp and input/output errors when it was still in the old ESXi 4.0 server.  Why would the disk suddenly have issues now that I moved it to a new server?  Is there a way I can check the integrity of the disk?  Something like chkdsk but for ESXi?

Thanks

[loligator]

I also tried using the built in datastore browser and the VM is powered off but the move still fails.  Does anyone know how I can check the disk for errors?

[continuum]

you can try to rescue the files with this LiveCD I made
http://communities.vmware.com/message/1877235#1877235

[loligator]

So that's it?  I have to resort to recovering the data now?  There's no way to actually test the drive?

[loligator]

I was able to resolve the issue and can now scp vmdk files from the ESXi 4 server to the 5.  I had to reformat the datastore that I migrated from my ESXi 4 server.  I put the hard drive containing the datastore back into my ESXi 4 server.  I backed up all VMs on the datastore.  I put the datastore back in the new ESXi 5 server.  I reinitialied the disk with the RAID controller's BIOS, deleting all data on the volume.  I then booted up ESXi 5 and added the disk, formatting it with VMFS 5.  Files now copy without issue.

[loligator]

Refer to the last post I made in the thread. Basically, I had to reformat the datastore and now all works as expected.