Downloading files from Datastore - i/o error occurred

[daunce]

Using ESX 3.5, and using the Virtual Infrastructure client, I am able to browse the datastore of our SAN, and upload files all ok, but when I try and download any files, after 10-20 seconds, the file stops with a message "i/o error occurred".

Tried several attempts, some take longer than others to get the error message, but it's always the same result.

Any ideas?

Thanks.

[kjb007]

Check your vmkernel log to see if there are errors generated when your download fails.

-KjB

[daunce]

Check your vmkernel log to see if there are errors generated when your download fails.

-KjB

I've checked /var/log/messages and /var/log/vmkernel and there's no entries for this. tried watching it with tail -f.

Not sure where else i should look. ?:|

[kjb007]

Are you trying to download files that are in use? Such as a vm that is up and running? Try uploading a file, and then downloading the same file.

-KjB

[Joshua_Mally]

First, check the permissions/privileges granted to the user you have logged in with. Do you have privilege to do File management in a Datastore??

Try copying/moving the files e.g vmware log files, between folders in the same datestore and check if that works. Also try downloading the file from the service console instead of VI client.

[daunce]

I have admin rights.

I tried downloading a file from a different datastore (local storage of a host) and that worked.

Seems it's just files on the SAN.. local host storage and other NFS shares seem fine.

So for now it's a slow scp to get the files.

Thanks for your help.

[kjb007]

Are you connecting directly to ESX, or through virtual center using the vi client. If vc, can you connect using vi client to ESX using root, and see if download is successful?

-KjB

[mhribljan]

I have the same problem using ESX server 3.5 VI client and trying to download files from virtual storage with datastore browser to local machine. I/O error after few seconds. Logged in with root account. Anybodu have any idea?

[lstcuser]

We have the same problem. Files smaller than 300mb could be up-/downloaded in 9 of 10 cases. Files between 300-1000mb succeeded 8/10 in downloading, 2/10 in uploading. Files >= 1gb almost never succeed. There are also no entries in any logfile.

We use a Tyan- Mainboard with 8GB- RAM an two AMD- Quadcores, ESX v.3.5 including all current updates. Networking from Client to ESX contains two "Zyxel GS-105A" switchs and a "HP ProCurve 2824". All NICs and switches support GB- ethernet. The storage I try to up/download to/from is a SATA- RAID connected through a PCI-E RAID- Controller.

Do you use a similiar hardware setup? Any solutions so far?

[mhribljan]

Same for me. Small files copy just fine. Big ones - no way. IBM 3650 server. 2 x Xeon 5405, 18GB RAM, 6 x 145GB SAS HDD. Linksys 24 port Gigabit switch.

[matthewk]

Same issue on ESXi.

This is a problem for the free version since without VC, the datastore browser is the only supported way to transfer files.

I can trigger the I/O error by doing anything that is slightly disk intensive on the client. I can kill the connection by simply web browsing. Something as mundane as clearing my Internet Cache or a virus scan would kill it, too. Here's my hostd.log:

2008-08-14 19:00:49.341 'BaseLibs' 49156 info NfcFile_ContinueSend: about to send 262144 bytes from file

2008-08-14 19:00:49.539 'BaseLibs' 49156 warning NfcNetTcpWrite: bWritten: -1

2008-08-14 19:00:49.540 'BaseLibs' 49156 warning NfcNet_Send: requested 262144, sent only -1 bytes

2008-08-14 19:00:49.540 'BaseLibs' 49156 warning NfcFileSendMessage: data send failed:

2008-08-14 19:00:49.540 'BaseLibs' 49156 warning NfcFile_ContinueSend: failed to send next file portion

2008-08-14 19:00:49.540 'BaseLibs' 49156 warning NfcServerLoop: failed to continue send

2008-08-14 19:00:49.540 'Nfc' 49156 error Nfc_AuthdServerLoop saw error: Network error -- Failed to send data message

2008-08-14 19:00:49.540 'BaseLibs' 49156 warning NfcNetTcpWrite: bWritten: -1

2008-08-14 19:00:49.540 'BaseLibs' 49156 warning NfcNet_Send: requested 264, sent only -1 bytes

2008-08-14 19:00:49.540 'BaseLibs' 49156 warning NfcSendMessage: send failed:

2008-08-14 19:03:11.660 'App' 147466 error Failed to send response to the client: Broken pipe

Uploading works fine, however.

[xamdam]

I have the same problem: I/O error after a while downloading virtual disks (of stopped VMs) which are greater than 800Mb.

I have an E1000 adapter, quad core q6600, LSI Sas RAID-1 with 2 500gb disks, 6GB ram, ESXi, using root as account.

Using the web browser I can download the same file, though...(BTW..is there a way to upload something by web?).

I'm rather worried becase I just wanted to backup some VMs quickly, and it seems very complicated.

[kenebert]

Same issue here as well on ESXi on multiple servers. This has not been an issue for us internally, but for our partners that are trying out the ESXi product (combined with the Aug 12 lockout bug), this has created some hesitation to adopt.

To any others that need the assistance until this issue is corrected, the VMs (as a whole) can be moved more efficiently (than the web interface) using the VMWare Converter to a local filesystem.

[mschipper]

We are also experiencing the same issue.

ESX 3.5 U1, VC 2.5

IBM HS21 XM Blades, 2x Xeon 5345's, 32 GB's.

Dual Emulex FC HBA's, SAN Attached to IBM SVC 2145, DS4800 Back End, through Brocade switches, max four paths.

Dual Gigabit Broadcom NIC's Bound in 802.3ad Link Agg with VLAN Trunking.

[matthewk]

Its strange to me that this problem even exists. I can download files fine from the web browser. I can use the remote command line. Both of those methods use a simple HTTP GET. I don't understand why the VI Client doesn't use HTTP GET and PUT. Maybe they'll rethink that in a future VI Client update.

[xamdam]

Does anybody know if there's a way to submit this (assumed) bug to vmware, and have some feedback that it will be fixed in the future ?

Best,

Max

[dennispedersen]

Same story here.

HP DL 380 G5 with internal storage on the P400 ctrl.

Does anyone had any success with another free type of backup?

I've installed a freenas server and nfs mounted a new backup store. But its a bit overkill with another server just for backup

Any luck on getting scp to work? (Winscp does not work out of the box and Veeam FastSCP does not support esxi yet..)

[ethergnome]

I have this problem downloading a 26GB vmdk from a 3.5i U2, DL380 G3 4GB, local 15K SCSI in RAID 10 to an XP workstation with single SATA drive. However, when I put the VI client on a DL385 2003 R2 32bit, 16GB, 15K SCSI RAID5, the download succeeded. It's almost like the destination has to have storage fast enough to keep up with the source's ability to fill a gig pipe, otherwise it will fail. Sounds silly, but...

[stpeev]

Is there a person from this thread ready to volunteer and do the following two things:

1) Start the following tool on the machine where the VMware Client is running: (this is the tool page ).

In case of the Datastore browser upload/download it may report some low

level windows error code. Just run the tool, then start the UI, go to

the Datastore browser and do the usual things that you do. When the IO

error occurs, look at the tool and check if there are some messages

with the following format ' blabla .... failed: last error = XXXXXX.

2) Also if possible look if there is a copy of libeay32.dll (and

ssleay32.dll) on the machine other than the one in the Virtual

Infrastructure Client\2.5 folder. If that is the case, you may try

temporarily renaming or temporarily removing the other copies and check

if the upload/download problem still persists.

Please, if you do the steps above, share the outcome.

[bladealslayer]

Hi,

Same problem here, though with uploading. Here is the output of the debugger:

00000000 0.00000000 DLSYM: Failed to resolve FIPS_mode_set: 127

00000001 0.00008800 DLSYM: Failed to resolve FIPS_mode: 127

00000002 0.00014974 DLSYM: Failed to resolve SHA256: 127

00000003 0.00186616 DLSYM: Failed to resolve SHA512: 127

00000004 0.00193963 DLSYM: Failed to resolve EVP_sha224: 127

00000005 0.00199942 DLSYM: Failed to resolve EVP_sha256: 127

00000006 0.00205752 DLSYM: Failed to resolve EVP_sha384: 127

00000007 0.00211535 DLSYM: Failed to resolve EVP_sha512: 127

00000008 1.53627264 SSLVerifyCertAgainstSystemStore: Subject mismatch: localhost.localdomain vs 10.100.0.62

00000009 1.53634858 SSLVerifyCertAgainstSystemStore: The remote host certificate has these problems:

00000010 1.53634858

00000011 1.53634858 * The host name used for the connection does not match the subject name on the host certificate

00000012 1.53634858

00000013 1.53634858 * A certificate in the host's chain is based on an untrusted root.

00000014 1.53641820 SSLVerifyIsEnabled: failed to open the product registry key. Assuming verification is disabled. LastError = 0

00000015 1.53647017 SSLVerifyCertAgainstSystemStore: Certificate verification is disabled, so connection will proceed despite the error

00000016 1.75374973 Succeeded!

00000017 26.77101898 Select failed (retval = -1)

The machine is HP DL585 running ESXi 3.5. update 2.