kevin000
Contributor
Contributor

ESX 3.5, external USB hard drive for disaster recovery - lots of errors

I have a few ESX servers (3.5.0 Update 2 build 110268) that travel around to various sites. I need to put together a disaster recovery toolkit for the field that does not rely on a backup infrastructure being present. I built a custom ESX install CD with our Kickstart file to do an automated install of ESX. I was planning on copying about 200GB of VMs to the local VMFS datastore from an external USB hard drive to complete the field restoration. Unfortunately, when testing this, I've found that the copy from the USB drive to the VMFS dies after a few gigabytes, and the USB drive becomes inaccessible. Here are some error messages I receive:

On the COS terminal:

Message from syslogd@hostname at Wed Aug 27 15:23:55 2008 ...

hostname kernel: scsi: device set offline - not ready or command retry0

On the physical console:

I/O error: dev 08:01, sector 528400

(lots of these, on different sectors)

Lots of similar messages in /var/log/messages, as well, including this:

Aug 27 15:23:55 hostname kernel: scsi: device set offline - not ready o

r command retry failed after bus reset: host 1 channel 0 id 0 lun 0

Aug 27 15:23:55 hostname kernel: SCSI disk error : host 1 channel 0 id

0 lun 0 return code = 70000

The weird thing, though, is that there's nothing wrong with this drive and it works fine on Windows systems. This problem persists through host reboots.

The server in question is a HP Proliant DL360 G5. The USB hard drive is a 250GB Western Digital MyBook Essential Edition. I turned up some anecdotes of poor USB hard drive performance in Linux when searching around, but all of those pertained to 2.6 kernels. I didn't see any suggested fixes that would apply to 2.4, and I have no idea if they'd apply to the ESX service console anyways.

Any ideas? (I'm well aware the USB hard drive could be formatted NTFS and the VMs could be scp'd from a Windows system to work around this problem.) I haven't tried other hard drives yet, but if I can find other makes/models, I will.

Tags (3)
0 Kudos
16 Replies
weinstein5
Immortal
Immortal

it is possible to access a USB drive from ESX and I am surprised you have even gotten as far as you have - I think you will have to set it as a NTFS and do it from a workstation -

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful
0 Kudos
Texiwill
Leadership
Leadership

Hello,

It could be the USB drive as well, the cable, etc. Or could be the driver it is using, I would install a system with CentOS3 U8 and see if it works with that. It may not be anything related to ESX per say but to the USB driver for RHEL3 U8.

However, this should work. But try copying to local disk then to the VMFS? Are you using SCP or CP to do the copy as well? I would suggest SCP.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
kevin000
Contributor
Contributor

Texiwill,

I installed RHEL AS3 U8 x86 on an identically configured HP Proliant DL360 G5 server. I connected the same USB hard drive, with the same USB cable, to this server, and was able to copy all of the contents from the USB drive (with cp; about 185GB) to the local disk (HP Smart Array, RAID 5, 5x 146GB SAS drives, 1 hot spare) in about 2 hours, with no USB-related errors.

When I was doing this operation on the ESX server, I was doing it like so:

host# mount -o ro /dev/sda1 /mnt/usb

host# cp -R /mnt/usb/VMs/* /vmfs/volumes/storage1

The VMFS datastore storage1 was located on the local disk (same HP Smart Array configuration as listed above on the RHEL server). The ESX service console could only copy a few gigs before USB-related errors caused the copy to die.

(How would scp from local storage to local storage help?)

I'd really like to get this working because the more complicated the restore operation, the less likely the people who are doing the work will be able to follow my instructions.

-Kevin

0 Kudos
Texiwill
Leadership
Leadership

Hello,

I installed RHEL AS3 U8 x86 on an identically configured HP Proliant DL360 G5 server. I connected the same USB hard drive, with the same USB cable, to this server, and was able to copy all of the contents from the USB drive (with cp; about 185GB) to the local disk (HP Smart Array, RAID 5, 5x 146GB SAS drives, 1 hot spare) in about 2 hours, with no USB-related errors.

Good test, definitely not the SC GNU/Linux drivers then.

host# mount -o ro /dev/sda1 /mnt/usb

host# cp -R /mnt/usb/VMs/* /vmfs/volumes/storage1

I would always use SCP, if the VMs are thin provisioned then they are sparse files and cp could corrupt the file. scp will prevent that from happening when you copy to the USB and then back from the USB devices. It is just safer.

(How would scp from local storage to local storage help?)

Mainly because you are copying to an ext3 instead of a VMFS filesystem. While all writes to the SC are throttled, writing to a VMFS has its own issues, I.e. it writes 15MB at a time, then gets a new lock, etc. then you wait for the lock to happen then writes more, etc. Writing to an ext3 local disk mimics what your test did.

You should be able to copy to the VMFS from local ext3 with no real issues.

I'd really like to get this working because the more complicated the restore operation, the less likely the people who are doing the work will be able to follow my instructions.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
kevin000
Contributor
Contributor

I would always use SCP, if the VMs are thin provisioned then they are sparse files and cp could corrupt the file. scp will prevent that from happening when you copy to the USB and then back from the USB devices. It is just safer.

Good point. We don't do thin provisioned VMDKs for various reasons (basically the same reasons why they aren't the default behavior), but you're definitely correct about handling them appropriately.

(How would scp from local storage to local storage help?)

Mainly because you are copying to an ext3 instead of a VMFS filesystem. While all writes to the SC are throttled, writing to a VMFS has its own issues, I.e. it writes 15MB at a time, then gets a new lock, etc. then you wait for the lock to happen then writes more, etc. Writing to an ext3 local disk mimics what your test did.

True again. I suppose VMFS-aware tools, like Converter, get the reservation, get all inodes needed for the destination file, drop the reservation, and can then write the large .vmdk file without needing further reservations. cp isn't one of those tools - but also, this is a standalone server, local (non-shared) VMFS, and if we're doing this, it's not running any VMs, and performance isn't an issue.

You should be able to copy to the VMFS from local ext3 with no real issues.

So by local ext3, you specifically mean non-USB local ext3? 200GB is a lot to waste on a ext3 filesystem on internal disk that isn't going to be used for anything other than USB -> local temp ext3 storage -> local VMFS; also, in the future, the next set of VMs may very well exceed 200GB. If we ever get to a point where the set of VMs exceeds 50% of the local internal RAID array, this solution just wouldn't work at all. Perhaps I should try a 30GB or 40GB local ext3 filesystem, and do the USB -> temp -> VMFS operation one VM at a time, but I'm still skeptical that it will help much. I might not have a chance to build a test ESX system with that layout until late next week, but if I can do it tomorrow I'll report back my findings. Thanks for helping!

Kevin

0 Kudos
Rumple
Virtuoso
Virtuoso

Personally if its disaster recovery I would have a laptop setup with FTP server on it and use the free esXpress version to restore your VM's in the event of a disaster.

then you could have them run the scripted install of ESX, run a script to copy the esxpress rpm's to the system, kick off the installation and then initiate a restore based on known ftp parameters....

Bit of work to configure but def better and faster then copying from usb...and I bet less trouble in the end...

0 Kudos
Texiwill
Leadership
Leadership

Hello,

I just ran the same test on my dl380G5 and the USB device fails. I think you should open a SR with your VMware Support Representative. In the meantime using the method Rumple suggested would work as well.

VMFS ESX <-SCP-> Laptop w/Large USB device <-SCP-> ESX VMFS

It will get around the issue you are having.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
Texiwill
Leadership
Leadership

Hello,

Check out for more information.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
kevin000
Contributor
Contributor

Ed,

Thanks for the update. I just got back from my vacation. Poor but reliable performance with USB would be acceptable since we don't plan on doing any backups to or restores from USB on a host in production. Unfortunately, it looks like it just might not work for disaster recovery at all.

Kevin

0 Kudos
Texiwill
Leadership
Leadership

Hello,

Unfortunately it does not but using a laptop w/attached USB would work....


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
gmacarthii
Contributor
Contributor

Hi there... Just recently I tried hooking up to ver 3.5 with a western digital usb 2.0 drive. It doesnt seem to be working properly. Do you have the protocol you followed to get the drive to connect?

As you can see from the dmesg, the drive is actually seen. I may be overlooking something... but cannot map it as a drive, nor see it in the infrastructure client.

My goal was to have the virtuals talk to it as well.

Thanks all

from dmesg...

usb.c: registered new driver usbdevfs

usb.c: registered new driver hub

usb-ohci.c: USB OHCI at membase 0xd20c6000, IRQ 16

usb-ohci.c: usb-00:0f.2, ServerWorks OSB4/CSB5 OHCI USB Controller

usb.c: new USB bus registered, assigned bus number 1

hub.c: USB hub found

hub.c: 4 ports detected

usb.c: registered new driver hiddev

usb.c: registered new driver hid

hid-core.c: v1.8.1 Andreas Gal, Vojtech Pavlik &lt;vojtech@suse.cz&gt;

hid-core.c: USB HID support drivers

mice: PS/2 mouse device common for all mice

hub.c: new USB device 00:0f.2-3, assigned address 2

input0: USB HID v1.11 Device on usb1:2.1

hub.c: new USB device 00:0f.2-4, assigned address 3

input1: USB HID v1.00 Keyboard http://P.I. Engineering PC Keyboard/Mouse to USB Adapter on usb1:3.0

input2: USB HID v1.00 Mouse http://P.I. Engineering PC Keyboard/Mouse to USB Adapter on usb1:3.1

Initializing USB Mass Storage driver...

usb.c: registered new driver usb-storage

scsi1 : SCSI emulation for USB Mass Storage devices

Vendor: WD Model: My Book Rev: 1010

Type: Direct-Access ANSI SCSI revision: 02

Vendor: WD Model: My Book Device Rev: 1010

Type: Enclosure ANSI SCSI revision: 02

VMWARE SCSI Id: Supported VPD pages for sda : 0x0 0x1 0x2 0x80 0x83 0xc1

VMWARE SCSI Id: Device id info for sda: 0x2 0x1 0x57 0x44 0x20 0x20 0x20 0x20 0x20 0x20 0x4d 0x79 0x20 0x42 0x6f 0x6f 0x6b 0x20 0x20 0x20 0x20 0x20 0x20 0x20 0x20 0x20 0x57 0x55 0x32 0x51 0x31 0x30 0x30 0x31 0x39 0x35 0x31 0x37 0x20 0x20

VMWARE SCSI Id: Id for sda 0x57 0x55 0x32 0x51 0x31 0x30 0x30 0x31 0x39 0x35 0x31 0x37 0x20 0x20 0x20 0x20 0x4d 0x79 0x20 0x42 0x6f 0x6f

VMWARE: Unique Device attached as scsi disk sda at scsi1, channel 0, id 0, lun 0

Attached scsi disk sda at scsi1, channel 0, id 0, lun 0

SCSI device sda: 3907017568 512-byte hdwr sectors (2001367 MB)

sda: sda1

WARNING: USB Mass Storage data integrity not assured

USB Mass Storage device found at 2

USB Mass Storage support registered.

EXT3 FS 2.4-0.9.19, 19 August 2002 on cciss(104,2), internal journal

Adding Swap: 554200k swap-space (priority -1)

kjournald starting. Commit interval 5 seconds

EXT3 FS 2.4-0.9.19, 19 August 2002 on cciss(104,1), internal journal

EXT3-fs: mounted filesystem with ordered data mode.

kjournald starting. Commit interval 5 seconds

EXT3 FS 2.4-0.9.19, 19 August 2002 on cciss(104,6), internal journal

EXT3-fs: mounted filesystem with ordered data mode.

microcode: CPU0 already at revision 0x38 (current=0x38)

microcode: No suitable data for cpu 0

ip_tables: (C) 2000-2002 Netfilter core team

ip_conntrack version 2.1 (2176 buckets, 17408 max) - 308 bytes per conntrack

Intel(R) PRO/1000 Network Driver - version 7.3.15

Copyright (c) 1999-2006 Intel Corporation.

Intel(R) PRO/1000 Network Driver - version 7.3.15

Copyright (c) 1999-2006 Intel Corporation.

0 Kudos
Texiwill
Leadership
Leadership

Hello,

My goal was to have the virtuals talk to it as well.

This is not possible within VMware ESX or VMware ESXi. You can not present a USB device to a VM. You can however use a USB over IP device.

The problem this thread is addressing is the use of a USB drive within the Service Console fails with SC kernel errors to which there is no solution yet.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
gmacarthii
Contributor
Contributor

thanks...

It's interesting that you can boot a usb device... a la cd, and present it

via the console. but not a quick storage solution.

0 Kudos
Texiwill
Leadership
Leadership

Hello,

Most CDROM devices are PATA devices and SOme SATA devices. These are presentable. However no USB CDROm is presentable unless the hardware is first translating it to some other hardware form. Installation is not the same as running with VMs.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
kevin000
Contributor
Contributor

Hello,

I just ran the same test on my dl380G5 and the USB device fails. I think you should open a SR with your VMware Support Representative.

Now this is interesting. I repeated the test on the same server, with the same hard drive, on ESX 3.0.2 (52542), and it worked fine. I was able to copy 189GB from USB to my VMFS datastore in 139min - not fast, but absolutely acceptable for our needs. I need to do what you've suggested and open a case with support.

kevin

0 Kudos
Toronto_VMUG
Contributor
Contributor

Same error booting from USB DVD Drive.

Always the CD rom would load (ESX Server 3.5 Update 5) Press Enter for Graphical...

Starts running the installer then hangs on usb.c new USB bus registered, assigned bus number 1

HP Proliant DL165 G6 Single 6 way AMD Opteron, 8 GB Ram, 4x 146 GB SAS drives in an array.

Tried bootable USB key, would not detect/boot.

Tried to force USB device to be CDROM (instead of AUTO)

Finally. In the Bios I selected the following:

Advanced

USB Configuration

USB 2.0 Controller Mode

Changed the speed setting from "High Speed" (Default) to "Full Speed"

Saved settings.

The installation went through completely.

Remove CD, restarts every time.

0 Kudos