davev
Contributor
Contributor

vcbMounter Fails with "No space left on device"

Jump to solution

I am trying to backup our virtual machine to a remote server. Here's the command I'm using:

#vcbMounter -a ipaddr:<ip> -r scp://<user>@<server>:/backups/VMs

The process starts but then errors out:

vcbMounter 3076460672 error Error: Failed to export the disk: No space left on device

I haven't been able to find anything on this error. The remote server I'm backing up to has plenty of space. The Virtual Machines I'm trying to backup should have enough, unless it requires a very large temp space during the backup. Any ideas?

Thank you.

0 Kudos
1 Solution

Accepted Solutions
BryanMcC
Expert
Expert

You are right.. Doesnt look bad.. But your above command to place the backup in /home/VMs will not work as you have less than two GB on / which is where /home lives.. Try -r /storage1/VMBackup and see what the outcome is. You have plenty of space on local VMFS3.

The error is a typical linux error you receive when you are running out of disk space in a partition.. All signs point to this so I am just trying to rule out where you are placing the backup.. Hope you understand.

Help me help you by scoring points... Smiley Happy

Help me help you by scoring points.

View solution in original post

0 Kudos
11 Replies
BryanMcC
Expert
Expert

The vcbMounter will create a snapshot of the VM prior to backing it up so it can have disk control and allow I/O to write to delta files. Are you sure there is enough space on the VMs LUN/storage to create snapshots? Also is there enough space on the destination share that you are scp'ing to?

Maybe you could try this to a LUN or even local storage that the ESX server can see natively that you are sure has available space.

Help me help you by scoring points... Smiley Happy

Help me help you by scoring points.
davev
Contributor
Contributor

I checked the storage space on one of the VM's I'm trying to backup. This one in particular has 15.1 GB of free space, with 4.8 of used space (on C:). It would seem 15.1 GB of free space would be plenty to store a snapshot right? The remote server has plenty of space.

I just tried storing this locally using the following command and it gave me the same error:

  1. vcbMounter -a ipaddr:<ip> -r /home/VMs

0 Kudos
BryanMcC
Expert
Expert

It would be the utilization of the vmfs volumes you need to look at... Try going to the Service Console and typing the following...

vdf -h

That should give you a disk utilization for your vmfs volumes.

Help me help you by scoring points.
davev
Contributor
Contributor

Ahh, ok. So here's what I get:

  1. vdf -h

Filesystem Size Used Avail Use% Mounted on

/dev/cciss/c0d0p2 4.9G 1.4G 3.3G 29% /

/dev/cciss/c0d0p1 97M 29M 63M 32% /boot

none 131M 0 131M 0% /dev/shm

/dev/cciss/c0d0p6 2.0G 60M 1.8G 4% /var/log

/vmfs/devices 1.4T 0 1.4T 0% /vmfs/devices

/vmfs/volumes/464dd34c-5b785a0c-db80-0014c2c1c80b

551G 362G 188G 65% /vmfs/volumes/storage1

So again it appears I should be ok on disk space.

0 Kudos
BryanMcC
Expert
Expert

You are right.. Doesnt look bad.. But your above command to place the backup in /home/VMs will not work as you have less than two GB on / which is where /home lives.. Try -r /storage1/VMBackup and see what the outcome is. You have plenty of space on local VMFS3.

The error is a typical linux error you receive when you are running out of disk space in a partition.. All signs point to this so I am just trying to rule out where you are placing the backup.. Hope you understand.

Help me help you by scoring points... Smiley Happy

Help me help you by scoring points.

View solution in original post

0 Kudos
davev
Contributor
Contributor

Ok, so that worked. Thanks! However can you answer the question as to why this command does not work:

  1. vcbMounter -a ipaddr:<ip> -r scp://administrator@<server>:/backups/VMs

but his one does:

  1. vcbMounter -a ipaddr:<ip> -r /vmfs/volumes/storage1/VMs

I guess I don't understand why I'm unable to store this remotely. How did changing it to a localized backup make it work? It's almost like the 1st command is trying to store the temp snapshot on the "/" directory on the linux box before it copies it over to the remote server. And since "/" does not have enough space it errors out. Any ideas? Is there a line in a config file that I can change the default temp location to "/storage1" so there's enough space (assuming that's the problem).

Thanks again for all your help.

0 Kudos
BryanMcC
Expert
Expert

I bet the temporary file is trying to store in temp.. Unfortuantley I have nothing on how to change the storage location.. What type of file system is this you are using on the remote server?

Help me help you by scoring points... Smiley Happy

Help me help you by scoring points.
0 Kudos
BryanMcC
Expert
Expert

I'd be curious to see what the /var/log/messages file was gettig from this. You could open putty and run tail -f /var/log/messages and then open another putty session and try the vcbMounter... Any logged messages should be displayed on the other console in real time.

Help me help you by scoring points... Smiley Happy

Help me help you by scoring points.
0 Kudos
davev
Contributor
Contributor

I'll see what I can uncover in the logs. I did find this line in the /etc/vmware/backuptools.conf:

  1. Temporary directory (required for legacy VM restoration and for

  2. restoring using scp.

TEMPDIR=/tmp

It only mentions "restoration" and not backups but I wonder if they both use the same. I might change this to "TEMPDIR=/storage1/VMs" just to see what happens.

0 Kudos
BryanMcC
Expert
Expert

Sounds like a lead to me.. You may need to restart some service to have the file re-read.

Help me help you by scoring points.
0 Kudos
dcahill
Contributor
Contributor

I'm having the same problem on my system as the original poster.

This is my current space:

Filesystem Size Used Avail Use% Mounted on

/dev/sda2 4.9G 1.4G 3.2G 31% /

/dev/sda1 99M 32M 63M 34% /boot

none 131M 0 131M 0% /dev/shm

/dev/sda6 2.0G 130M 1.7G 7% /var/log

/vmfs/devices 2.0T 0 2.0T 0% /vmfs/devices

/vmfs/volumes/452f496a-1b5dd670-1fd6-00188b35a813

128G 628M 127G 0% /vmfs/volumes/internal-losal139

/vmfs/volumes/45471ce6-8c1c6357-760f-00188b36f7c7

599G 358G 240G 59% /vmfs/volumes/disk3

/vmfs/volumes/45471d01-19c54d43-024f-00188b36f7c7

599G 444G 155G 74% /vmfs/volumes/disk2

/vmfs/volumes/45471d12-5344317e-1b71-00188b36f7c7

599G 419G 179G 70% /vmfs/volumes/disk1

The VM in that fails the backup is configured with a drive on disk2 from the list above. The size configured for the VM is 100GB with only 60GB actually used. Disk2 has 155GB free...so is it using a temp location other than the disk that the VM is configured on?

0 Kudos