There is no more space for the redo log

[letoatrads]

I'm getting the follow error when making some changes in a VM.

"There is no more space for the redo log of XXXXX.vmdk You may be able to continue this session by freeing disk space on the relevant partition, and clicking Retry, Otherwise, click abort to terminate"

Here is the kicker, this VM does not HAVE any snapshot's ( at least none shown in Snapshot manager) and there are no REDO log's in this VM's directory. There are some XXXX-delta.vmdk files which I suspect have something to do with this mess.

There was a snapshot created of this VM, but then deleted, and now with no snapshots I keep running into this error when any change is made to this VM.

Any thoughts? I'de vmware-cmd or vmfsktools to commit the redo logs if I saw any....

[hicksj]

Since you mentioned "Snapshot Manager" I'll assume you have an ESX3 host.

There are no more "REDO" logs. What you'll find is exactly what you have... XXXX-delta.vmdk files. (I think that error message should be updated)

Now, why they're not showing up in Snapshot Manager, I don't know. So you'll have to work with them at the console. To commit these files, you can use vmware-cmd

See:

vmware-cmd revertsnapshot

or

vmware-cmd removesnapshots

revert will go back to previous snapshots, remove will commit the changes.

Regards,

J

[letoatrads]

Yes, correct an ESX3 host.

I think you've got me on the right track, but I'm still not quite there. The command

vmmware-cmd getconfigfile

Returns

getconfigfile() = Sybase.vmx

So I think it see's the valid config file, and knows this should be a VM....any more ideas?

[Michelle_Laveri]

are you using vmware-cmd with the long unique names, rather the vmfs volume name...

vmware-cmd -l

should show you this...

Regards

Mike

[letoatrads]

Nope, I had been using the 'friendly name' thanks for that tip. NOW I'm getting the error message

VMControl error -3: Invalid arguments: Virtual machine has no snapshots

I'de love to believe that....however, with the delta.vmdk files sitting there, and it error out saying there isn't any space for a redo log....well pphhbbttt.

I've also tried shutting down the VM and making all disks Independent....nope, still give the space error.

[hicksj]

Hmmm...

What does your vmx define as the disks in use? Does it point at the -00000#.vmdk (which then points to the #-delta.vmdk) or just the original vmdk's?

If just the original, maybe something got messed up with a past snapshot attempt and they can just be manually deleted. However, I'd make darn sure that's the case - for obvious reasons

Regards,

J

[hicksj]

And what result is returned with the following:

vmware-cmd /vmfs/volumes/xyz/vm/vm.vmx hassnapshot

[letoatrads]

And what result is returned with the following:

vmware-cmd /vmfs/volumes/xyz/vm/vm.vmx hassnapshot

hassnapshot() =

Now ever more interesting/annoying. I have tried renaming the delta files to see what happens, the VM won't boot with a "file missing error".

Now the vmx file looks like it is pointing to the delta files, -00000#.vmdk, so I tried editting that to point to the correct VMDK. Saved the VMX, went back in to verify the change took, then booted the VM. Low and behold the vmx file has reverted to pointing to the delta file.

[Chris_S_UK]

Are the delta files' time/date stamp actually changing when the VM is running?

I suggest you post the contents of the vmx file together with an 'll' command of the VM's directory.

A "vdf -h" might also be useful.

Chris

[letoatrads]

Are the delta files' time/date stamp actually

changing when the VM is running?

Yes, it looks like the delta files ( at least some ) are changing.

I suggest you post the contents of the vmx file

#!/usr/bin/vmware

config.version = "8"

virtualHW.version = "4"

floppy0.present = "true"

nvram = "Sybase.nvram"

powerType.powerOff = "default"

powerType.powerOn = "default"

powerType.suspend = "default"

powerType.reset = "default"

displayName = "Sybase"

extendedConfigFile = "Sybase.vmxf"

numvcpus = "1"

scsi0.present = "true"

scsi0.sharedBus = "none"

scsi0.virtualDev = "lsilogic"

memsize = "4096"

scsi0:0.present = "true"

scsi0:0.fileName = "Sybase_1-000001.vmdk"

scsi0:0.deviceType = "scsi-hardDisk"

ide0:0.present = "true"

ide0:0.clientDevice = "FALSE"

ide0:0.deviceType = "cdrom-image"

ide0:0.startConnected = "false"

floppy0.startConnected = "false"

floppy0.clientDevice = "true"

ethernet0.present = "true"

ethernet0.allowGuestConnectionControl = "false"

ethernet0.networkName = "VM Network"

ethernet0.addressType = "vpx"

guestOS = "rhel4"

floppy0.fileName = "/dev/fd0"

ethernet0.generatedAddress = "00:50:56:9e:5a:42"

uuid.bios = "50 1e 11 4d 82 61 78 dd-b2 54 c9 5e 2c 93 c9 96"

ide0:0.fileName = "/vmfs/volumes/44b98510-33e64fee-c261-00144f285bf4/ISO/sybase1

2_5_2.iso"

scsi0:0.redo = ""

uuid.location = "56 4d c6 d4 1a 12 79 d2-28 d6 34 99 fa 22 b7 2d"

sched.cpu.min = "0"

sched.cpu.units = "mhz"

sched.cpu.shares = "normal"

sched.mem.minsize = "0"

sched.mem.max = "unlimited"

sched.mem.shares = "normal"

sched.swap.derivedName = "/vmfs/volumes/44a08f32-ae442055-8a1b-00144f29aca4/Syba

se/Sybase-846a2b3b.vswp"

tools.syncTime = "TRUE"

migrate.hostlog = "./Sybase-846a2b3b.hlog"

scsi0:1.present = "true"

scsi0:1.fileName = "Sybase-000001.vmdk"

scsi0:1.deviceType = "scsi-hardDisk"

scsi0:2.present = "true"

scsi0:2.fileName = "Sybase-000002.vmdk"

scsi0:2.deviceType = "scsi-hardDisk"

scsi0:1.redo = ""

scsi0:2.redo = ""

log.fileName = "vmware.log"

workingDir = "."

scsi0:0.mode = "independent-persistent"

sched.scsi0:0.shares = "normal"

scsi0:1.mode = "independent-persistent"

sched.scsi0:1.shares = "normal"

scsi0:2.mode = "independent-persistent"

sched.scsi0:2.shares = "normal"

together with an 'll' command of the VM's directory.

total 33624448

drwxr-xr-x 1 root root 4060 Aug 30 12:02 ./

drwxrwxrwt 1 root root 4480 Aug 24 11:38 ../

-rw------- 1 root root 1828716544 Aug 30 11:35 Sybase-000001-delta.vmdk

-rw------- 1 root root 306 Aug 30 11:34 Sybase-000001.vmdk

-rw------- 1 root root 16777216 Aug 28 15:44 Sybase-000002-delta.vmdk

-rw------- 1 root root 305 Aug 30 11:31 Sybase-000002.vmdk

-rw------- 1 root root 369098752 Aug 30 11:35 Sybase_1-000001-delta.vmdk

-rw------- 1 root root 251 Aug 30 11:31 Sybase_1-000001.vmdk

-rw------- 1 root root 32212254720 Aug 28 15:40 Sybase_1-flat.vmdk

-rw------- 1 root root 340 Aug 28 15:29 Sybase_1.vmdk

-rw-rr 1 root root 37 Aug 28 17:28 Sybase-846a2b3b.hlog

-rw------- 1 root root 8664 Aug 30 11:30 Sybase.nvram

-rw------- 1 root root 668 Aug 30 11:31 Sybase.vmsd

-rwx------ 1 root root 1996 Aug 30 11:42 Sybase.vmx

-rw------- 1 root root 250 Aug 30 11:42 Sybase.vmxf

-rwxr-xr-x 1 root root 1791 Aug 30 10:06 Sybase.vmxMike

-rwx------ 1 root root 1996 Aug 30 11:38 Sybase.vmxMikeNew

-rw-rr 1 root root 15337 Aug 30 10:13 vmware-30.log

-rw-rr 1 root root 28908 Aug 30 10:57 vmware-31.log

-rw-rr 1 root root 15793 Aug 30 11:07 vmware-32.log

-rw-rr 1 root root 29277 Aug 30 11:35 vmware-33.log

-rw-rr 1 root root 18128 Aug 30 11:37 vmware-34.log

-rw-rr 1 root root 17977 Aug 30 11:41 vmware-35.log

-rw-rr 1 root root 17977 Aug 30 11:42 vmware.log

-r----

---

1 root root 61440 Aug 29 17:38 vmware-vmx-zdump.0

-r----

---

1 root root 1052672 Aug 30 09:40 vmware-vmx-zdump.1

-r----

---

1 root root 1052672 Aug 30 10:11 vmware-vmx-zdump.2

-r----

---

1 root root 61440 Aug 30 10:57 vmware-vmx-zdump.3

-r----

---

1 root root 61440 Aug 30 11:35 vmware-vmx-zdump.4

A "vdf -h" might also be useful.

Filesystem Size Used Avail Use% Mounted on

/dev/sda2 4.9G 1.4G 3.2G 31% /

/dev/sda1 99M 30M 65M 32% /boot

none 132M 0 132M 0% /dev/shm

/dev/sda6 2.0G 40M 1.8G 3% /var/log

/vmfs/devices 3.8T 0 3.8T 0% /vmfs/devices

/vmfs/volumes/44a08f32-ae442055-8a1b-00144f29aca4

199G 195G 4.0G 97% /vmfs/volumes/VM_OS

/vmfs/volumes/44a350c2-7c6e2eb9-ff0a-00144f285bf4

99G 33G 66G 33% /vmfs/volumes/VM_Templates

/vmfs/volumes/44ac83f7-a45ee13c-6591-00144f285bf4

999G 990G 8.9G 99% /vmfs/volumes/SybaseMediaHome

/vmfs/volumes/44ac8425-8264678a-2a25-00144f285bf4

279G 275G 4.1G 98% /vmfs/volumes/VNIHome

/vmfs/volumes/44b98510-33e64fee-c261-00144f285bf4

99G 16G 83G 16% /vmfs/volumes/ISO

/vmfs/volumes/44ea13c5-1feab50a-c3ed-00144f285bf4

60G 627M 59G 1% /vmfs/volumes/LocalSUNESX1

/vmfs/volumes/44f45a90-130462cb-e4d0-00144f29aca4

299G 55G 244G 18% /vmfs/volumes/VM_OS2

Chris

All as requested. Thanks!

[hicksj · Accepted solution]

Ok weird. Was this a clone of another system?

What is odd to me is that you have scsi0:1 and scsi0:2 as Sybase-000001.vmdk and Sybase-000002.vmdk, which are your pointers to the delta files. BUT There is no corresponding "Sybase.vmdk" disk file (your scsi0:0, Sybase_1 setup and files look right)

You should normally have disk.vmdk, then after subsequent snapshots, disk-000001.vmdk, disk-000002.vmdk, and so on.

Subsequent disks should be disk_1, disk_2... not disk-000001, disk-000002.

Are these disk mounts available and operational within your VM? I'd be very surprised if they were. Otherwise I'm missing something here.

Does this make sense? If so, and you're not actually mounting scsi0:1 and scsi0:2 in your guest OS, then here's what you do: Manually remove all the entries for scsi0:1 and scsi0:2. Then you should be all set.

If they are working, somehow (?), then I have a followup idea for you.

Message was edited by:

hicksj (further clarification)

[letoatrads]

Well, the kicker is they WERE working...and visible to the OS, just couldn't make changes....since it said out of space.

I did figure out it wasn't the OS partition that was causing the problem ( renamed it's delta file and it came up fine ). So in the end, I just blew away the 2 data partitions, reformatted and re-added. Everything seems to be ok now.

Odd issue, I'm going to be very careful in the future with using snapshots.

Thanks everyone for their help!

[dzo]

Hey guys,

having the same odd problem.

vmware-cmd says there are no snapshots but the filesystem is full of delta-files which are actually used (verified that via timestamp)

I hope they are going to fix this \*very* soon.

PS: Found a nice workaround:

\- Shut down the VM

\- Mark the not affected disks as "independent"

\- Create a new snapshot with vmware-cmd

\- Issue the command: vmware-cmd removesnapshots

\- Grep a coffee

\[...]

\- Go to lunch

\- All Snapshots should be deleted and your VM is "clean" again.

found workaround

Message was edited by:

dzo

[AlexPT]

Hey folks. I have this same issue and would like more info / fix.

Is there not a way of committing the delta files hot?

[AlexPT]

Just had the same issue.

Run a manual vcbMounter for your guest.

vcbMounter -h 10.10.x.1x -a ipaddr:10.10.x.x -r /titan/vm_backups/PROD_-_Ginger -t fullvm -L 6

Worked a treat.

Background: My automated hot backup script snapped all but this one problem VM. The VM was in an unusable state when I came in and I noticed the redo / delta files. I had to kill the vm's process and run the above. This was done HOT, i.e the guest was up.

[skoban]

I also have the same Problem on ESX 3.

none of the suggestions did work for me.

When I tried vcbMounter i did get an Error message while processing the xxxxxx-flat.vmdk. " ... Not enough space left on device ... "

For now i did a workaround using symlinks. I.e. I moved some of the delta files to another local storage (from /vmfs/volumes/storage2 to /vmfs/volumes/storage1) and created symlinks (ln -s ) in the source storage. Now the guest starts and runs ...

But well.. that is not too satisfying, as the deltas still grow.

There should be a way too get rid of the deltas without destroying the guest!

[skoban]

Szenario:

A VM with a Windows Server 2K3 SP1, where your delta files (xxxxxx-delta.vmdk) grow and grow ... Somehow I was not able to get rid of those delta files so I did the following to return to a single vmdk without any delta files:

0. Write down every config detail of your VM

1. Shut Down the VM

2. Boot with TrueImage (V 8 SRV)

3. Write an image to a cifs volume

4. Delete the VM

5. Create a new (empty) VM equal to the old one (see 0.)

6. Write back the created image

7. Boot the VM

8. Be happy

I had a 32 GB disk, with about 10 GB used. The image I created was about 6 GB. The hole thing took about 3.5 hours (GBit Network).

Make sure the cifs volume to store the image is ntfs or if you are using samba it should be ext3 or xfs, so you can store files >> 4 GB!

At least this did work for me but I don't know whether this is useful for someone else.

[azn2kew]

I'm having the same issue with my VM on ESX3 host? I can see all VM vmdk files using Veeam FastSCP but how do I make my VM to restart again. Error: ""There is no more space for the redo log of XXXXX.vmdk You may be able to continue this session by freeing disk space"

Should I delete xxdelta.vmdk file on the VM will that free up disk space? I shows I have 2 snapshots*.vmsn file about 2GB total? Can it be save to delete this snapshot files?

Can you give me complete command to revert my snapshot.

vmware-cmd in this argument mean? Where do i specify the snapshot file or vmdk file?

Please give me exact command structure that would be great since i'm newbie.

Thanks!

[azn2kew]

Both my VMs are up and running now because I've removed 1 test machine so it adds more space. I'm still thinking if I have 2 version of .vmsn file can i delete one of them?

[multik]

Just ran into same issue. VMFS volume contained .delta file that was about 135MB. I was able to shut down VM, though with error complaining about not enough space for redo. But I wasn't so lucky when powering up VM. I had to delete snapshot.

Is there something going on with snapshots or am I doing something wrong? All I want to do is a snapshot of VM before patching Windows. Any help is greatly appreciated.

Cheers,

Martin