ERROR CLDELSN3, details: [lon-d-squid] error: cannot delete snapshot

[robertkwild]

hi all,

got an error on one of my vms when trying to back it up using a backup solution called XSiBackup, the thing is it backs up all my other running vms, just this one all the time generates an error

the error is

ERROR CLDELSN3, details: [lon-d-squid] error: cannot delete snapshot, details: /vmfs/volumes/datastore11/XSIBackup-Free/xsibackup: eval: line 1: disk0: not found /vmfs/volumes/datastore11/XSIBackup-Free/xsibackup: eval: line 1: disk1: not found

its saying it cant delete a snapshot but i see no snapshots (i attach pics)

https://i.postimg.cc/fLgs5C1J/datastore.png

https://i.postimg.cc/3N4TqpH9/settings.png

https://i.postimg.cc/906hHMm2/snap.png

someone mentioned to me, make sure the vmx file is correct and ie the disks are present, i have checked it and all looks good, heres a brief of my vmx with disks

scsi0:0.deviceType = "scsi-hardDisk"

scsi0:0.fileName = "lon-d-squid.vmdk"

scsi0:0.present = "TRUE"

scsi0:1.deviceType = "scsi-hardDisk"

scsi0:1.fileName = "lon-d-squid_1.vmdk"

scsi0:1.present = "TRUE"

can anyone shed some light on this as it would be very much appreciated

many thanks,

rob

[nachogonzalez]

Hey, hope you are doing fine:

let me ask you a few questions:
- Are you using an RDM?
- Have you checked if CD/DVD drive has any iso mounted?
- Do you have any passtrough devices?
- If you try to take a snapshot from vSphere what happens?
- Is there any consolidation pending?
- Have you tried doing a storage vMotion of the VM?
- Have you restarted management agents of the ESXi hosts?

Let me know

[robertkwild]

Hi nachogonzalez, im doing fine thanks, hope your doing well

this VM is not using an RDM

the VM is using "client cd/dvd drive" but its not connected (like all my other vms)

no passthrough devices, like usb, gfx, sound etc

i made a snapshot using vcenter and it made it successfully, i then deleted it

theres no snapshots to consolidate and the consolidation is greyed out so i cant click it (in vcenter)

so im migrating just the computer resource only NOT the storage, vmotioned to the other host and the backup still fails

can i restart managment agents on the hosts using vcenter?

this is the error which i really dont get as its complaining about the disks but it picks them up?!?!?

/vmfs/volumes/datastore1/XSIBackup-Free/xsibackup: eval: line 1: disk0: not found

/vmfs/volumes/datastore1/XSIBackup-Free/xsibackup: eval: line 1: disk1: not found

Backing up virtual disks...

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

scsi0:0.fileName = "lon-d-squid.vmdk"

scsi0:1.fileName = "lon-d-squid_1.vmdk"

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Destination disk format: VMFS thin-provisioned

Cloning disk '/vmfs/volumes/datastore2/lon-d-squid/lon-d-squid.vmdk'...

Clone: 100% done.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[lon-d-squid] Disk copied to /vmfs/volumes/datastore2/backup/20200907180651/lon-d-squid/lon-d-squid.vmdk

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Destination disk format: VMFS thin-provisioned

Cloning disk '/vmfs/volumes/datastore2/lon-d-squid/lon-d-squid_1.vmdk'...

Clone: 100% done.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[lon-d-squid] Disk copied to /vmfs/volumes/datastore2/backup/20200907180651/lon-d-squid/lon-d-squid_1.vmdk

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------------

|              Backup of ESXi configuration is not available in XSIBACKUP-FREE             |

|                          Get XSIBACKUP-PRO at http://33hops.com                          |

--------------------------------------------------------------------------------------------

Errors detected in backup, check logs

so its complaining about the disks but as you can clearly see it knows about both vmdk files

[nachogonzalez]

Okay, let's stripe this down:

If you can take snapshots from vSphere there is not an issue on the vSphere side and what we need to tackle is the Backup Server and why is it failing.

QQ: has this VM been restored or any change on the VM (de registered and registered again)?

I'm not really familiar with this backup server, but with  veeam backup and replication i've faced  an issue in which it's backup proxies lock VM VMDK files so you can't take VM snapshots.
Does this backup server use proxies? have you checked it's virtual hardware?

[robertkwild]

this VM did come from a Xen server like all my other linux VMs i have on it (i used vcenter converter standalone)

i havnt re registered it, do you think i should do that?

no the backup vm doesnt use proxies at all, you install it on the hosts, so no vm just installed on the hosts and tbh its really good, if your interested heres the link

https://33hops.com/xsibackup-tutorial-1.html

i have installed the latest version of xsibackup on both servers as i thought it was that tbh, but that wouldnt explain why its only failing on this VM

[nachogonzalez]

I don't think you should re register it, since a change in VM-ID can alter backup chains.

[nachogonzalez]

Have you checked this: https://33hops.com/xsibackup-tutorial-1.html ?

[robertkwild]

i just tried to coonvert the server again (using vmware vcenter converter standalone)

it converted the vm to a new vm fine and i could spin up

but when i tried to do the backup of the new VM, got that error again

yes, thats what im doing (the link you sent me)

i putty into a host and run the following

/vmfs/volumes/datastore11/XSIBackup-Free/xsibackup --backup-point=/vmfs/volumes/datastore99/backup/esx02 --backup-type=running --date-dir=yes --backup-room=6000 --snapshot=doquiesce --mail-from=email@email.com --mail-to=email@email.com --subject=esx02 --smtp-srv=10.110.130.68 --smtp-port=587 --smtp-auth=none --smtp-usr=email@email.com --smtp-pwd=any

i have this running on a cron and it works really well but the only vm it errors out on is the server squid, it sends a report to my email

[nachogonzalez]

does the VM has VMware tools installed?

[robertkwild]

yes it has open-vm-tools

or is it best to install the one that vcenter provides?

EDIT - ok did a yum -y remove open-vm-tools, rebooted the vm and then installed the one vcenter provided

will let you know

thanks alot for your help so far, its much appreciated!!!

[robertkwild]

no...what a suprise, it didnt work

when i converted all my linux vms over from xen to esxi it has made them all "guest os - centos 4/5 or later (64bit)"

saying that my other linux vms back up

[scott28tt]

What did the XSIBackup team suggest when you asked them for help?

[robertkwild]

they said this scott,

There's some inconsistency in the VM layout. Look for the 'eval' keyword in the source code and debug or check your .vmx file to find the cause. Looks like some missing disk is in the .vmx file.

and when they say look at the source code they mean xsibackup file, that kicks off the backup script ie

/vmfs/volumes/datastore11/XSIBackup-Free/"xsibackup"

[scott28tt]

Have you compared the content of a working VM VMX file against the VMX file of this problem VM?

Have you tried creating a new VM and attaching the existing disks of this VM?

[robertkwild]

i have compared a good vmx to this one and tbh i see no difference on the disk section

scsi0.virtualDev = "lsilogic"

scsi0.present = "TRUE"

scsi0:0.deviceType = "scsi-hardDisk"

scsi0:0.fileName = "lon-d-squid.vmdk"

scsi0:0.present = "TRUE"

scsi0:1.deviceType = "scsi-hardDisk"

scsi0:1.fileName = "lon-d-squid_1.vmdk"

scsi0:1.present = "TRUE"

scsi0.virtualDev = "lsilogic"

scsi0.present = "TRUE"

scsi0:0.deviceType = "scsi-hardDisk"

scsi0:0.fileName = "lon-c-smtp10.vmdk"

scsi0:0.present = "TRUE"

scsi0:1.deviceType = "scsi-hardDisk"

scsi0:1.fileName = "lon-c-smtp10_1.vmdk"

scsi0:1.present = "TRUE"

and screenshot of the datastores for both are below

https://i.postimg.cc/HnDNxqZW/smtp10.png

https://i.postimg.cc/5yBRFsTB/squid.png

as you can see both vmdk files are there for both vms

sorry i forgot to mention, i created a new vm with no disks, copied my squid disks to the new vm, it booted up fine no problem but when i tried to do a backup, i got the same error

thanks,

rob

[scott28tt]

Have you looked at the content of the VMDK descriptor files between a working VM and the non-working VM?

Have you tried creating a new virtual disk and transferring the data from old to new via the guest OS?

[robertkwild]

sorry scott, when you say look at the vmdk descriptor files, do you mean just downloading the vmdk files and just editing them?

thanks,

rob

[scott28tt]

Literally the lon-d-squid.vmdk descriptor file rather than the accompanying lon-d-squid-flat.vmdk raw data file that is usually present in the same folder.

[robertkwild]

do you mean this

# Disk DescriptorFile

version=1

encoding="UTF-8"

CID=d70dd275

parentCID=ffffffff

createType="vmfs"

# Extent description

RW 2101248 VMFS "lon-d-squid-flat.vmdk"

# The Disk Data Base

#DDB

ddb.adapterType = "lsilogic"

ddb.deletable = "true"

ddb.geometry.biosCylinders = "513"

ddb.geometry.biosHeads = "128"

ddb.geometry.biosSectors = "32"

ddb.geometry.cylinders = "513"

ddb.geometry.heads = "128"

ddb.geometry.sectors = "32"

ddb.longContentID = "7e311909dfddbc02d1582ddad70dd275"

ddb.toolsInstallType = "2"

ddb.toolsVersion = "10357"

ddb.uuid = "60 00 C2 96 79 ac ec e3-1c 19 f3 f3 3b e4 1a 1d"

ddb.virtualHWVersion = "14"

and other vmdk

# Disk DescriptorFile

version=1

encoding="UTF-8"

CID=eb5be4d2

parentCID=ffffffff

createType="vmfs"

# Extent description

RW 2101248 VMFS "lon-c-smtp10-flat.vmdk"

# The Disk Data Base

#DDB

ddb.adapterType = "lsilogic"

ddb.geometry.biosCylinders = "513"

ddb.geometry.biosHeads = "128"

ddb.geometry.biosSectors = "32"

ddb.geometry.cylinders = "513"

ddb.geometry.heads = "128"

ddb.geometry.sectors = "32"

ddb.longContentID = "ff83afe4c0bc5fcf8f5c2e10eb5be4d2"

ddb.toolsInstallType = "2"

ddb.toolsVersion = "9541"

ddb.uuid = "60 00 C2 9a 81 d1 d9 81-8e 6b 34 9e f9 64 e6 25"

ddb.virtualHWVersion = "14"

[robertkwild]

i dont see the flat vmdk file, i only see the vmdk file

ok this is interesting... the only difference, i see between the two files are

ddb.deletable = "true"

is this the reason?