removing snapshot failed

[marcus_31]

In order to migrate one of my vhost to another clusterfarm i have to merge/delete all snapshots beforehand. I' am aware that VC has an hardcoded 15min buildin timeout and snapshot process will still working in the backgroud even if a timeout/error in VC occurs. SO today i tried to remove this 30gb snapshot in VI and it ran into that timeout, waited a few hours to let it finish in the background but it looks like it failed and created a new snapshot file. I still see this snapshot in VI and at filelevel (the first snapshot1, check below) .

So do you know how to fix this Problem ?

Jul 08 09:36:53.266: vmx\

SnapshotVMX_Consolidate: starting Jul 08 09:36:53.410: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : open successful (21) size = 146707142656, hd = 0. Type 8 Jul 08 09:36:53.420: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : closed. Jul 08 09:36:53.458: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : open successful (21) size = 146707142656, hd = 0. Type 8 Jul 08 09:36:53.476: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : closed. Jul 08 09:36:53.510: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-flat.vmdk" : open successful (21) size = 146707142656, hd = 0. Type 3 Jul 08 09:36:53.522: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-flat.vmdk" : closed. Jul 08 09:36:53.522: vmx\

ConsolidateOnlineCB: nextState = 0 uid 0 Jul 08 09:36:53.524: vcpu-0\

Sync monModules(1). Jul 08 09:36:53.524: vcpu-1\

Sync monModules(1). Jul 08 09:36:53.524: vcpu-1\

Done Sync monModules(1). Jul 08 09:36:53.524: vcpu-0\

Done Sync monModules(1). Jul 08 09:36:53.534: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : closed. Jul 08 09:36:53.534: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-flat.vmdk" : closed. Jul 08 09:36:53.534: vmx\

Virtual Device for scsi0:0 was already successfully destroyed Jul 08 09:36:53.534: vmx\

ConsolidateOnlineCB: nextState = 1 uid 0 Jul 08 09:36:53.570: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : open successful (21) size = 146707142656, hd = 0. Type 8 Jul 08 09:36:53.584: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : closed. Jul 08 09:36:53.601: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : open successful (21) size = 146707142656, hd = 0. Type 8 Jul 08 09:36:53.610: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : closed. Jul 08 09:36:53.631: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-flat.vmdk" : open successful (21) size = 146707142656, hd = 0. Type 3 Jul 08 09:36:53.644: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-flat.vmdk" : closed. Jul 08 09:36:53.696: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : open successful (21) size = 146707142656, hd = 0. Type 8 Jul 08 09:36:53.700: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : closed. Jul 08 09:36:53.756: vmx\

DUMPER: Creating checkpoint file /vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-Snapshot2.vmsn Jul 08 09:36:53.771: vmx\

FILEIO: pread got 0 out of 4 bytes. Jul 08 09:36:53.788: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : open successful (21) size = 146707142656, hd = 0. Type 8 Jul 08 09:36:53.858: vmx\

DISKLIB-VMFS_SPARSE :ExtentCreate "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000002-delta.vmdk" : success Jul 08 09:36:53.887: vmx\

DISKLIB-LINK : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000002.vmdk" : creation successful. Jul 08 09:36:53.916: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000002-delta.vmdk" : open successful (17) size = 146707142656, hd = 0. Type 8 Jul 08 09:36:53.963: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000002-delta.vmdk" : closed. Jul 08 09:36:53.966: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : closed. Jul 08 09:36:54.048: vmx\

VMXVmdb_LoadRawConfig: Loading raw config Jul 08 09:36:54.056: vmx\

ConsolidateOnlineCB: nextState = 1 uid 2 Jul 08 09:36:54.117: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000002-delta.vmdk" : open successful (21) size = 146707142656, hd = 0. Type 8 Jul 08 09:36:54.120: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000002-delta.vmdk" : closed. Jul 08 09:36:54.132: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : open successful (21) size = 146707142656, hd = 0. Type 8 Jul 08 09:36:54.142: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : closed. Jul 08 09:36:54.159: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000002-delta.vmdk" : open successful (21) size = 146707142656, hd = 0. Type 8 Jul 08 09:36:54.162: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000002-delta.vmdk" : closed. Jul 08 09:36:54.173: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : open successful (21) size = 146707142656, hd = 0. Type 8 Jul 08 09:36:54.176: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : closed. Jul 08 09:36:54.190: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-flat.vmdk" : open successful (21) size = 146707142656, hd = 0. Type 3 Jul 08 09:36:54.193: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-flat.vmdk" : closed. Jul 08 09:36:54.199: vmx\

Checkpoint_Unstun: vm stopped for 675661 us Jul 08 09:36:54.302: vcpu-0\

Sync monModules(5). Jul 08 09:36:54.303: vcpu-0\

Done Sync monModules(5). Jul 08 09:36:54.404: vcpu-1\

Sync monModules(5). Jul 08 09:36:54.404: vcpu-1\

Done Sync monModules(5). Jul 08 09:36:54.407: vmx\

DISK: OPEN scsi0:0 '/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000002.vmdk' persistent R[(null)] Jul 08 09:36:54.454: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000002-delta.vmdk" : open successful (8) size = 282112, hd = 614466. Type 8 Jul 08 09:36:54.455: vmx\

DISKLIB-DSCPTR: Opened : "vmsrv023-000002-delta.vmdk" (0x8) Jul 08 09:36:54.455: vmx\

DISKLIB-LINK : Opened '/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000002.vmdk' (0x8): vmfsSparse, 286537388 sectors / 139911 Mb. Jul 08 09:36:54.496: vmx\

DISKLIB-LIB : Opened "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000002.vmdk" (flags 0x8). 883C114 Jul 08 09:36:54.530: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001-delta.vmdk" : open successful (8) size = 31088463360, hd = 385092. Type 8 Jul 08 09:36:54.530: vmx\

DISKLIB-DSCPTR: Opened : "vmsrv023-000001-delta.vmdk" (0x8) Jul 08 09:36:54.530: vmx\

DISKLIB-LINK : Opened '/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001.vmdk' (0x8): vmfsSparse, 286537388 sectors / 139911 Mb. Jul 08 09:36:54.568: vmx\

DISKLIB-LIB : Opened "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000001.vmdk" (flags 0x8). 88FA684 Jul 08 09:37:24.676: vmx\

DISKLIB-VMFS : "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-flat.vmdk" : open successful (10) size = 146707142656, hd = 282694. Type 3 Jul 08 09:37:24.676: vmx\

DISKLIB-DSCPTR: Opened : "vmsrv023-flat.vmdk" (0xa) Jul 08 09:37:24.676: vmx\

DISKLIB-LINK : Opened '/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023.vmdk' (0xa): vmfs, 286537388 sectors / 139911 Mb. Jul 08 09:37:24.681: vmx\

DISKLIB-LIB : Opened "/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023.vmdk" (flags 0xa). 88FA684 Jul 08 09:37:24.769: vmx\

DISK: OPEN '/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000002.vmdk' Geo (17836/255/63) BIOS Geo (0/0/0) freeSpace=245224Mb, lsilogic Jul 08 09:37:24.772: vmx\

Creating virtual dev for 0:0 Jul 08 09:37:24.775: vmx\

DumpDiskInfo: scsi0:0 createType=11, capacity = 286537388, numLinks = 3, allocationType = 0 Jul 08 09:37:24.775: vmx\

SCSIDiskESXPopulateVDevDesc: Using FS backend Jul 08 09:37:24.775: vmx\

VMMon_SetStorageSchedPolicy: Not supported Jul 08 09:37:24.778: vmx\

ConsolidateOnlineCB: nextState = 2 uid 2 Jul 08 09:37:24.779: vmx\

ConsolidateThread: Created thread 6 for scsi0:0 Jul 08 09:37:24.779: SnapshotVMXCombiner\

ConsolidateOnlineCB: nextState = 3 uid 2 Jul 08 09:37:24.779: SnapshotVMXCombiner\

ConsolidateOnlineCB: Starting combine of scsi0:0 /vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000002.vmdk. 2 links, starting from 1 Jul 08 09:37:24.788: SnapshotVMXCombiner\

DISKLIB-LIB : Combine 2 links at offset 1 in chain 883C114. Jul 08 09:37:24.845: vcpu-0\

HBACommon: First write on scsi0:0.fileName='/vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023-000002.vmdk' Jul 08 09:37:24.853: vcpu-0\

DISKLIB-CHAIN : UpdateContentID: old = 0xc1c14299, new = 0xae71ec6f Jul 08 10:48:08.134: mks\

SOCKET 3 recv error 5: Input/output error Jul 08 10:48:08.143: mks\

SOCKET 3 destroying VNC backend on socket error: 5

ls -ltrh total 167G -rw------- 1 root root 3.0G Apr 9 12:37 vmsrv023-74efdabe.vswp -rw-rr 1 root root 45K Jun 3 10:39 vmware-9.log -rw-rr 1 root root 45K Jun 3 10:52 vmware-10.log -rw-rr 1 root root 45K Jun 3 11:12 vmware-11.log -rw-rr 1 root root 53K Jun 3 17:26 vmware-12.log -rw-rr 1 root root 46K Jun 3 17:41 vmware-13.log -rw-rr 1 root root 37 Jun 3 18:25 vmsrv023-74efdabe.hlog -rw------- 1 root root 252 Jun 3 18:25 vmsrv023-000001.vmdk -rw-rr 1 root root 47K Jun 3 18:25 vmware-14.log -rw------- 1 root root 8.5K Jun 3 18:25 vmsrv023.nvram -rw------- 1 root root 263 Jul 3 23:11 vmsrv023.vmxf -rw------- 1 root root 19K Jul 8 10:37 vmsrv023-Snapshot2.vmsn -rw------- 1 root root 29G Jul 8 10:37 vmsrv023-000001-delta.vmdk -rwxr-xr-x 1 root root 2.2K Jul 8 10:37 vmsrv023.vmx -rw------- 1 root root 494 Jul 8 10:37 vmsrv023.vmsd -rw------- 1 root root 403 Jul 8 10:37 vmsrv023.vmdk -rw------- 1 root root 233 Jul 8 10:37 vmsrv023-000002.vmdk -rw-rr 1 root root 53K Jul 8 11:49 vmware.log -rw------- 1 root root 137G Jul 8 14:28 vmsrv023-flat.vmdk -rw------- 1 root root 1.3G Jul 8 14:28 vmsrv023-000002-delta.vmdk*

[AndreTheGiant]

Do you have enough free space on your datastore?

Andre

[MauroBonder]

Try vmware-cmd pathtovmxfile removesnapshots

**If you found this information useful, please consider awarding points for

"Correct" or "Helpful"**

[marcus_31]

thank you for your replies.

1. vmware-cmd /vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023.vmx removesnapshots

VMControl error -999: Unknown error: SoapError: ServerFaultCode(0) : (Operation failed since another task is in progress.)

is there any way to check if there is still a snapshot process running in the backround? ps ax etc.

supposed in a worst case scenario: how can i fix the vm - if the recovery of snapshots fails?

New Snapshot of current, running state of the server ? Migrating with VMware Converter into a new VM ?

[MauroBonder]

try restart a service of Vcente.

See which hardware (Host) the Vm are using.

Access this ESX, and restart mgmt-vmware ( service mgmt-vmware restart) and (service vmware-vpxa restart).

try again run # vmware-cmd /vmfs/volumes/37997983-4fd6ffeb/vmsrv023/vmsrv023.vmx removesnapshots

apparently this action is locked.

**If you found this information useful, please consider awarding points for

"Correct" or "Helpful"**

[marcus_31]

ok that worked so far

1. vmware-cmd /vmfs/volumes/37997983-4fd6ffeb/vmsrv023 /vmsrv023 .vmx removesnapshots

removesnapshots() = 1

in vi the snapshot does'nt exist anymore but at filelevel: 29G Jul 8 10:37 vmsrv023-000001-delta.vmdk

i guess the file still needs to be merged into my vm but can't see any actions in VC or ESX Server or even

via SSH (ps ax).

ls -ltrhaS

total 168G

-rw-rr 1 root root 37 Jun 3 18:25 vmsrv023-74efdabe.hlog

-rwxrwxrwx 1 root root 84 Jul 8 2009 .lck-1400710400000000

-rwxrwxrwx 1 root root 84 Jul 8 2009 .lck-1200710400000000

-rwxrwxrwx 1 root root 84 Jul 8 2009 .lck-0500710400000000

-rw------- 1 root root 233 Jul 8 10:37 vmsrv023-000002.vmdk

-rw------- 1 root root 252 Jun 3 18:25 vmsrv023-000001.vmdk

-rw------- 1 root root 263 Jul 8 2009 vmsrv023.vmxf

-rw------- 1 root root 403 Jul 8 10:37 vmsrv023.vmdk

-rw------- 1 root root 494 Jul 8 2009 vmsrv023.vmsd

-rwxr-xr-x 1 root root 2.2K Jul 8 2009 vmsrv023.vmx

drwxr-xr-x 1 root root 4.0K Jul 1 18:17 ..

drwxr-xr-x 1 root root 4.0K Jul 8 2009 .

-rw------- 1 root root 8.5K Jun 3 18:25 vmsrv023.nvram

-rw-rr 1 root root 44K Jun 3 10:52 vmware-10.log

-rw-rr 1 root root 44K Jun 3 10:39 vmware-9.log

-rw-rr 1 root root 44K Jun 3 11:12 vmware-11.log

-rw-rr 1 root root 46K Jun 3 17:41 vmware-13.log

-rw-rr 1 root root 46K Jun 3 18:25 vmware-14.log

-rw-rr 1 root root 53K Jun 3 17:26 vmware-12.log

-rw-rr 1 root root 55K Jul 8 2009 vmware.log

-rw------- 1 root root 2.1G Jul 8 2009 vmsrv023-000002-delta.vmdk

-rw------- 1 root root 3.0G Apr 9 12:37 vmsrv023-74efdabe.vswp

-rw------- 1 root root 29G Jul 8 10:37 vmsrv023-000001-delta.vmdk

-rw------- 1 root root 137G Jul 8 2009 vmsrv023-flat.vmdk

Any advices?

Thank you

[MauroBonder]

this Virtual Machine is powered on?

u too can check with vdf -h the free space of disk

**If you found this information useful, please consider awarding points for

"Correct" or "Helpful"**

[marcus_31]

yes the vm is running, database server in production environment and free space should not be the problem, 250GB left.

[MauroBonder · Accepted solution]

you check with vdf -h ?

Get a WINSCP, connect in your ESX and check too.

consider awarding points for

"Correct" or "Helpful"**

[marcus_31]

hm according to the timestamp it is writing changes in a new snapshotfile (2,1GB. Jul 8 2009 vmsrv023-000002-delta.vmd)

i 'd try to delte the snapshot while powering off the vm but iam afraid if the vm will boot normaly. Any way to check if the current running state is consistent compared to the vmdk files.

vdf -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda2 4.9G 1.4G 3.3G 30% /

/dev/sda1 99M 27M 68M 29% /boot

none 132M 0 132M 0% /dev/shm

/dev/sda6 2.0G 105M 1.8G 6% /var/log

/vmfs/devices 506G 0 506G 0% /vmfs/devices

/vmfs/volumes/37997983-4fd6ffeb

1007G 770G 237G 76% /vmfs/volumes/nfs0

/vmfs/volumes/49dcc345-38bb9ab7-4964-001143e3efc7

161G 75G 86G 46% /vmfs/volumes/esxhost34:storage1

/vmfs/volumes/fa6e6c62-a5be11f6

1007G 672G 335G 66% /vmfs/volumes/nfs1

[marcus_31]

YES!!

the first big 30GB snapshot is gone !!

ls -ltrhaS

total 139G

-rw-rr 1 root root 37 Jun 3 18:25 vmserver023-74efdabe.hlog

-rwxrwxrwx 1 root root 84 Jul 8 2009 .lck-1500710400000000

-rwxrwxrwx 1 root root 84 Jul 8 2009 .lck-1400710400000000

-rwxrwxrwx 1 root root 84 Jul 8 2009 .lck-1200710400000000

-rwxrwxrwx 1 root root 84 Jul 8 2009 .lck-0500710400000000

-rw------- 1 root root 226 Jul 8 2009 vmserver023-000002.vmdk

-rw------- 1 root root 233 Jul 8 2009 vmserver023-000003.vmdk

-rw------- 1 root root 252 Jul 8 2009 vmserver023-000001.vmdk

-rw------- 1 root root 263 Jul 8 2009 vmserver023.vmxf

-rw------- 1 root root 403 Jul 8 2009 vmserver023.vmdk

-rw------- 1 root root 494 Jul 8 2009 vmserver023.vmsd

-rwxr-xr-x 1 root root 2.2K Jul 8 2009 vmserver023.vmx

drwxr-xr-x 1 root root 4.0K Jul 1 18:17 ..

drwxr-xr-x 1 root root 4.0K Jul 8 2009 .

-rw------- 1 root root 8.5K Jun 3 18:25 vmserver023.nvram

-rw------- 1 root root 19K Jul 8 2009 vmserver023-Snapshot3.vmsn

-rw-rr 1 root root 44K Jun 3 10:52 vmware-10.log

-rw-rr 1 root root 44K Jun 3 10:39 vmware-9.log

-rw-rr 1 root root 44K Jun 3 11:12 vmware-11.log

-rw-rr 1 root root 46K Jun 3 17:41 vmware-13.log

-rw-rr 1 root root 46K Jun 3 18:25 vmware-14.log

-rw-rr 1 root root 53K Jun 3 17:26 vmware-12.log

-rw-rr 1 root root 65K Jul 8 2009 vmware.log

-rw------- 1 root root 48M Jul 8 2009 vmserver023-000003-delta.vmdk

-rw------- 1 root root 2.2G Jul 8 2009 vmserver023-000002-delta.vmdk

-rw------- 1 root root 3.0G Apr 9 12:37 vmserver023-74efdabe.vswp

-rw------- 1 root root 137G Jul 8 2009 vmserver023-flat.vmdk

[marcus_31]

the snapshots are still there:

-rw------- 1 root root 48M Jul 8 2009 vmserver023-000003-delta.vmdk

-rw------- 1 root root 2.2G Jul 8 2009 vmserver023-000002-delta.vmdk

and vmware-cmd doesn't report any snapshots

vmware-cmd /vmfs/volumes/37997983-4fd6ffeb/vmserver023/vmserver023.vmx removesnapshots

VMControl error -3: Invalid arguments: Virtual machine has no snapshots

any suggestions?

Thank you very much:)

marcus

[marcus_31]

So problem fixed - finally

I just created a new Snapshot with VI (manage snapshots) and removed it with vmware-cmd.

All snapshots have been removed.

The important thing is to be patient!

[MauroBonder]

ok. nice job!

**If you found this information useful, please consider awarding points for

"Correct" or "Helpful"**

[jasoncllsystems]

I always getting a similar problem when the SNAPSHOT get bigger than 20G.

Regards,

Jas aka Superman

MALAYSIA VMware Communities

'If you found this or any other answer useful please consider allocating points for helpful or correct answers ***