VM quiesced snapshot Red Hat Linux 8 open-vm-tools v11 and v12 fail to run pre-freeze-script

[GeoPerkins]

We have uploaded debug logs (and more) to VMware support, the case has been escalated by account executive, but remains stuck with no progress, hoping the community can help.

Our goal is to run a custom script provided by the DB vendor (Intersystems Iris DB) to freeze/thaw the DB prior/after the VM snapshot. This is vSphere 7 environment (ESXi U2, vCenter U3).

• WORKING: In RedHat Enterprise Linux 7 (with open-vm-tools 10.3.0.5330 (build-8931395), aka 10.3.0-2.el7) we can successfully perform a quiesced snapshot that incorporates the desired DB freeze/thaw scripted functionality located at
• /usr/sbin/pre-freeze-script
• /usr/sbin/post-thaw-script
• FAILS: In RedHat Enterprise Linux 8 (with open-vm-tools 11.2.5.26209 (build-17337674) aka 11.2.5-2.el8) the quiesced snapshot fails.
• FAILS: In RedHat Enterprise Linux 8 (with open-vm-tools 12.0.5.35655 (build-19716617) aka 12.0.5-2.el8) the quiesced snapshot fails.

Our goal is to use the latest RHEL 8 and latest v12 tools.

Background:

• https://community.intersystems.com/post/intersystems-data-platforms-and-performance-%E2%80%93-vm-bac...
• https://kb.vmware.com/s/article/1006671.

This is the Error in the vmware.log on the guest VM:

2023-01-05T20:45:20.591Z| vmx| | I005: SNAPSHOT: SnapshotPrepareTakeDoneCB: Prepare phase complete (The operation completed successfully).
2023-01-05T20:45:20.613Z| vcpu-1| | I005: ToolsBackup: changing quiesce state: IDLE -> STARTED
2023-01-05T20:45:40.627Z| vmx| | I005: Msg_Post: Warning
2023-01-05T20:45:40.627Z| vmx| | I005: [msg.snapshot.quiesce.vmerr] The guest OS has reported an error during quiescing.
2023-01-05T20:45:40.627Z| vmx| | I005+ The error code was: 2
2023-01-05T20:45:40.627Z| vmx| | I005+ The error message was: Custom quiesce script failed.
2023-01-05T20:45:45.732Z| vmx| | I005: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (29).

We know the script returns an exit code zero -0- and cannot determine why VMware Tools are passing exit code 2.

Any help appreciated. 

[GeoPerkins · Accepted solution]

Turns out this was a ID10T problem.

Summarizing the essential engineering recommendation:

Remove duplicate freeze/thaw scripts present on the server in these directories:

   '"/etc/vmware-tools/backupScripts.d/freezethaw" freeze' 

   '/usr/sbin/pre-freeze-script'

and

   '"/etc/vmware-tools/backupScripts.d/freezethaw" thaw' 

   '/usr/sbin/post-thaw-script'

Today the action was to delete the directory /etc/vmware-tools/backupScripts.d/ (which we believe was empty) and Quiesce Snapshot  incorporating desired freeze was successful.