I think you should check those 2 links about lockfiles :
VMware Knowledge Base (Manipulating a file on a Datastore fails with the error: device or resource busy / file is locked (1008728)
VMware Knowledge Base (Investigating virtual machine file locks on ESXi (10051))
Else, have you checked if the volume is in use ? If you can , temporarly remove any links between your VMs and the datastore then unmap -l the datastore then rebuild links.
If you use HA check if your datastore is not used for heartbeat (let me know if you need help)
How to monitor the progress of space recycling?
I don't think you'll be able to monitor it only with esxcli command...
Someone succeed to automate it using PowerCLI , you can monitor the action using PowerCLI so :
Else, take a look at the step 7 in this link, VAAI Thin Provisioning Block Reclaim/UNMAP In Action - VMware vSphere Blog
It explain you how you can "monitor" the process via ESXTOP command (you can see the quantity of MB deleted)
But i didn't find any tools or cli to monitor the action as you can monitor the progress of anything in vCenter, sorry..