We have this feature in the upcoming GA release that prevents the deletion of EPHEMERAL_TEMP_VM

But as a workaround, you may either want to stop the CSE service where the OVA is installed. Powering off the OVA vm also brings down the service. Either of them is required to stop EPHEMERAL_TEMP_VM getting deleted. 

But, the above step should be done before the timeout happens. If you get to notice that it is looping on any existing phase. But timing this (stopping the service) requires little judgement on how fast the cluster creation happen.

Just a quick reminder. If you need to stop the CSE service vApp, please use Power On, force recustomization to restart the VM inside CSE service vApp. OtherWise, the vcd-ke server wouldn't restart. (Please feel free to use systemctl status cse to check whether CSE server is alive).

Thanks for the reply.
Unfortunately if I stop CSE service too early, maybe the task that trigger the issue will not be executed in the VM making the VM useless for troubleshooting.
This future feature is really a must have for the GA,

Yes, troubleshooting is much more difficult without the equivalent to rollbackOnFailure: false.