You may be able to remove the vswp file but the VMkernel has created it to cover the difference between the memory limit (maximum memory size) and memory reservation (minimum guaranteed amount of memory available for the VM). Therefore if you remove the vswp file for a VM and there is no physical memory available the VM will have to wait for it to become available.
