You should be able to remove the disk and add it to the new server without a problem - you may remove the VM from the inventory on the old ESX, then browse the datastore on the new host and right click the .vmx file and say "add to inventory". Once it's added you can upgrade virtual machine hardware etc.
It looks like you can seamlessly upgrade a VMFS 3 disk to 5, although I will leave the final say to someone more expert than myself
http://blogs.vmware.com/vsphere/2011/07/new-vsphere-50-storage-features-part-1-vmfs-5.html
Regards,
Larry