Sounds like a blocked port. But you should attach the agent logs to be sure (located on the source physical machine under C:\windows\temp\vmware-temp\)
This is a generic guide and all may not apply - for instance if you are not using VirtualCenter, there is no need to test the connection between ESX and VC.
Check basic network
ping from source system to VC host
ping from VC to ESX target system
ping from source system to ESX target system
ping from ESX target system to VC host
ping from ESX target to source system
Check the port usage (use telnet to verify)
Converter application to remote physical machine - 445 and 139
Converter application to VirtualCenter Server - 443
Converter application to ESX Server 3.x - 443
Physical machine to VirtualCenter Server - 443
Physical machine to ESX Server 3.x - 443 and 902
If you are using Converter 3.0.1 or 3.0.2 make sure you do not have a web server using port 443 while running Converter
Check credentials and privileges
Ensure that each username can login to all systems (you may want to create one admin account on all systems to simply the testing)
Ensure the user has permission to create a VM on the target system
Ensure the user has access to LUNs on the target system
Ensure the user has write privilege on target system storage devices
Verify that the user can create a new VM on each target system from VirtualCenter
Your Virtual Centcer user (if applicable) needs the following privileges:
Inventory of VMs
Verify space requirements
Check available file space on all systems (Depending on your method you will either need space equal to the selected drive or to the space used on the source drive.)
Verify that the Target VMname does not exist on target system
Optionally, turn firewall off on 3.0.x ESX servers
Are you running the P2V through VirtualCenter?
We had a very similar issue.
We resolved it with help from these forums -- added a hosts file entry for the ESX server on the physical server.
We also found that going by IP direct to ESX (no VirtualCenter) worked as well.