I just found that ESXi 4.0 doesn't support SFTP, so in the WinSCP console, I changed the File Protocol to SCP. That did the trick.
I know I'm necroing, but thanks for replying with the solution.. Fixed me up.
I would never have tried that, since under protocal it has "Allow SCP fallback" checked as default. Go figure.