Are you facing this issue with Flash client or HTML5 client or both ?
Try uploading the ISO by directly connecting the ESXi through web browser.
Try uploading a different ISO and see if you still face the same issue.
I think that's the normal untrusted certificate error. That link (http://vmwlin01...) should be a link to one of your hosts. If not maybe copy and paste it into a new browser tab. You should get a certificate warning. Accept it, then go back to your other tab, try your upload again and it should work.
To upload an ISO to a datastore the most convenient option is IMHO
1. use putty and connect to ESXi
2. cd /vmfs/volumes/datastore/some-dir/
3. wget http://sanbarrow.com/livecds/moa64-nogui/MOA64-nogui-incl-src-111014-efi.iso
This makes sense if you otherwise use your admin-host to download the iso from the ISO-provider first. Safes one step !
Works very well to download new Linux-isos ....
If you already have the iso on your admin host just use WinSCP