bought a esxi 4.0 licenses a while ago and jut got them.. but now i want to install esxi 5 instead. can i apply my 4.0 license to version 5? will that work?
This will not work. You'll have to upgrade or get it replaced.
No, you'll need to log into your account on vmware.com and upgrade the license to 5.x and then use that. After you upgrade to 5, you'll have the 60 day evaluation period to enter the upgraded key.
will the upgrade to 5.x be at an additional cost?
If you pay for maintenance, no. If not, yes.