If I try to upgrade my ESXi 5 server which is in maintenance mode, I get
C:\Program Files\VMware\VMware vSphere CLI>vihostupdate.pl -server 10.51.80.186
-username root -install -bundle C:\Documents and Settings\wdey\Desktop\scsi-fnic
This operation is NOT supported on 5.0.0 platform.
C:\Program Files\VMware\VMware vSphere CLI>
Look at this thread