Yes, so GuestOps lets you run scripts or progams in the GuestOS through the VMware Tools channel. However, since you are trying to install VMware Tools, that's not an option 
In terms of automating the VMware Tools install, you can get more information from the vSphere 5 documentation. There's an option for unattended installation of VMware tools in vCenter.
Sorry, missed the original question. For API invocation, I haven't tried it, but UpgradeTools_Task() may be the solution. You'll see the installerOptions property, which is just a string command line argument. So I would guess despite the name it will also invoke the base setup.exe (and do an install). Never tried it myself though. I found a reference to using it with PowerCLI (which is the same API call) for installation, so once you have your command line arguments correct, should do what you need.