You are correct about the agent upgrade installation. There is no setup script. One need only run the agent, and set it up as you did initially. It is possible for agent properties to survive upgrades, but one must plan for this by creating .hq/agent.properties according to the HQ Agent Configuration documentation (http://support.hyperic.com/confluence/display/DOCSHQ30/HQ+Agent+Configuration).
From your description, it sounds as if you made the installed HQ server directory the home directory of the user running the server instead of configuring the server directory to be a subdirectory of the home directory. I can see how this might cause some headaches. The installer should ask the installation path for the server (e.g. /home/hq), and then put a sever-3.0.0 directory within that path (e.g. /home/hq/server-3.0.0). If you follow this course, it should have no implications on passwd and shadow files, or on the home directory of the user configured to run the software.
Start-up scripts and other things referencing the server directory would break in the scenario where the directories change names during upgrade, but this can be mitigated through the use of a symbolic link that points to the current server directory for such things relying on the path. For example, creating a symbolic link to server-3.0.0 called server-current, then referencing server-current for things like start-up scripts, etc. In this example, changes to where your start-up scripts look to start HQ can be made by updating the server-current symbolic link after the upgrade is complete.
For the betas that have been upgradeable (i.e. 236 to 242), I've simply been shutting down the old server, moving that directory out of the way by renaming it, then performing the `setup.sh -upgrade`. During the process you are asked where the server you are upgrading from is located, you then tell it the renamed directory path of the old server, and it finds your database and performs the upgrade as expected.
I know that documentation is going to continue to be updated prior to moving 3.0.0 beyond beta, so we will certainly examine the installation instructions as a point for further clarification. Thank you for sharing your experience.