I've got a test windows 2003 Sharepoint server in a VI3.5 environment. The environment is not over loaded and all other servers with it respond ok. I've noticed however that if I point by browser (outside the VMWARE environment) at the sharepoint server it takes up to 60 seconds to come back. However, after that initial connection the http traffic is fast until the next day.
Its as though the initial connection (after it hasn't been used for a while) takes longer while "something" gets established. Could this be something in the VMWARE side of things?
If all else is well in your environment, then this probably seems like a Sharepoint specific issue. Check this link for a possible cause.