In some cases I guess the client machine could be a problem, as flash certainly isn't going to win any speed contests, but I think in most cases it's the server, especially if you are using the virtual appliance.  The server side is based on Java and it's slow as heck.  I use the virtual appliance and I've logged into it with SSH and watched it with the "top" command to see what goes on when you do something on the client, and those Java processes within it often peg the CPUs (the appliance is setup to use 2 vCPUs) when you are doing something trivial with the client.  On top of that the vpxclient process within the appliance is competing for CPU in order to actually pass on the instructions to the ESXi host you are working with.

I haven't tried this yet, but just for kicks I'm thinking about bumping up the vCPU count on the appliance to 3 or 4 just as a test to see how much that improves things.  However, given that the host is running on a 12 core AMD system (no hyperthreading), it's not acceptable to have to use that much of it's processing power to get acceptable response time from this client.

I think VMWare chose Java and flash so that they could get this out the door quickly, and they weren't giving any thought to the real world consequences of those choices.  Having been a manager of software developers in a big corporation I know exactly what must have happened.  From on high came the directive that there will be a replacement for the viclient that would be introduced with a version of ESXi that already had a specific rollout date.  As these rollout dates for upgrades affect sales and maintenance contract renewals, they can't be missed without the person causing the miss being blamed for any lost revenue that results (usually a career ending mistake that demands someone in management be sacrificed).  Therefore the software development manager got talked into using flash and Java by the programmers who said that using those tools was the only way they could make it happen anytime close to the immovable deadline.  I've been in that hot seat before, where "No, we can't do it in that timeframe" is not an answer compatible with continued employment.