I see your point, however there are many, many servers that run that "one more attack point" SSH daemon - it is a proven, secure method for remote access. (I'm not saying it's unbeatable, unattackable, or that no one has ever broken into a server over SSH, just that it is one of the better methods for remote access to a system.) SSH access to ESXi will allow you to shoot yourself in the foot pretty easily because the "protection" inherent in using the client, which limits what sorts of tasks you can do, is not there. However, in terms of a hacker's ability to get at your ESXi system and compromise it, download data, etc., I don't see how SSH is substantially more or less secure than the client login itself. Sure, it may let them install some sort of Trojan or other monitoring application that may go undetected, but it isn't going to do any worse a job at preventing someone from stealing your data, deleting virtual machines, uploading their own data, creating VMs, etc., than the VI Client and it's associated interfaces. Furthermore, ESXi also has a web interface that allows you to download files from the host, and a couple other methods of remote access, like the RCLI, that allow people to get at the system in a similar fashion to SSH. In my estimation, the overall impact to the security of the system is minimal when you enable SSH. Furthermore, I do have a management network, and not only to I prevent people from logging in via SSH to my ESXi hosts, I prevent them from making any connections at all to my ESXi hosts.
I guess my "bottom line" point is this: whether or not SSH is enabled on your ESXi hosts is not going to make or break the security of those hosts. It may go a long way toward preventing accidental mistakes on ESXi, but if someone really wants to and really has the ability to screw with your system, they're going to do it with or without SSH.
