What are some industry-standard metrics for determining when a vm has insufficient memory?
You can always look for execessive paging inside of the guest OS.
As written, you must watch in guest OS parameters, like memory paging.
For host memory you can use esxtop: http://www.yellow-bricks.com/esxtop/
Andre