The video ram affects the maximum resolution the guest will support. I don't think this has any affect on RDP sessions, but would affect situations where the console is used (and perhaps PCoIP / View).
Sometimes you will see an error when doing a vMotion if it is too low.
I usually set it to autodetect, which normally picks 8MB. If you want to modify it en mass, here's a great script.
Ah ok, so I can just turn it off then ?
You do need some video memory on there, for example 8MB is the minimum requirement to use the wddm driver. Maybe this KB will help.