Would an extremely (in my opinion) handy extension that would work with/like vmtools have to be implemented inside of a distributable VM in order to submit it? What if Windows were targetted with the developed extension? I know that a VM running Windows could not be submitted.
I believe your final submission has to be in the form of a complete VM that is fully redistributable and that does what it is intended to do when powered on. In other words, I don't think you can submit some kind of installer that the user has to install in an existing VM in order to work.
Bear in mind that the idea of the contest is Virtual Appliances -- self-contained VMs that perform a specific function with little to no configuration or hassle on the part of the user. I'm not an expert on the contest rules, but it seems to me that an enhancement to the Tools functionality (while potentially cool and useful) would not qualify as an appliance.