We've been doing this a lot without any issues ever, although it is not a supported scenario. Typically when a new version pops up, we put the newer agent into some test pools first and later on move them to production even before Connection Servers are upgraded to the same level. Hence the differences haven't been much more than a one step, so really cannot tell about the situation you're thinking now.
But give it a try, might work.
