You can enable/disable the gateway services on each connection server separately. So if you enable them on the 2 connection servers for internal connections and disable them on the 2 connection servers for the external users, you should be fine.
You'll probably need to split them up with 2 load balancers I think, one for internal, one for external and put 2 connection servers behind each LB.