Every user with a client and a user account on the AD can enter VCenter Server and "adminstrate".
How do I prevent that?
This shouldnt be the case. Start by checking the permissions you have assigned in vCenter for your datacenter object. The local administrators group may have the administrator role assigned. Check your local admins group on the vcenter server and see if domain users has been added to it.