Can you change security policy on a standard network switch to reject or accept with running vms or does host need to be in maintenance mode
You can change it live.
No need to putting the host in to the Maintenancec mode, you can even change the security policy settings per each port group based on network design and requirement, not always in the vSwitch