What are the pros/cons of running vCenter Appliance VM on the hosts it is managing?
As a general guideline, I prefer NOT to run a vCenter VM on the same host environment it is managing. Rather, I have a dedicated management cluster for such VMs. However, I'm looking at a vCenter installation in a secured network that doesn't have access to the management cluster. It would greatly simplify the configuration if I can install vCenter on the same cluster as the hosts it will manage. But I'd like to know what the implications of doing so are before moving down that path.
A best practice is run the vCenter on a separated cluster, like you already stated, but there is no many implications if you run the vCenter on the same cluster that it is managing, just follow some recommendation, like if your cluster is DRS enable, create a affinity rule to make sure you will find the vSphere ESXi host that vCenter VM is running in case you loose access to the vCenter.
If you don't have access to a Management Cluster to run the vCenter you are left with no choice but to run it on the cluster it is managing.
There are arguments for and against hosting it in the environment it is destined to manage, I have seen a number of clients place their vCenter in a cluster it manages, while it does have some problems for the most part it works fine.
Having VMware HA is a huge benefit to mitigate any issues that can arise from host failure, as HA is not dependant on vCenter it will still work if the vCenter goes down.
Some things (important) that will still keep going if you loose the vCenter Server
Some important things what will stop working if you lose your vCenter
There are many more to both of those lists, but I would like to call those out.
In particular Backups, your backups if you are doing VM level backups with Veeam for example, it is all brokered via the vCenter and when it does you cannot perform any backups, your restore options will be limited to hosts only and / or directories. This is relevant as when you are attempting to recover a vCenter in a cluster it manages you lose the management layer to perform easy restores and need to rely on a slightly different restore method to get yourself back into production mode.
Another note about backups if you are doing VM level backups, try and keep the vCenter in its own separate job away from other VMs, in environments that you have limited storage capabilities a vCenter that is releasing its own snapshot and attempting to release other VMs snapshots that have finished backing up creates long VM stun times. This is more relevant to storage performance, but this can be mitigated by making sure the vCenter is on its own schedule.
As this is more than likely a 2 part issue of Security vs Money, I would present to business if you have that opportunity the pros and cons, recommend a small management cluster and if you start to have failures that warrant a management cluster you will be in a very defendable position. Also make sure your backups are squeaky clean and explore vCenter-less restore and document it.
In 2015 I wouldn't recommend a physical server, I would only recommend virtual servers.
Have fun, let us know what you decide to do