You can remove it from the inventory with VMs running only if you don't have a distributed service (NSX, vSAN, vSphere Distributed Switch, etc)
If the host is using vSphere Standard Switch it wouldn't be an issue.
I would recommend placing it in maintenance mode and evacuating all its VMs before removing it from inventory, just to be sure.
Edit: I just checked, you can't remove a host from inventory without putting it in maintenance mode first.