I'm trying to send out an email by creating a notification that's based om a custom group as object. When I select a VM as object it works but not with custom groups.
Can you screenshot and post the alert & symptom definition
One thing to remember you are alerting on the custom group object not the objects in it. The object within it will contribute to the overall health of the group for example if 1 VM goes offline the group moves from green to orange for example. If you want to apply an alert to the objects within the group then that is doe by policy attached to the group