Hello Hans,
You need to adjust your alert thresholds probably, you can do this from the Manage Alerts dialog, select the alert that needs adjustment and click the pencil icon to Edit the Alert. Then adjust the time and the number of matches in the Raise an alert section of the dialog to adjust the threshold. Also make sure your alert is enabled, if not enable it to get it to alert.
Like Steve said a picture of what your edit alert dialog looks like will help in answering you in better way.
Thanks,
-Yogita.