OK, I'm going to put this down to PICNIC - Problem In Chair Not In Computer
There are TWO save buttons on the page and my natural word blindness meant I scanned right over the one that saved the mailserver setting and just pressed the one that saved the alerts settings.
Apparently that one is up for a GUI redesign as support are fed up answering questions about it
If it makes you feel any better, you weren't the only one that did this! I was pulling my hair out, did a Google search, and found your post. All I could do was laugh at myself!