We have numerous alerts setup for m/monit and slack integration. Previously we had 3 slack integration alerts
- send green alert to slack for any host, any service, state = "succeeded", and any event except "action" and "monit instance" events
- send orange alert to slack for state = "changed" otherwise same as above
- send red alert to slack for state = "failed" otherwise same as above
This has served us well for quite some time but we have decided to create a specific slack channel for system alerts to prevent cpu/memory/disk/.. issues from overwhelming the primary alert channel we use for app monitoring. To do this I first removed these alert from the three listed above by changing the "any service" portion of the alert to all service groups except "System". This worked fine and we no longer see the system alerts in our "app" channel.
I then created 3 new alerts just like the ones above, but with the "any service" portion of the alert set to the "System" group ONLY. I would assume this would now report those System alerts that have been removed from our "app" channel to this new channel -- but so far they are not getting through. I can run the test on the code to execute and it delivers an alert to slack just fine. It seems as though it is never getting fired when we get the CPU/Memory/Disk/... events.
tl;dr
Setting an alert in m/monit to execute a program for "Any Host", "Servicegroup System", "Any State", "Any Event" does not fire when I receive a CPU/Memory/Disk Space/... event.
TIA.