Event Commands

Let icinga do stuff on check failure

Example swap:

Each time when a service check execution occures, a post (event) task is triggerred. When swap reaches warning or critical state icinga agent should trigger a script on this event. The script should then do swapoff and swapon to clear cache if enough memory is free. When using icinga agent the script must exist on endpoint node. Additinal sudoers rules must be createt because it is executed as nagios user.

Best case scenario:

After the first warning soft state clear_swap is executed and cleared swap before check reaches even hardstate.

Event Object definition

object EventCommand "event_clear_swap" {
    import "plugin-event-command"
    command = [ PluginDir + "/clear_swap" ]
    timeout = 1m
    arguments += {
        "-s" = "$service.state$"
    }
    vars.clear_swap = "$procs_command$"
}

Service definition

apply Service "Swap" {
    import "template-service-via-agent-swap"

    enable_event_handler = true                  # <---- interessting stuff
    event_command = "sudo_event_clear_swap"      # <---- interessting stuff
    assign where match("*Linux*", host.vars.os) && host.vars.agent

    import DirectorOverrideTemplate
}

clear_swap script

#!/bin/bash

while getopts "s:" opt; do
  case $opt in
    s) servicestate=$OPTARG ;;
  esac
done

[ -z $servicestate ] && exit 3

# Clear SWAP only when enough free memory is available and state is warning or critical
if [ "${servicestate,,}" = "warning" ] || [ "${servicestate,,}" = "critical" ]; then
  /usr/bin/free | \
  /usr/bin/awk '
    /Mem:/  {mem  = $4};
    /Swap:/ {swap = $3};
    END { 
      if (swap <= mem)
        {exit 0}
      else 
        {exit 1}
  }'
  if [ $? -eq 0 ]; then
    /sbin/swapoff -a;
    /sbin/swapon  -a;
  fi
fi

---

Last update: April 13, 2020