External scripts for alerts

The “Process” channel of external alert delivery runs your program or script on the node for every alert event: an incident being raised and being cleared. This way alerts reach any system a script can call: an SMS gateway, a messenger, a webhook, your own monitoring or a ticketing system. This page describes how the node runs the command, what the command receives and how the result is handled, and gives two ready-made templates, one for /bin/sh and one for Python 3.

The general model of external delivery and the two other channels, Telegram and e-mail, are described in External alert delivery, and the settings window in Alert destinations.

Channel settings

The channel is configured on the “Process” tab of the “Alert destinations” window. The window opens with the “Alert destinations…” button in the Alerter settings (Alert destinations). The same values are stored in the alerter section of the settings file.

Field	Key	Purpose
“Enable process destination”	process-enable	Enables the channel. Off by default.
“Command”	process-command	The command line: the program and its arguments. Empty by default; an enabled channel with an empty command runs nothing.
“Minimum severity”	process-min-severity	The lower severity threshold for an event: Info (3), Warning (4, the default), Error (5), Critical (6), AdminAction (7).
“Timeout, ms”	process-timeout-ms	How many milliseconds a single run is given: from 100 to 600000, 5000 by default.

All four values are read on every event, so a change takes effect from the next alert, without restarting the service.

How the command is run

No shell. The “Command” line is split into words by shell rules, but no shell is started. Words are separated by spaces; text in single quotes is taken as is; inside double quotes a backslash escapes only " and \; outside quotes a backslash escapes the next character; empty quotes '' give an empty argument. The first word is the program, the rest are its arguments. Pipes |, redirections >, variable expansion $VAR, ~ and file name patterns do not work: they reach the program as plain text. Do all of this inside the script.

Finding the program. If the first word contains /, it is a path to a file: the file must exist and be executable, otherwise every event ends with the error process-command not executable. A name without / is looked up in the service’s PATH. Giving the full path is always the safer choice.

Run environment.

• The user and group are the same as the service’s: pss.

• The working directory is the root /; give full paths in the script.

• Environment variables are inherited from the service (including its PATH), and the PSS_ALERT_* variables are added to them.

• The service’s open files and sockets are not passed to the script: it has only standard input, standard output and standard error.

• The script’s standard output goes to the service’s system log, that is, to the output of journalctl -u pss (Service log). The node reads standard error itself, after the script exits (Result, timeout and log).

What the script receives

Each event is passed in two ways at once: as fields in environment variables and as a whole in JSON on standard input. The node adds no arguments; the script gets only those written in “Command”.

Environment variables

Variable	Value
PSS_ALERT_ID	A stable incident identifier, the same for its raise and its clear, for example 2:12:3.
PSS_ALERT_STATE	raised means the incident has been raised, cleared that it has been cleared.
PSS_ALERT_SEVERITY	Severity as a number: 3–7 (see “Minimum severity” above).
PSS_ALERT_SEVERITY_STR	Severity as a word: Info, Warning, Error, Critical, AdminAction.
PSS_ALERT_CODE	The incident kind code as a number (Alert code catalogue).
PSS_ALERT_CODE_STR	The same code as a text designation, for example no-data, io-error.
PSS_ALERT_SRC_TYPE	Source type: System, Stream, Cluster, EPG, Mosaic, DVB or CS.
PSS_ALERT_SRC_ID	Source identifier, for example the stream number.
PSS_ALERT_NODE	The name of the node on which the incident was raised. For an alert received from a neighbour in the Meshwork domain, this is the neighbour’s name.
PSS_ALERT_DISPLAY_NAME	A human-readable name of the object, for example the stream name.
PSS_ALERT_MESSAGE	The alert text.
PSS_ALERT_TS	The time of this event, in Unix seconds (UTC).
PSS_ALERT_RAISED_TS	The time the incident was raised.
PSS_ALERT_CLEARED_TS	The time the incident was cleared; 0 while the incident is active.

The variables are convenient in /bin/sh: text with spaces and line breaks passes through them intact, and they do not show in the process list (ps).

JSON on standard input. A single line with no trailing newline, after which the input is closed. The JSON carries the same information, but the severity and the source type are given only as numbers (source type: 1 for System, 2 for Stream, 3 for Cluster, 4 for EPG, 5 for Mosaic, 6 for DVB, 7 for CS), and there are no text designations of the code and the severity. Fields:

• always: id, state, ts, raised_ts, cleared_ts, severity, src_type, src_id, node, display_name, code, message;

• for incidents of a single input or output: io_id and io_kind (input or output);

• for AdminAction alerts: admin_action_src;

• for a “flapping” incident: flapping (true) and flap_cycles, the number of its cycles (When the script runs).

The script does not have to read standard input: everything it needs is also in the variables. If it does read the input, it must read it to the end, not just one line.

Example: on node edge-1, stream Sport HD (number 12) has had no data for 15 seconds. When the incident is raised:

{"id":"2:12:3","state":"raised","ts":1761134400,"raised_ts":1761134400,
 "cleared_ts":0,"severity":4,"src_type":2,"src_id":12,"node":"edge-1",
 "display_name":"Sport HD","code":3,
 "message":"[Sport HD] no data 15s (no-data threshold 15s)"}

When the same incident is cleared two minutes later:

{"id":"2:12:3","state":"cleared","ts":1761134520,"raised_ts":1761134400,
 "cleared_ts":1761134520,"severity":4,"src_type":2,"src_id":12,"node":"edge-1",
 "display_name":"Sport HD","code":3,
 "message":"[Sport HD] no data 15s (no-data threshold 15s)"}

Here the JSON is split into lines for readability; the node passes it as a single line. The clear keeps the severity and the text of the incident. There is no “recovered” text: the state field tells you it is a clear. The environment variables for the same raise:

PSS_ALERT_ID=2:12:3
PSS_ALERT_STATE=raised
PSS_ALERT_SEVERITY=4
PSS_ALERT_SEVERITY_STR=Warning
PSS_ALERT_CODE=3
PSS_ALERT_CODE_STR=no-data
PSS_ALERT_SRC_TYPE=Stream
PSS_ALERT_SRC_ID=12
PSS_ALERT_NODE=edge-1
PSS_ALERT_DISPLAY_NAME=Sport HD
PSS_ALERT_MESSAGE=[Sport HD] no data 15s (no-data threshold 15s)
PSS_ALERT_TS=1761134400
PSS_ALERT_RAISED_TS=1761134400
PSS_ALERT_CLEARED_TS=0

When the script runs

• Raise and clear. The script runs once when an incident is raised and once when it is cleared. Intermediate updates of an active incident (a repeat, new text, a change of severity) are not passed to the script.

• Severity threshold. “Minimum severity” applies to the incident, not to each event. An incident below the threshold is not passed on. If its severity later rose to the threshold, the script receives raised once, at the moment of that rise. cleared arrives if and only if raised was passed on, whatever the severity at the moment of the clear. An event counts as passed on once it has been handed to the channel: if the delivery failed, the incident still counts as open, and its clear will be passed on.

• Master switch. With the node’s alerter switched off (Alerter), the script does not run at all, not even for the clear of an incident that has already been reported.

• Domain alerts. When alerts are replicated across the Meshwork domain, the script also receives the neighbours’ alerts; the source node is given in PSS_ALERT_NODE (Domain-wide replication of alerts). If the channel is enabled on several nodes of the domain, each of them reports the same incident. To avoid duplicates, enable the channel on one node only.

• Flapping incidents. If an incident is raised and cleared over and over (by default three cycles in a row no more than 600 seconds apart), the node stops clearing it on every cycle: the incident stays active, and a single clear arrives once the condition has stayed recovered for at least 600 seconds. The script gets one raise and one clear instead of a pair for every cycle. The values are set by the flap-suppress, flap-cycles, flap-window-sec and flap-settle-sec keys in the alerter section of the settings file (The pss.json settings file).

• Sticky alerts, which are cleared only manually, send a clear when the operator clears the alert (Viewing and clearing alerts).

• Service restart. The list of active incidents is kept in memory and is lost on a service restart, without any clears. A condition that still holds is raised again, and the script gets a second raised with the same PSS_ALERT_ID.

So it is best to handle events by PSS_ALERT_ID and PSS_ALERT_STATE: a repeated raised updates a known incident rather than opening a new one.

Queue and order

• The script runs strictly one instance at a time: the next run starts after the previous one has finished, and events are handled in the order they arrive.

• The runs are handled by a separate background thread of the channel. A slow script holds up neither the other channels nor the node’s work, but it does hold up the following events of the same channel.

• While the script is running, new events wait in the channel’s queue, which holds 512 events. When the queue is full, the newest events are dropped and delivery queue full, dropped alert(s) is written to the service log.

• When the service stops or restarts, the events in the queue are dropped; the service waits only for the run already in progress.

Plan the time budget: in a storm of a hundred events, a script that takes two seconds per run falls more than three minutes behind. Move long deliveries to the background (see the next section).

Result, timeout and log

• Exit code 0 means the event has been delivered.

• A non-zero code means a delivery error. A line like [alert:process] deliver failed: process-command '/usr/local/bin/pss-alert.sh' exit 1: <text from stderr> with the last 4 KB of standard error is written to the service log (Service log).

• Standard error is read by the node while the script runs, and shown only together with a non-zero code. Output to stderr with code 0 is not an error and has no effect; heavy output to stderr does not slow the script down and does not lead to a timeout.

• Timeout. If the script has not finished within “Timeout, ms”, the node kills it (SIGKILL) and writes timed out after <N> ms. Only the script itself is terminated; processes it has started keep running.

• No retries. A failed event is not sent again; the channel moves on to the next one.

• Log rate limit. The first error of the channel is written immediately, the following ones no more than once a minute, with the number of errors over that period at the end of the line ((x12)).

Note

Background processes. A process the script leaves in the background does not hold up the channel’s next events, even if it inherited standard error. In versions before 2.0.2.415 such a process stopped the channel until it exited, so on nodes where earlier versions may run, and simply for tidiness, start background work with input and output closed:

some-command </dev/null >/dev/null 2>&1 &

or with output redirected to a log file of your own:

setsid some-command </dev/null >>/var/log/pss-alerts/bg.log 2>&1 &

Security

• The script runs with the rights of the service, as user pss. Make root the owner of the script file and give user pss only read and execute permission (mode 750, group pss): then the service cannot modify the script it runs.

• Do not pass secrets in the arguments of “Command”. Arguments are visible to every user of the system in the process list (ps), are stored in the settings file and are visible in the web interface to the Admin role (for the Restricted admin and Viewer roles the command line is shown masked). Keep tokens and passwords in the script itself or in a separate file readable only by user pss.

• The alert text (stream name, message) comes from the settings and from the sources. Do not substitute it into shell commands unquoted and do not pass it to eval. The templates below pass values only in quotes.

Template for /bin/sh

The script writes one line to the log for every event and exits with code 0. The place for the real delivery is marked with a comment.

#!/bin/sh
# Perfect Streamer alert receiver ("Process" channel).
# The node runs the script on every raise and every clear of an incident.
# The event fields are in the PSS_ALERT_* variables, the whole JSON is on stdin.

LOG=/var/log/pss-alerts/alerts.log

# JSON from stdin: a single line with no trailing newline,
# so read it with cat, not with read.
json=$(cat)

# One line to the log. A write error gives a non-zero code: the node logs it
# in its own log together with the stderr text.
printf '%s %s %s %s [%s] %s: %s\n' \
    "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$PSS_ALERT_STATE" "$PSS_ALERT_SEVERITY_STR" \
    "$PSS_ALERT_NODE" "$PSS_ALERT_CODE_STR" "$PSS_ALERT_DISPLAY_NAME" \
    "$PSS_ALERT_MESSAGE" >> "$LOG" || exit 1

# Plug the real delivery in here, for example a webhook.
# A synchronous call must fit within "Timeout, ms" (5 s by default):
#
# printf '%s' "$json" | curl -sS -m 3 -X POST \
#     -H 'Content-Type: application/json' --data-binary @- \
#     https://hooks.example.com/pss >/dev/null || exit 1
#
# Long delivery goes to the background only, with input and output closed,
# otherwise the channel stalls (see "Result, timeout and log"):
#
# ( printf '%s' "$json" | curl -sS -m 30 -X POST \
#     -H 'Content-Type: application/json' --data-binary @- \
#     https://hooks.example.com/pss ) </dev/null >/dev/null 2>&1 &

exit 0

Template for Python 3

The same in Python 3, standard library only (Python 3.6 or later).

#!/usr/bin/env python3
# Perfect Streamer alert receiver ("Process" channel).
# The node runs the script on every raise and every clear of an incident.
# The event fields are in the PSS_ALERT_* variables, the whole JSON is on stdin.

import json
import os
import sys
import time

LOG = "/var/log/pss-alerts/alerts.log"


def main():
    # All of stdin: a single JSON line with no trailing newline.
    raw = sys.stdin.read()
    alert = json.loads(raw) if raw else {}
    env = os.environ

    line = "{} {} {} {} [{}] {}: {}\n".format(
        time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        alert.get("state", env.get("PSS_ALERT_STATE", "")),
        env.get("PSS_ALERT_SEVERITY_STR", ""),
        alert.get("node", ""),
        env.get("PSS_ALERT_CODE_STR", ""),
        alert.get("display_name", ""),
        alert.get("message", ""),
    )
    # A write error ends the script with a non-zero code, and the error text
    # goes to stderr: the node writes it to its own log.
    with open(LOG, "a", encoding="utf-8") as f:
        f.write(line)

    # Plug the real delivery in here, for example a webhook.
    # The call must fit within "Timeout, ms" (5 s by default):
    #
    # import urllib.request
    # req = urllib.request.Request(
    #     "https://hooks.example.com/pss",
    #     data=raw.encode("utf-8"),
    #     headers={"Content-Type": "application/json"},
    # )
    # urllib.request.urlopen(req, timeout=3).close()
    #
    # Start background processes (subprocess.Popen) with
    # stdin/stdout/stderr=subprocess.DEVNULL and start_new_session=True
    # (see "Result, timeout and log").

    return 0


if __name__ == "__main__":
    sys.exit(main())

Installation and testing

1. Create a log directory writable by user pss:

   sudo install -d -o pss -g pss -m 750 /var/log/pss-alerts
   

2. Save the template, for example as /usr/local/bin/pss-alert.sh (or pss-alert.py), and set the owner and permissions:

   sudo chown root:pss /usr/local/bin/pss-alert.sh
   sudo chmod 750 /usr/local/bin/pss-alert.sh
   

3. Test the script by hand as user pss with a sample event:

   echo '{"id":"2:12:3","state":"raised","severity":4,"node":"edge-1","display_name":"Sport HD","code":3,"message":"test"}' | \
     sudo -u pss env PSS_ALERT_ID=2:12:3 PSS_ALERT_STATE=raised \
       PSS_ALERT_SEVERITY=4 PSS_ALERT_SEVERITY_STR=Warning \
       PSS_ALERT_CODE=3 PSS_ALERT_CODE_STR=no-data \
       PSS_ALERT_NODE=edge-1 PSS_ALERT_DISPLAY_NAME='Sport HD' \
       PSS_ALERT_MESSAGE=test \
       /usr/local/bin/pss-alert.sh
   echo "exit=$?"
   tail -n 1 /var/log/pss-alerts/alerts.log
   

   You should get exit=0 and a new line in the log. Time the run as well: it should be well below “Timeout, ms”.

4. On the “Process” tab of the “Alert destinations” window, enter the full path /usr/local/bin/pss-alert.sh in “Command”, set “Minimum severity”, enable the destination and save.

5. Test delivery on a test stream: interrupt the data feed to its input (for example, stop the sender). After the no-data timeout (15 seconds by default, Alerter settings), a no-data alert with severity Warning is raised, and once data resumes, it is cleared. The lines raised and then cleared should appear in the script’s log.

Common problems

• The script does not run at all. Check the master switch of the alerter and “Enable process destination”, check that “Command” is not empty and that the event severity is not below the threshold. The default threshold is Warning: Info events do not go out.

• The service log shows process-command not executable. The file at the given path does not exist, or user pss has no execute permission. Check ls -l and the permissions on every directory in the path.

• The service log shows posix_spawnp failed. The program could not be started: most often the name is given without a path and is not found in the service’s PATH, or the first line of the script (#!) is wrong. Give the full path.

• The service log shows exit <N>. The script exited with an error; its text from stderr is given in the same line. A common cause is missing write permission for the log directory or for working files: the script runs as pss, not as root, and in the / directory.

• The service log shows timed out after <N> ms. The script did not finish within the timeout: speed up the delivery (a network call timeout shorter than “Timeout, ms”), increase “Timeout, ms” or move the long part to the background with output closed.

• Alerts arrive with a long delay or go missing; the log shows delivery queue full. The script is too slow for the flow of events: the channel’s queue is overflowing. Speed up the script or raise “Minimum severity”.

• The channel has gone silent without a single error in the log (versions before 2.0.2.415). Most likely the script left a background process with inherited standard error, and the node is waiting for it to exit. Find the process (ps -u pss -f), terminate it and fix the way background work is started: with </dev/null >/dev/null 2>&1 or with output redirected to a file. Since version 2.0.2.415 such a process does not stop the channel.

• The script works by hand but not from the node. The cause is the environment: a different user (pss), working directory /, a different PATH, none of the variables of your login session. Test with the command from step 3 of Installation and testing, through sudo -u pss.

• A second raised arrived with the same PSS_ALERT_ID. The service was restarted, and the active incident was detected again (see When the script runs).

• No cleared arrived. The node’s Alerter was switched off while the incident was active, or the service restarted before the clear (see When the script runs).

• The same alert arrives from several nodes. The channel is enabled on several nodes of a domain that replicates alerts. Keep it on one node only.

• The script’s output is not visible. Standard output goes to journalctl -u pss, and standard error reaches the service log only with a non-zero exit code. Write diagnostics to a log file of your own.