Alerts (alerter)

The alerter is a built-in alerting service. It watches the state of streams, node hardware, DVB reception and DVR recording, and raises alerts (incidents) when something goes beyond the set limits. The list of active alerts is visible in the admin panel, and in a network with replication across the domain enabled, every node shows a consolidated set of alerts for the whole domain.

The alerter also works on a single node — Meshwork is only needed for replicating alerts across the domain.

How an alert works

• Each incident has a stable identifier: a recurrence of the same problem is the same incident, not a new one.

• An incident goes through states: raised (first observed), updated (observed again with a significant change), cleared (the problem is gone). The active set is the incidents that have not yet been cleared.

• Each alert has a severity level (in increasing order: information, warning, error, critical, requires administrator action), a code (the numeric kind of incident — see the catalogue below), a source (stream, node, storage), a source node (who raised the alert) and a human-readable object name.

• A severity level is not attached to an incident forever. While the incident is open, the service re-evaluates it together with the rest of the checks (roughly every 5 seconds) and, on a change, moves the incident into the “updated” state — with the same identifier, so no second row appears in the list; the list is re-sorted instead. Such a change is not sent to the external recipients: a re-evaluation must not turn into an email or a chat message.

• Most alerts are level-based: the service periodically re-evaluates the condition and raises or clears the alert on its own. Some alerts that require administrator action are sticky: a brief recovery does not clear them. Of these, some clear by themselves only when the cause has really been eliminated, the others remain until they are cleared manually (see Viewing and clearing alerts).

The active alert list is kept in memory and is cleared when the service restarts.

Control at stream level. Every stream has a switch that mutes its alerts entirely (together with the alerts of its inputs and outputs), and an alert delay: a source that recovers within the time set raises no alert — this smooths out brief flickers. The same delay also swallows a short failure of the main input: if the stream managed to move to a backup before the delay ran out, there will be no input error alert at all. The switchover is reported by the running-on-backup alert (code 27), which the same delay defers. The no-data threshold of a stream is counted separately and is not held back by the delay, so it may fire earlier.

Alerter settings

The settings are in the alerts section and require the administrator role. The exact fields are described in the Web interface section; below is what can be configured.

• Master switch of the alerting service (enabled by default). A switched-off alerter is silent as a whole. The node raises no alerts of its own and sends nothing to the external destinations — neither its own alerts nor those replicated across the domain from neighbouring nodes.

• Domain-wide replication (enabled by default): a node shares its alerts with nodes of the same domain and receives theirs; every node shows a consolidated set. Effective only when a domain is set. A super-node in an additional domain accepts alerts only from the neighbours in its list and does not raise a code 26 alert about other nodes of that domain — their state is visible on the network map.

• Replication of system alerts (disabled by default): alerts about node hardware (host CPU and memory, the streamer process and transcoders, network and GPU load, GPU video memory fill, worker threads and the DVB frontend — codes 16–23, 40–45 and 64) are local by default; enable this so they are replicated across the domain as well. Storage alerts (codes 24–25) always remain local.

• Stream thresholds: no-data timeout (default 15 s), minimum allowed bitrate (check disabled by default), continuity counter error threshold over a 10 s window (warning and error levels).

• Backup inputs and resynchronisation (since version 2.0.2.420; the “Input re-sync (codes 58, 59, 61)” and “Backup inputs (code 65)” blocks of the Alerter screen): the alert about a faulty backup (code 65) and the period after which a backup input that is no longer checked leaves its list; the thresholds of codes 58, 59 and 61 (the code 61 threshold is off by default) and the number of quiet hours before they clear (Alert code catalogue).

• CC error alert hold time (code 15, since version 2.0.2.417). The alert is raised, or moves up a level, only when the losses go on for 30 seconds; short pauses between losses do not reset the count. It clears after 60 seconds without losses. The error level is held until the number of errors over the window drops below half the error threshold. So a single lost packet produces no alert, while a source that loses packets all the time produces one steady alert instead of a series of raises and clears. The times are set on the Alerter screen or by the stream-cc-raise-sec and stream-cc-clear-sec keys of the alerter section of the configuration file (The pss.json settings file): from 0 to 3600 seconds; 0 means immediately, as in earlier versions. The TR 101 290 check (1.4 and the overall verdict, code 49) does not change.

• Node resource thresholds — “warning / error” pairs in percent for host CPU and memory, the streamer process and transcoders, network interface and GPU load, and DVR storage fill. The host thresholds are set higher than those of the streamer process and transcoders, so that an individual component warns earlier than the whole machine saturates. A value of 0 disables the corresponding level. The NVIDIA GPU video memory fill thresholds (code 64) are the “GPU video memory” row, 85% and 95% by default (the gpu-mem-warning-pct and gpu-mem-error-pct keys of the alerter section of the configuration file, The pss.json settings file).

• System alert hold time (since version 2.0.2.416). A node resource alert (codes 16–24, 40 and 64) is raised, or moves up a level, only when the value stays beyond the threshold for 30 seconds. It moves down a level or clears when the value stays below the threshold for 60 seconds. So a load that jumps across the threshold from one measurement to the next does not produce alerts that appear and disappear within seconds. The times are set on the Alerter screen or by the system-alert-raise-sec and system-alert-clear-sec keys of the alerter section of the configuration file: from 0 to 3600 seconds; 0 means immediately, as in earlier versions.

• DVB frontend thresholds — signal level, SNR/quality, bit error rate and uncorrected blocks; applied to an adapter that has acquired signal lock.

• DVR recording thresholds — the number of consecutive failed writes, the recording stall multiplier, the archive gap ratio and the read/write time of chunks to disk.

Alert code catalogue

The code is the numeric kind of incident, convenient for grouping and localisation. The “Replication” column indicates whether the alert is replicated across the domain (when replication is enabled) or stays local to the node.

Streams: state and work cycle (source — stream)

Code	Event	Replication
1	an input or output entered the error state	across the domain
2	unrecoverable work-cycle error, manual reset needed (requires administrator action)	across the domain
3	a running stream produces no data for longer than the timeout	across the domain
4	the stream bitrate stays below the minimum	across the domain
13	the stream is stopped, waiting for administrator action: no usable input remains (there are no inputs, or all of them are paused)	across the domain
14	an input or output self-parked and does not recover on retry (requires administrator action)	across the domain
27	the stream is running on the backup input — a switchover occurred	across the domain
39	high CPU load by the stream’s worker thread	across the domain
58	a source PCR jump beyond the discontinuity window broke the output — what was buffered was discarded	across the domain
59	the output was resynchronised to real time: the feed does not sustain the configured delay	across the domain
60	a running stream has no feeding input left	across the domain
61	a source PCR jump beyond the discontinuity window passed without loss	across the domain
62	a separate CI module that the input hands the stream to is unavailable or has stopped returning packets: the input passes the stream on without descrambling	across the domain
65	while the stream runs through one input, its other inputs are in error — the backup is faulty (since version 2.0.2.420)	across the domain

Since version 2.0.2.420 a code 1 alert is always critical: it means the stream has lost its source or its delivery. A failed output always gets it — there is no backup behind an output. A failed input gets it only if the stream is running through that very input or there is no active input yet. Inputs that failed while the stream runs through another input get no code 1 alert of their own: as soon as the stream has taken another input, their rows are cleared, and the faulty backup is described by a single code 65 alert per stream. Thus a faulty backup does not paint a healthy stream critical. Before version 2.0.2.420 each such input held its own code 1 row with the “error” level, and an external destination received its update from “critical” to “error”; now it receives the clearing of the code 1 row and the raising of code 65. The reason the stream left an input stays visible in the interface as history (Source redundancy).

A code 65 alert (“warning” level) lists the faulty inputs of the stream in backup order, each with its last error: no more than five, then how many more. These are both a failed main input while the stream runs through a backup and a backup input that did not come up on a switchover attempt. The alert is raised after the stream alert delay and only once the stream is running steadily through one and the same input; a changed list rewrites the message only after it has held for at least a minute (or the stream alert delay, if that is longer). While the stream runs through no input at all, the alert does not change: the outage is described by codes 60 and 1. The alert clears when no faulty inputs are left — the input has recovered or been paused, the stream has returned to it — and also when the stream is stopped, paused or deleted. A backup input below the active one in the order is no longer checked while the stream is running, so a day after its last error it leaves the list. The period is set by the stream-backup-input-max-age-min key of the alerter section of the settings file (in minutes, 1440 by default, 0 — never drop by age), on the Alerter screen the “Forget a backup input’s error after, min” field; the stream-backup-input-alert key turns the alert off (on by default), on the screen the “Alert on failing backup inputs” switch. An input above the active one is checked all the time and stays in the list until it recovers.

Code 60 describes the stream as a whole rather than a single point: it holds as long as no input of a running stream brings any data, and it survives any number of switchovers to a backup. Code 1, by contrast, is always tied to a particular input or output and therefore breaks up into several short-lived incidents on a stream with redundancy. The condition has to last fifteen seconds, so a node start-up and the application of settings, where the inputs stop delivering data for a moment, do not raise code 60. It does not cover a paused stream or a stream that has stopped waiting for administrator action: that state is described by codes 13 and 14.

Codes 58, 59 and 61 mean a recommendation about the input synchronisation settings rather than a report of a one-off failure: the node counts such events in a sliding window one hour long and raises an incident when there are more of them than the threshold. They are separated by the cost to the recipient. Code 58 is a source PCR jump that cost content: the synchronisation buffer was discarded and the output was broken; the message states how much of the accumulated stream the last such case discarded. Code 61 is the same break of the source timeline, carried through without loss. Since version 2.0.2.420 code 61 has the “information” level (previously “warning”; with the default threshold of external destinations it does not go there) and is off by default: its threshold is set by a separate key stream-sync-absorbed-count of the alerter section (from 0 to 256, 0 — off), and the stream-sync-gap-count key now applies only to code 58. Events are counted even while the alert is off, so turning it on evaluates the accumulated history at once. Code 59 is not about the source at all: the synchronisation buffer was running empty, that is, the feed does not sustain the configured delay. The cure is a larger buffer (Synchronisation), and on a transcoder input also bitrate smoothing on the parent stream. All three are sticky: reconnecting the input does not clear them. The incident clears by itself when any of the synchronisation settings of the input or the stream is changed (the changed setting is evaluated anew), or when the number of events stays below the threshold for two hourly windows in a row. The number of windows is set by the stream-sync-quiet-clear-windows key of the alerter section (from 1 to 24, 2 by default); before version 2.0.2.420 six hours without a single event were needed, and a source with one or two breaks an hour held the row forever. Since version 2.0.2.420 the row of an input the stream has switched away from to another input is cleared too, as is the row of a stream that has been paused. A stopped stream keeps the incident: the recommendation still stands.

Transport stream conformance (TR 101 290 analyser) (source — stream; all replicated across the domain)

Code	Event
5	repeated PCR discontinuities (TR 101 290, 2.3)
6	PCR repetition interval greater than 40 ms (2.3a)
7	PAT repetition interval greater than 500 ms (1.3)
8	PMT repetition interval greater than 500 ms (1.5)
9	PTS repetition interval greater than 700 ms (2.5)
10	PCR accuracy worse than 500 ns (2.4)
11	T-STD buffer over- or underflow for video (3.3)
12	PCR drift greater than 30 ppm (historical: the alert is no longer raised; drift is reported as a metric — Continuous measurements)
15	continuity counter errors above the threshold within a short window (1.4)
46	CRC error in PSI tables (2.2)
47	the transport_error_indicator flag is set (2.1)
48	SI table repetition interval exceeded (SDT / EIT / TDT / NIT)
49	overall verdict: the stream does not conform to TR 101 290

Some analyser codes appear only when the analysis options are enabled (deep analysis, PTS discontinuity analysis, T-STD buffer analysis) and apply only to constant-bitrate streams; the overall verdict (49) is issued only for a reliably constant bitrate. See the Analyser section for details.

Node resources and hardware (source — system)

Code	Event
16	host CPU load above the threshold
17	host memory usage above the threshold
18	streamer process CPU above the threshold
19	streamer process memory above the threshold
20	total transcoder CPU above the threshold
21	total transcoder memory above the threshold
22	network interface load above the threshold (per interface)
23	GPU load above the threshold (per device)
24	DVR storage fill above the threshold (per storage)
25	the DVR storage file system is unavailable (requires administrator action)
40	high CPU load by the DVB adapter’s worker thread
41	the DVB frontend lost the signal (lock)
42	DVB signal level below the threshold
43	DVB signal SNR/quality below the threshold
44	DVB bit error rate (BER) above the threshold
45	DVB uncorrected blocks above the threshold
63	the DVB frontend holds lock, but the multiplex is not received: the PAT has not been parsed for 20 seconds, three re-tunes in a row
64	NVIDIA GPU video memory fill above the threshold (per device)

All alerts in this group are local by default. Codes 16–23, 40–45 and 64 can be replicated across the domain with the system alert replication switch; codes 24–25 and 63 always remain local. The DVB frontend threshold alerts (42–45) are evaluated only on an adapter that has acquired signal lock; code 41 is raised on loss of lock. BER (44) is disabled by default, because the raw scale depends on the receiver (see DVB receiver). Code 63 is raised only on an adapter with streams attached, and the signal figures may stay normal meanwhile. After the third re-tune the adapter retries once every five minutes; the alert clears as soon as the PAT is parsed.

Code 23 tracks GPU load, and code 64 tracks how much of the card’s video memory is in use. A card with full video memory cannot start a new transcoder channel, even though its load may look normal at the same time. Code 64 is raised only for NVIDIA cards that report memory fill; there is none for Intel GPUs. If a card stops reporting the fill, the alert clears. A node of an earlier version (before 2.0.2.416) shows a code 64 alert replicated across the domain without a name until it is upgraded.

Meshwork and lifecycle (source — network/system; local)

Code	Event
26	a network node went silent — stopped confirming connectivity
28	the streamer process finished starting up (short-lived notification)
37	two nodes with the same name in one domain (requires administrator action)

Since version 2.0.2.417, a code 26 alert is raised only when the node has been silent for more than a minute, although it is marked unreachable on the network map after just 15 seconds. So a quick restart of a neighbour produces no alert. The delay is set by the “Node-down alert after, s” field of the “Meshwork contact (advanced)” block of the server settings (Server Settings), and in the settings file by the cluster-node-down-sec parameter of the cluster section (The pss.json settings file): 60 seconds by default, from 0 to 3600; it is never shorter than 15 seconds.

A node that is stopped or restarted deliberately (stopping or restarting the service, the “Restart now” action in the admin panel, restoring the configuration, updating the licence) first tells its neighbours so. For 5 minutes the neighbours do not raise a code 26 alert for it, and the node table lists it as “Restarting (N s)” (Nodes); if the node has not come back by then, the alert is raised as usual. The period is set by the “Planned-stop grace, s” field of the same block, and in the settings file by the cluster-leave-grace-sec parameter of the same section: 300 seconds by default, from 0 to 86400. The notice reaches the neighbours the node connects to itself; a neighbour that only connects to it (for example, a node behind NAT) does not get the notice, and the usual delay applies to it. Nodes of versions before 2.0.2.417 ignore the notice. Since version 2.0.2.420, a node warns only neighbours of version 2.0.2.417 and newer (and those whose version is not yet known), and all of them at once, so in a network with older nodes stopping the service takes about 3 seconds less. A service stop caused by the licence (expired, not genuine or rejected) does not warn the neighbours.

A code 26 alert holds for the whole time the node is listed as unreachable — up to a week (Network map). It clears on its own when the node comes back into contact, and it goes away together with the node’s row if the unreachable node has been forgotten manually.

DVR recording health (source — stream; local)

Code	Event
29	DVR segment or index writes are failing — the archive is not being written
30	DVR recording has stalled: there is input, but no new segment is saved for a long time
31	the DVR archive has degraded — gaps in recent coverage
32	the DVR archive index is damaged and is not rebuilt — a disk failure (requires administrator action); individual damaged index records are skipped without this alert
50	high read/write time of DVR chunks to disk

Configuration, transcoders, OTT, certificates and licence (source — system/stream)

Code	Event	Replication
33	the main configuration failed to load at startup, the service came up on the fallback one (requires administrator action)	across the domain
34	in low-latency HLS/DASH mode, audio tracks in an unsupported codec were dropped (AAC and AC-3 are supported)	across the domain
35	failed to issue or renew the HTTPS certificate via the built-in ACME client	across the domain
36	a configured transcoder failed to load at startup — no executable file or device (requires administrator action)	across the domain
38	the stream requires a transcoder (software, NVIDIA or Intel VPL) that is unavailable on this node (requires administrator action)	local
55	the node’s licence is expiring: a warning two weeks before the end of the term, and once the term is over — requires administrator action, and the node restarts the service; the service will not start again until a valid licence is present on the node	across the domain
56	the node’s licence has been found not to be genuine (requires administrator action): the node restarts the service and the start is refused — the service stays stopped until a genuine licence is present on the node and it is started again	across the domain
57	the node’s licence is genuine but belongs to a generation this version no longer accepts (requires administrator action): the term has not expired and there is no forgery — generation 2 is accepted only as a temporary trial licence, so a perpetual one of that generation is rejected outright	across the domain

A code 35 alert clears on its own at the next successful certificate issuance, and also when automatic issuance is turned off. Since version 2.0.2.417, an alert about a problem on the node itself (no HTTPS server with the host name, the HTTP-01 validation port is busy, an ACME account key error) clears as soon as the next issuance attempt gets past that step; previously it held until automatic issuance was turned off (HTTPS certificates).

Descrambling: conditional access module (CI/CAM, EN 50221) and BISS (source — system; local)

Code	Event
51	the CAM module has been removed (requires administrator action)
52	no subscription: the module reports that the programme is not paid for
53	CI/CA error — an unrecoverable channel or conditional access error
54	descrambling fails: a programme carrying the BISS key that has been set, or one assigned to the CAM, stays scrambled

A separate CI module that the input hands the stream to (CI/CAM on a stream input) reports a failure with stream code 62, not with the codes of this group. There is one such alert per input, even if the input uses several slots. It clears when the module has responded again for 30 seconds, when the module is removed from the input, or when the input is deleted. A module that responds but descrambles nothing does not raise this alert.

Viewing and clearing alerts

The admin panel has an alerts section with a list of alerts. It can be filtered by set (active only / cleared only / all), by severity level, by source type and identifier, by code, by domain and by time, and you can also search by message text. By default, both active and recently cleared records are shown (the transition log), so the total number of rows may exceed the number of active incidents. For the current active set, choose the “active only” mode.

You do not have to poll the state manually: the admin panel receives changes to the active set in real time and updates the list on its own, as soon as an alert is raised, updated or cleared.

Level-based alerts clear automatically when the condition goes away. Sticky alerts that require administrator action (for example, a configuration load error, storage unavailability, a hard work-cycle error) do not clear by themselves: after the cause is resolved, such an incident is cleared (acknowledged) manually. Clearing takes effect on the node that raised the alert: if you see someone else’s (domain-replicated) incident in the network, you must clear it on the source node (it is indicated in the row). If the source falls silent, the neighbours remove its rows from their consolidated list about forty-five seconds after its last message. This is not a clearing of the incident but a removal of the copy: at the source itself the incident remains, and when the node comes back into contact, the rows not cleared there return to the list. That is also how the expired-licence alert of a stopped node leaves the consolidated list — it cannot be cleared at the source, because the service there is not running.

An incident that is raised and cleared over and over is no longer cleared by the node right away. After three completed “raise — clear” cycles no more than ten minutes apart, the clears are held back: the row stays in the active set, and a single genuine clear goes out only once the condition has stayed cleared for at least the length of the series itself (by default the same ten minutes). A renewed raise cancels the countdown, and the series reads as one continuous incident rather than a dozen short ones. A first or infrequent recovery is cleared instantly, while a manual clear, deleting the stream, switching alerts off on the stream and switching off the service itself all bypass the hold. The mechanism is on by default and is needed above all for external delivery: without it a flapping condition would send the recipients a couple of messages per cycle.

Domain-wide replication of alerts

With replication enabled and a domain set, every node shows a consolidated active set for the whole domain: the node’s own alerts plus the alerts of neighbours in the same domain. Each row indicates the source node, so it is clear on exactly which node the problem occurred.

• Replicated across the domain: stream alerts and transport stream conformance alerts, switchover to the backup input, configuration and transcoder load errors, OTT audio rejection, ACME certificate error, licence expiry, a licence found not to be genuine and a licence of a generation this version no longer accepts.

• Remain local: resource alerts about hardware and storage, “node went silent”, node name collision, service startup notification, DVR recording health alerts, the demand for a transcoder unavailable on the node, conditional access module alerts, failed descrambling, and the loss of the multiplex under DVB lock (code 63). System hardware alerts (codes 16–23 and 40–45) can be switched to replication with the system alert replication switch.

Each node raises the “network node went silent” alert on its own about the neighbour that stopped responding, so for a single downed node, its neighbours will raise such an alert independently of one another.

External alert delivery

Besides the list in the browser, alerts can be delivered to an external command, to a Telegram chat and by email (SMTP). Each channel is independent and disabled by default, and carries out delivery on its own background thread, so a slow or unreachable recipient does not hold up the other channels or the service itself.

Each channel has a minimum severity level for delivery (warning by default): every alert visible on the node (its own or replicated across the domain) at or above this level is delivered, both on raise and on clear. The level applies to the incident as a whole: if the severity rose to the level only after the raise, the raise is reported at the moment of that rise, and the clear is always reported when the raise was — raise and clear arrive as a pair. The connection parameters of each channel are described in the Web interface section. How the external command is run, what it receives and how to write a script for it, with ready-made templates for /bin/sh and Python, is described in External scripts for alerts.

The channels are independent only of each other: all three are subject to the master switch of the alerter (Alerter settings). With the service switched off nothing goes out — neither for the node’s own alerts, nor for those replicated from neighbouring nodes, nor for the clearing of an incident that has already been reported.

Common problems

• Neighbours’ alerts are not visible. Check that domain-wide replication is enabled on the nodes and a common domain is set. System hardware alerts are local by default — enable system alert replication if you need to see them across the domain as well.

• A sticky alert does not go away after the cause is resolved. Clear it manually on the source node.

• External notifications do not arrive even though the channel is configured and the threshold matches. Check the master switch of the alerter on the node you expect the delivery from: a switched-off service is silent about someone else’s domain-replicated alerts as well, even though they are still visible in the admin panel.