Alerts (alerter)¶
The alerter is a built-in alerting service. It watches the state of streams, node hardware, DVB reception and DVR recording, and raises alerts (incidents) when something goes beyond the set limits. The active alert list is visible in the admin panel; in a network with domain-wide replication enabled, every node shows a consolidated alert set for the whole domain.
The alerter also works on a single node — Meshwork is only needed for replicating alerts across the domain.
How an alert works¶
Each incident has a stable identifier: a recurrence of the same problem is the same incident, not a new one.
An incident goes through states: raised (first observed), updated (observed again with a significant change), cleared (the problem is gone). The active set is the incidents that have not yet been cleared.
Each alert has a severity level (in increasing order: information, warning, error, critical, requires administrator action), a code (the numeric kind of incident — see the catalog below), a source (stream, node, storage), a source node (who raised the alert) and a human-readable object name.
Most alerts are level-based: the service periodically re-evaluates the condition and raises or clears the alert on its own. Some alerts that require administrator action are sticky — they do not clear by themselves until the cause is resolved (they are cleared manually, see Viewing and clearing alerts).
The active alert list is kept in memory and is cleared when the service restarts.
Control at the stream level. Each stream has a switch that fully mutes its alerts (together with the alerts of its inputs and outputs), and an alert delay: a source that recovers within the set time does not raise an alert — this smooths out short-lived flapping.
Alerter settings¶
The settings are in the alerts section and require the administrator role. The exact fields are described in the Web interface section; below is what can be configured.
Master switch of the alerting service (enabled by default).
Domain-wide replication (enabled by default): a node shares its alerts with nodes of the same domain and receives theirs — every node shows a consolidated set. Effective only when a domain is set.
Replication of system alerts (disabled by default): alerts about node hardware (host CPU and memory, streamer process and transcoders, network and GPU load, worker threads and the DVB frontend — codes 16–23 and 40–45) are local by default; enable this so they are replicated across the domain as well. Storage alerts (codes 24–25) always remain local.
Stream thresholds: no-data timeout (default 15 s), minimum allowed bitrate (check disabled by default), continuity counter error threshold over a 10 s window (warning and error levels).
Node resource thresholds — “warning / error” pairs in percent for host CPU and memory, the streamer process and transcoders, network interface and GPU load, and DVR storage fill. The host thresholds are set higher than those of the streamer process and transcoders, so that an individual component warns earlier than the whole machine saturates. A value of 0 disables the corresponding level.
DVB frontend thresholds — signal level, SNR/quality, bit error rate and uncorrected blocks; applied to an adapter that has acquired signal lock.
DVR recording thresholds — the number of consecutive failed writes, the recording stall multiplier, the archive gap ratio and the read/write time of chunks to disk.
Alert code catalog¶
The code is the numeric kind of incident, convenient for grouping and localization. The “Replication” column indicates whether the alert is replicated across the domain (when replication is enabled) or stays local to the node.
Streams: state and work cycle (source — stream)
Code |
Event |
Replication |
|---|---|---|
1 |
an input or output entered the error state |
across the domain |
2 |
unrecoverable work-cycle error, manual reset needed (requires administrator action) |
across the domain |
3 |
a running stream produces no data for longer than the timeout |
across the domain |
4 |
the stream bitrate stays below the minimum |
across the domain |
13 |
the stream is paused: no usable input remains (requires administrator action) |
across the domain |
14 |
an input or output self-parked and does not recover on retry (requires administrator action) |
across the domain |
27 |
the stream is running on the backup input — a switchover occurred |
across the domain |
39 |
high CPU load by the stream’s worker thread |
across the domain |
Transport stream conformance (TR 101 290 analyzer) (source — stream; all replicated across the domain)
Code |
Event |
|---|---|
5 |
repeated PCR discontinuities (TR 101 290, 2.3) |
6 |
PCR repetition interval greater than 40 ms (2.3a) |
7 |
PAT repetition interval greater than 500 ms (1.3) |
8 |
PMT repetition interval greater than 500 ms (1.5) |
9 |
PTS repetition interval greater than 700 ms (2.5) |
10 |
PCR accuracy worse than 500 ns (2.4) |
11 |
T-STD buffer over- or underflow for video (3.3) |
12 |
PCR drift greater than 30 ppm (historical: the alert is no longer raised; drift is reported as a metric — Continuous measurements) |
15 |
continuity counter errors above the threshold within a short window (1.4) |
46 |
CRC error in PSI tables (2.2) |
47 |
the transport_error_indicator flag is set (2.1) |
48 |
SI table repetition interval exceeded (SDT / EIT / TDT / NIT) |
49 |
overall verdict: the stream does not conform to TR 101 290 |
Some analyzer codes appear only when the analysis options are enabled (deep analysis, PTS discontinuity analysis, T-STD buffer analysis) and apply only to constant-bitrate streams; the overall verdict (49) is issued only for a reliably constant bitrate. See the Analyzer section for details.
Node resources and hardware (source — system)
Code |
Event |
|---|---|
16 |
host CPU load above the threshold |
17 |
host memory usage above the threshold |
18 |
streamer process CPU above the threshold |
19 |
streamer process memory above the threshold |
20 |
total transcoder CPU above the threshold |
21 |
total transcoder memory above the threshold |
22 |
network interface load above the threshold (per interface) |
23 |
GPU load above the threshold (per device) |
24 |
DVR storage fill above the threshold (per storage) |
25 |
the DVR storage file system is unavailable (requires administrator action) |
40 |
high CPU load by the DVB adapter’s worker thread |
41 |
the DVB frontend lost the signal (lock) |
42 |
DVB signal level below the threshold |
43 |
DVB signal SNR/quality below the threshold |
44 |
DVB bit error rate (BER) above the threshold |
45 |
DVB uncorrected blocks above the threshold |
All alerts in this group are local by default. Codes 16–23 and 40–45 can be replicated across the domain with the system alert replication switch; codes 24–25 always remain local. The DVB frontend threshold alerts (42–45) are evaluated only on an adapter that has acquired signal lock; code 41 is raised on loss of lock. BER (44) is disabled by default, because the raw scale depends on the receiver (see DVB receiver).
Meshwork and lifecycle (source — network/system; local)
Code |
Event |
|---|---|
26 |
a network node went silent — stopped confirming connectivity |
28 |
the streamer process finished starting up (short-lived notification) |
37 |
two nodes with the same name in one domain (requires administrator action) |
DVR recording health (source — stream; local)
Code |
Event |
|---|---|
29 |
DVR segment or index writes are failing — the archive is not being written |
30 |
DVR recording has stalled: there is input, but no new segment is saved for a long time |
31 |
the DVR archive has degraded — gaps in recent coverage |
32 |
the DVR archive index is corrupted (requires administrator action) |
50 |
high read/write time of DVR chunks to disk |
Configuration, transcoders, OTT and certificates (source — system/stream)
Code |
Event |
Replication |
|---|---|---|
33 |
the main configuration failed to load at startup, the service came up on the fallback one (requires administrator action) |
across the domain |
34 |
in low-latency HLS/DASH mode, audio tracks in an unsupported codec were dropped (AAC and AC-3 are supported) |
across the domain |
35 |
failed to issue or renew the HTTPS certificate via the built-in ACME client |
across the domain |
36 |
a configured transcoder failed to load at startup — no executable file or device (requires administrator action) |
across the domain |
38 |
the stream requires a transcoder (software, NVIDIA or Intel VPL) that is unavailable on this node (requires administrator action) |
local |
Conditional access module (CI/CAM, EN 50221) (source — system; local)
Code |
Event |
|---|---|
51 |
the CAM module has been removed (requires administrator action) |
52 |
no subscription: the module reports that the program is not paid for |
53 |
CI/CA error — an unrecoverable channel or conditional access error |
Viewing and clearing alerts¶
The admin panel has an alerts section with a list of alerts. It can be filtered by set (active only / cleared only / all), by severity level, by source type and identifier, by code, by domain and by time, and you can also search by message text. By default, both active and recently cleared records are shown (the transition log), so the total number of rows may exceed the number of active incidents — for the current active set, choose the “active only” mode.
You do not have to poll the state manually: the admin panel receives changes to the active set in real time and updates the list on its own, as soon as an alert is raised, updated or cleared.
Level-based alerts clear automatically when the condition goes away. Sticky alerts that require administrator action (for example, a configuration load error, storage unavailability, a hard work-cycle error) do not clear by themselves: after the cause is resolved, such an incident is cleared (acknowledged) manually. Clearing takes effect on the node that raised the alert: if you see someone else’s (domain-replicated) incident in the network, you must clear it on the source node (it is indicated in the row).
Domain-wide replication of alerts¶
With replication enabled and a domain set, every node shows a consolidated active set for the whole domain: the node’s own alerts plus the alerts of peers in the same domain. Each row indicates the source node, so it is clear on exactly which node the problem occurred.
Replicated across the domain: stream alerts and transport stream conformance alerts, switchover to the backup input, configuration and transcoder load errors, OTT audio rejection, ACME certificate error.
Remain local: resource alerts about hardware and storage, “node went silent”, node name collision, service startup notification, DVR recording health alerts and conditional access module alerts. System hardware alerts (codes 16–23 and 40–45) can be switched to replication with the system alert replication switch.
Each node raises the “network node went silent” alert on its own about the peer that stopped responding — so for a single downed node, its peers will raise such an alert independently of one another.
External alert delivery¶
Besides the list in the browser, alerts can be delivered to an external command, to a Telegram chat and by email (SMTP). Each channel is independent and disabled by default, and carries out delivery on its own background thread, so a slow or unreachable recipient does not hold up the other channels or the service itself.
Each channel has a minimum severity level for delivery (warning by default): every alert visible on the node (its own or replicated across the domain) at or above this level is delivered, both on raise and on clear. The connection parameters of each channel are described in the Web interface section.
Common problems¶
Peer alerts are not visible. Check that domain-wide replication is enabled on the nodes and a common domain is set. System hardware alerts are local by default — enable system alert replication if you need to see them across the domain as well.
A sticky alert does not go away after the cause is resolved. Clear it manually on the source node.
