---
title: Alerts (alerter)
url: https://doc2.pstreamer.tv/en/manual/meshwork/alerts.html
lang: en
product: Perfect Streamer
version: 2.0.2.362
---

# Alerts (alerter)

The alerter is a built-in alerting service. It watches the state of streams, node hardware, DVB reception and DVR recording, and raises **alerts** (incidents) when something goes beyond the set limits. The active alert list is visible in the admin panel; in a network with domain-wide replication enabled, every node shows a consolidated alert set for the whole domain.

The alerter also works on a single node — Meshwork is only needed for replicating alerts across the domain.

## How an alert works

- Each incident has a **stable identifier**: a recurrence of the same problem is the same incident, not a new one.
- An incident goes through states: **raised** (first observed), **updated** (observed again with a significant change), **cleared** (the problem is gone). The active set is the incidents that have not yet been cleared.
- Each alert has a **severity level** (in increasing order: information, warning, error, critical, requires administrator action), a **code** (the numeric kind of incident — see the catalog below), a **source** (stream, node, storage), a **source node** (who raised the alert) and a human-readable object name.
- A severity level is not attached to an incident forever. While the incident is open, the service re-evaluates it together with the rest of the checks (roughly every 5 seconds) and, on a change, moves the incident into the “updated” state — with the same identifier, so no second row appears in the list; the list is re-sorted instead. Such a change is not sent to the external recipients: a re-evaluation must not turn into an email or a chat message.
- Most alerts are **level-based**: the service periodically re-evaluates the condition and raises or clears the alert on its own. Some alerts that require administrator action are **sticky** — they do not clear by themselves until the cause is resolved (they are cleared manually, see Viewing and clearing alerts).

The active alert list is kept in memory and is cleared when the service restarts.

Control at stream level. Every stream has a switch that mutes its alerts entirely (together with the alerts of its inputs and outputs), and an alert delay: a source that recovers within the time set raises no alert — this smooths out brief flickers. The same delay also swallows a short failure of the main input: if the stream managed to move to a backup before the delay ran out, there will be no input error alert at all — the switchover is reported by the running-on-backup alert (code 27), which the same delay defers. The no-data threshold of a stream is counted separately and is not held back by the delay, so it may fire earlier.

## Alerter settings

The settings are in the alerts section and require the administrator role. The exact fields are described in the [Web interface](../webui/index.md#webui) section; below is what can be configured.

- **Master switch** of the alerting service (enabled by default). A switched-off alerter is silent as a whole: the node raises no alerts of its own and sends nothing to the external destinations — neither its own alerts nor those replicated across the domain from its peers.
- **Domain-wide replication** (enabled by default): a node shares its alerts with nodes of the same domain and receives theirs — every node shows a consolidated set. Effective only when a domain is set.
- **Replication of system alerts** (disabled by default): alerts about node hardware (host CPU and memory, streamer process and transcoders, network and GPU load, worker threads and the DVB frontend — codes 16–23 and 40–45) are local by default; enable this so they are replicated across the domain as well. Storage alerts (codes 24–25) always remain local.
- **Stream thresholds**: no-data timeout (default 15 s), minimum allowed bitrate (check disabled by default), continuity counter error threshold over a 10 s window (warning and error levels).
- **Node resource thresholds** — “warning / error” pairs in percent for host CPU and memory, the streamer process and transcoders, network interface and GPU load, and DVR storage fill. The host thresholds are set higher than those of the streamer process and transcoders, so that an individual component warns earlier than the whole machine saturates. A value of 0 disables the corresponding level.
- **DVB frontend thresholds** — signal level, SNR/quality, bit error rate and uncorrected blocks; applied to an adapter that has acquired signal lock.
- **DVR recording thresholds** — the number of consecutive failed writes, the recording stall multiplier, the archive gap ratio and the read/write time of chunks to disk.

## Alert code catalog

The code is the numeric kind of incident, convenient for grouping and localization. The “Replication” column indicates whether the alert is replicated across the domain (when replication is enabled) or stays local to the node.

**Streams: state and work cycle** (source — stream)

| Code | Event | Replication |
| --- | --- | --- |
| 1 | an input or output entered the error state | across the domain |
| 2 | unrecoverable work-cycle error, manual reset needed (requires administrator action) | across the domain |
| 3 | a running stream produces no data for longer than the timeout | across the domain |
| 4 | the stream bitrate stays below the minimum | across the domain |
| 13 | the stream is paused: no usable input remains (requires administrator action) | across the domain |
| 14 | an input or output self-parked and does not recover on retry (requires administrator action) | across the domain |
| 27 | the stream is running on the backup input — a switchover occurred | across the domain |
| 39 | high CPU load by the stream’s worker thread | across the domain |
| 58 | a source PCR jump beyond the discontinuity window broke the output — what was buffered was discarded | across the domain |
| 59 | the output was resynchronized to real time: the feed does not sustain the configured delay | across the domain |
| 60 | a running stream has no feeding input left | across the domain |
| 61 | a source PCR jump beyond the discontinuity window passed without loss | across the domain |

The level of a code 1 alert is decided by the consequences of the failure, not by the mere fact of an error. A failed output is always “critical”: there is no backup behind it, this is delivery lost. A failed input is critical only if the stream is running through that very input or there is no active input yet; the failure of an input the stream is not using at the moment is a loss of redundancy, and such an alert is given the “error” level. The summary severity of a stream thus agrees with its state: a faulty backup does not paint a healthy stream critical. And when there is nothing left to switch to and the stream has stopped, an incident on an input that is still trying to come up reads as “critical” — there is no redundancy behind it any more. On a switchover to a backup, the incident on the abandoned input is cleared entirely: the input is stopped, while the reason the stream left it stays visible in the interface as history ([Source redundancy](../streamer/spts.md#streamer-spts-reservation)).

Code 60 describes the stream as a whole rather than a single point: it holds as long as no input of a running stream brings any data, and it survives any number of switchovers to a backup — unlike code 1, which is always tied to a particular input or output and therefore breaks up into several short-lived incidents on a stream with reservation. The condition has to last fifteen seconds, so a node start-up and the application of settings, where the inputs stop delivering data for a moment, do not raise code 60. A stream stopped by the administrator or put on pause does not fall under it — that state is described by codes 13 and 14.

Codes 58, 59 and 61 are a recommendation about the input synchronization settings rather than a report of a one-off failure: the node counts such events in a sliding window one hour long and raises an incident when there are more of them than the threshold. They are separated by the cost to the recipient. Code 58 is a source PCR jump that cost content: the synchronization buffer was discarded and the output was broken; the message states how much of the accumulated stream the last such case discarded. Code 61 is the same break of the source timeline, carried through without loss. Code 59 is not about the source at all: the synchronization buffer was running empty, that is, the feed does not sustain the configured delay — the cure is a larger buffer ([Synchronization](../streamer/spts.md#streamer-spts-sync)), and on a transcoder input also bitrate smoothing on the parent stream. All three are sticky: reconnecting the input does not clear them. The incident clears by itself when any of the synchronization settings of the input or the stream is changed — the changed setting is evaluated anew — or after six hours without a single event. A stopped stream keeps the incident: the recommendation still stands.

**Transport stream conformance (TR 101 290 analyzer)** (source — stream; all replicated across the domain)

| Code | Event |
| --- | --- |
| 5 | repeated PCR discontinuities (TR 101 290, 2.3) |
| 6 | PCR repetition interval greater than 40 ms (2.3a) |
| 7 | PAT repetition interval greater than 500 ms (1.3) |
| 8 | PMT repetition interval greater than 500 ms (1.5) |
| 9 | PTS repetition interval greater than 700 ms (2.5) |
| 10 | PCR accuracy worse than 500 ns (2.4) |
| 11 | T-STD buffer over- or underflow for video (3.3) |
| 12 | PCR drift greater than 30 ppm (historical: the alert is no longer raised; drift is reported as a metric — [Continuous measurements](../streamer/analyzer.md#streamer-analyzer-metrics)) |
| 15 | continuity counter errors above the threshold within a short window (1.4) |
| 46 | CRC error in PSI tables (2.2) |
| 47 | the transport_error_indicator flag is set (2.1) |
| 48 | SI table repetition interval exceeded (SDT / EIT / TDT / NIT) |
| 49 | overall verdict: the stream does not conform to TR 101 290 |

Some analyzer codes appear only when the analysis options are enabled (deep analysis, PTS discontinuity analysis, T-STD buffer analysis) and apply only to constant-bitrate streams; the overall verdict (49) is issued only for a reliably constant bitrate. See the [Analyzer](../streamer/analyzer.md#streamer-analyzer) section for details.

**Node resources and hardware** (source — system)

| Code | Event |
| --- | --- |
| 16 | host CPU load above the threshold |
| 17 | host memory usage above the threshold |
| 18 | streamer process CPU above the threshold |
| 19 | streamer process memory above the threshold |
| 20 | total transcoder CPU above the threshold |
| 21 | total transcoder memory above the threshold |
| 22 | network interface load above the threshold (per interface) |
| 23 | GPU load above the threshold (per device) |
| 24 | DVR storage fill above the threshold (per storage) |
| 25 | the DVR storage file system is unavailable (requires administrator action) |
| 40 | high CPU load by the DVB adapter’s worker thread |
| 41 | the DVB frontend lost the signal (lock) |
| 42 | DVB signal level below the threshold |
| 43 | DVB signal SNR/quality below the threshold |
| 44 | DVB bit error rate (BER) above the threshold |
| 45 | DVB uncorrected blocks above the threshold |

All alerts in this group are local by default. Codes 16–23 and 40–45 can be replicated across the domain with the system alert replication switch; codes 24–25 always remain local. The DVB frontend threshold alerts (42–45) are evaluated only on an adapter that has acquired signal lock; code 41 is raised on loss of lock. BER (44) is disabled by default, because the raw scale depends on the receiver (see [DVB receiver](../streamer/dvb.md#streamer-dvb)).

**Meshwork and lifecycle** (source — network/system; local)

| Code | Event |
| --- | --- |
| 26 | a network node went silent — stopped confirming connectivity |
| 28 | the streamer process finished starting up (short-lived notification) |
| 37 | two nodes with the same name in one domain (requires administrator action) |

A code 26 alert holds for the whole time the node is listed as unreachable — up to a week ([Network map](map.md#meshwork-map-nodes)). It clears on its own when the node comes back into contact, and it goes away together with the node’s row if the unreachable node has been forgotten manually.

**DVR recording health** (source — stream; local)

| Code | Event |
| --- | --- |
| 29 | DVR segment or index writes are failing — the archive is not being written |
| 30 | DVR recording has stalled: there is input, but no new segment is saved for a long time |
| 31 | the DVR archive has degraded — gaps in recent coverage |
| 32 | the DVR archive index is corrupted (requires administrator action) |
| 50 | high read/write time of DVR chunks to disk |

**Configuration, transcoders, OTT, certificates and license** (source — system/stream)

| Code | Event | Replication |
| --- | --- | --- |
| 33 | the main configuration failed to load at startup, the service came up on the fallback one (requires administrator action) | across the domain |
| 34 | in low-latency HLS/DASH mode, audio tracks in an unsupported codec were dropped (AAC and AC-3 are supported) | across the domain |
| 35 | failed to issue or renew the HTTPS certificate via the built-in ACME client | across the domain |
| 36 | a configured transcoder failed to load at startup — no executable file or device (requires administrator action) | across the domain |
| 38 | the stream requires a transcoder (software, NVIDIA or Intel VPL) that is unavailable on this node (requires administrator action) | local |
| 55 | the node’s license is expiring: a warning two weeks before the end of the term, and once the term is over — requires administrator action, and the node restarts the service | across the domain |
| 56 | the node’s licence has been found not to be genuine (requires administrator action): the node restarts the service and the start is refused — the service stays stopped until a genuine licence is present on the node and it is started again | across the domain |
| 57 | the node’s licence is genuine but belongs to a generation this version no longer accepts (requires administrator action): the term has not expired and there is no forgery — generation 2 is accepted only as a temporary trial licence, so a perpetual one of that generation is rejected outright | across the domain |

**Descrambling: conditional access module (CI/CAM, EN 50221) and BISS** (source — system; local)

| Code | Event |
| --- | --- |
| 51 | the CAM module has been removed (requires administrator action) |
| 52 | no subscription: the module reports that the program is not paid for |
| 53 | CI/CA error — an unrecoverable channel or conditional access error |
| 54 | descrambling fails: a programme carrying the BISS key that has been set, or one assigned to the CAM, stays scrambled |

## Viewing and clearing alerts

The admin panel has an alerts section with a list of alerts. It can be filtered by set (active only / cleared only / all), by severity level, by source type and identifier, by code, by domain and by time, and you can also search by message text. By default, both active and recently cleared records are shown (the transition log), so the total number of rows may exceed the number of active incidents — for the current active set, choose the “active only” mode.

You do not have to poll the state manually: the admin panel receives changes to the active set in real time and updates the list on its own, as soon as an alert is raised, updated or cleared.

Level-based alerts clear automatically when the condition goes away. Sticky alerts that require administrator action (for example, a configuration load error, storage unavailability, a hard work-cycle error) do not clear by themselves: after the cause is resolved, such an incident is cleared (acknowledged) manually. Clearing takes effect on the node that raised the alert: if you see someone else’s (domain-replicated) incident in the network, you must clear it on the source node (it is indicated in the row). If the source falls silent, the neighbours remove its rows from their consolidated list about forty-five seconds after its last message. This is not a clearing of the incident but a removal of the copy: at the source itself the incident remains, and when the node comes back into contact, the rows not cleared there return to the list. That is also how the expired-licence alert of a stopped node leaves the consolidated list — it cannot be cleared at the source, because the service there is not running.

An incident that is raised and cleared over and over is no longer extinguished by the node right away. After three completed “raise — clear” cycles no more than ten minutes apart, the clears are held back: the row stays in the active set, and a single genuine clear goes out only once the condition has stayed cleared for at least the length of the series itself — by default the same ten minutes. A renewed raise cancels the countdown, and the series reads as one continuous incident rather than a dozen short ones. A first or infrequent recovery is extinguished instantly, as before, while a manual clear, deleting the stream, switching alerts off on the stream and switching off the service itself all bypass the hold. The mechanism is on by default and is needed above all for external delivery: without it a flapping condition would send the recipients a couple of messages per cycle.

## Domain-wide replication of alerts

With replication enabled and a domain set, every node shows a consolidated active set for the whole domain: the node’s own alerts plus the alerts of peers in the same domain. Each row indicates the source node, so it is clear on exactly which node the problem occurred.

- **Replicated across the domain:** stream alerts and transport stream conformance alerts, switchover to the backup input, configuration and transcoder load errors, OTT audio rejection, ACME certificate error, licence expiry and a licence found not to be genuine.
- **Remain local:** resource alerts about hardware and storage, “node went silent”, node name collision, service startup notification, DVR recording health alerts, the demand for a transcoder unavailable on the node, conditional access module alerts and failed descrambling. System hardware alerts (codes 16–23 and 40–45) can be switched to replication with the system alert replication switch.

Each node raises the “network node went silent” alert on its own about the peer that stopped responding — so for a single downed node, its peers will raise such an alert independently of one another.

## External alert delivery

Besides the list in the browser, alerts can be delivered to an external command, to a Telegram chat and by email (SMTP). Each channel is independent and disabled by default, and carries out delivery on its own background thread, so a slow or unreachable recipient does not hold up the other channels or the service itself.

Each channel has a minimum severity level for delivery (warning by default): every alert visible on the node (its own or replicated across the domain) at or above this level is delivered, both on raise and on clear. The connection parameters of each channel are described in the [Web interface](../webui/index.md#webui) section.

The channels are independent only of each other: all three are subject to the master switch of the alerter (Alerter settings). With the service switched off nothing goes out — neither for the node’s own alerts, nor for those replicated from peer nodes, nor for the clearing of an incident that has already been reported.

## Common problems

- **Peer alerts are not visible.** Check that domain-wide replication is enabled on the nodes and a common domain is set. System hardware alerts are local by default — enable system alert replication if you need to see them across the domain as well.
- **A sticky alert does not go away after the cause is resolved.** Clear it manually on the source node.
- **External notifications do not arrive even though the channel is configured and the threshold matches.** Check the master switch of the alerter on the node you expect the delivery from: a switched-off service is silent about someone else’s domain-replicated alerts as well, even though they are still visible in the admin panel.