Redundancy: what protects against what¶
Delivery of a channel rests on five links: the source, the transmission link, the network, the node and, for OTT, the viewer’s connection to the node. Each link has its own protection mechanism, and each mechanism covers only its own link. Below is which scheme to choose for the task, and then each level: what it protects, what it costs and how the levels combine.
The settings are on the reference pages linked from here.
Five levels of redundancy. Levels 3–5 are features of version 2.1 (beta).¶
Choosing a scheme for the task¶
Task |
The schema |
|---|---|
Delivering a channel over the internet between your own nodes |
SRT or PS1 with a buffer sized for the link’s RTT; a second input with the same channel over another path or from another source. |
Headend → multicast receivers (operator IPTV, hotel) |
Backup inputs on every stream and a hot reserve of multicast outputs on two nodes; a third node that only votes, if there are two nodes. |
Managed broadcast network with third-party receivers |
RTP or Pro-MPEG FEC; with two independent networks, ST 2022-7. The sending node is protected by the multicast output reserve. |
OTT for viewers on the internet |
Backup inputs and the OTT/DVR reserve on two or more nodes with public HTTPS addresses; cache or CDN as in 8.1. OTT/DVR reserve and CDN. |
Hotel |
Comparison¶
Level |
Break on failure |
Bandwidth |
What else is needed |
Latency |
|---|---|---|---|---|
Backup inputs |
Stream timeout |
No extra |
A second source |
Does not grow |
ARQ (PS1, SRT, RIST) |
No, as long as the loss is within the buffer |
Headroom for retransmissions |
Return channel |
Receiver buffer, several RTTs |
Pro-MPEG FEC |
No, with small losses |
+10–30 % or more |
— |
Waiting for the FEC matrix |
ST 2022-7 |
No |
× 2 |
A second independent network |
Lagging path window |
Multicast output reserve |
No |
A source for each node |
A second node, Meshwork domain |
“Output delay”, from 500 ms |
OTT/DVR reserve |
A pause while the player detects the failure |
No extra while the nodes are healthy |
A second node with headroom, public addresses |
Does not grow |
1. Source: backup inputs¶
A stream can have several inputs, for example a channel from a satellite and the same channel from the internet. One is active. If it is deemed failed (no data for longer than the stream timeout, or the stream fails a check), the stream moves to the next input in the list and then returns to the main one (Source redundancy).
Protects against a source failure: the satellite, the provider, the encoder, the line to the node.
Cost. A second source of the same channel. The switchover is not seamless: the stream timeout passes before it, and the receivers see a short break.
Monitoring. Running on a backup input is reported by alert code 27 (
input-fallback), a faulty backup input by code 65 (backup-input-error), a stream without a single working input by code 60 (stream-no-source) (List of alerts). Every switch is written to the stream’s event log (Events on the charts).
2. Transmission link: retransmission and FEC¶
Loss and jitter in the link between nodes or from third-party equipment are compensated by the transport protocol (Transport protocols):
ARQ (PS1, SRT, RIST): retransmission of lost packets. A return channel is needed. The cost is a receiver buffer of several RTTs (delay) and bandwidth headroom for the retransmissions.
FEC (Pro-MPEG): constant redundancy without a return channel. The cost is 10–30 % or more on top of the bitrate (depending on the matrix) and a delay for waiting for FEC. It recovers only small losses.
The protocol protects against loss, but not against the link going down: if the link has vanished entirely, only a backup input with a different delivery path helps (level 1).
3. Network: two SMPTE ST 2022-7 paths¶
Important
Beta version 2.1. What is described here exists only in beta version 2.1: it is installed from the beta repository and only with a licence; the stable version 2.0.2 and the demo edition do not have it. The beta version comes without warranty, and support is best effort (Beta testing programme).
One and the same RTP or Pro-MPEG stream goes to the receiver over two independent networks, and the receiver assembles it by packet numbers. A packet lost in one network is taken from the other (Network requirements for ST 2022-7).
Protects against the failure of an entire network (a switch, a line, the power supply) and against loss in it, without switching and without a break.
Cost. Two separate networks, a network card for each, double bandwidth. The receive delay grows by the wait window for the lagging path (450 ms by default, less is possible if the paths are close).
Does not protect against the failure of the sending node itself: it sends both paths.
Monitoring. A lost path is alert code 75 (
input-path-down), paths with different streams are code 76 (input-paths-differ).
4. Node: hot reserve of multicast outputs¶
Important
Beta version 2.1. What is described here exists only in beta version 2.1: it is installed from the beta repository and only with a licence; the stable version 2.0.2 and the demo edition do not have it. The beta version comes without warranty, and support is best effort (Beta testing programme).
Several nodes of one Meshwork domain receive one source and keep identical UDP, RTP or Pro-MPEG outputs to one multicast address. One of them sends. If it, its output or its input fails, another node takes the group over without a break and without repeats, and the receivers stay on the same address (Hot reserve of multicast outputs).
Protects against the failure of a server, its output or its input. The servers can be serviced one at a time: a node that is stopped normally hands over its groups first.
Cost. Each node receives the source and processes the stream the same way as the sender: full resources and incoming bandwidth on every node. The output delay grows by the group’s “Output delay”: from 500 to 2500 ms, 500 by default.
Requirements. One Meshwork domain, the control channel UDP port open between the nodes (43975 by default), synchronised clocks, identical output addresses.
Monitoring. The “Reserves” screen (Reserves), the “Reserve” lane of the stream statistics (The “Reserve” lane), alerts with codes 66–74 (Stream redundancy).
5. OTT viewer: OTT/DVR reserve¶
Important
Beta version 2.1. What is described here exists only in beta version 2.1: it is installed from the beta repository and only with a licence; the stable version 2.0.2 and the demo edition do not have it. The beta version comes without warranty, and support is best effort (Beta testing programme).
The nodes that deliver one channel over HLS, LL-HLS or DASH are joined into an OTT group. The master playlist names backup nodes to the player, and if the node fails, the player continues from another one. A node that cannot serve a new viewer or an archive request redirects them to a node that can (OTT/DVR reserve).
Protects OTT viewers against the failure of a node or of a stream on it; the archive of past programmes is served by a node that has it.
Cost. The viewers of a failed node are spread over the other eligible nodes: out of N nodes, each must handle 1 + 1/(N−1) of its normal load. Every node needs an address that viewers can reach directly.
Beta limitations. The move during playback works only for single HLS streams and only with players that support it (Players and the OTT/DVR reserve).
Monitoring. The “Reserves” screen, alerts with codes 70, 78 and 79.
How the levels combine¶
Backup inputs inside a reserve group. If the sender’s input has gone silent or switched to a backup, and another member has a better stream, the group moves to that member.
ST 2022-7 on the input and the output reserve. The group’s “Output delay” depends on the input’s second-path window (Who sends, and the delay).
Pro-MPEG FEC and the output reserve. Waiting for FEC on the input delays the stream; a seamless switchover of the group covers it for no longer than 2100 ms (FEC wait limit).
Multicast and OTT reserve on the same nodes. The two reserves work independently: multicast groups elect a sender, OTT groups choose backups for viewers. Calculate the resource headroom for both loads at once.
Monitoring¶
A reserve is useful only if someone learns about the switchover. The state of all groups of this node, the members, the source comparison results and the problems are shown on the “Reserves” screen (The “Reserves” screen); switchovers, on the “Reserve” lane of the statistics charts. Set up alert delivery to the on-call engineer (External alert delivery) and metrics export to an external monitoring system (Connecting external monitoring systems).
