Reserves

Important

Beta version 2.1. What is described here exists only in beta version 2.1: it is installed from the beta repository and only with a licence; the stable version 2.0.2 and the demo edition do not have it. The beta version comes without warranty, and support is best effort (Beta testing programme).

The Meshwork reserve is a hot reserve of multicast outputs between the nodes of one domain. Several nodes receive one and the same source and keep identical outputs to one multicast address and port; only one of them — the elected sender — sends to that address on air. If the sender fails or is stopped, another node takes over the group, and the receivers keep getting the stream from the same address without a break and without repeats.

Besides multicast groups, the Meshwork reserve builds OTT groups: the OTT delivery of one channel on different nodes (OTT groups).

The reserve status and settings of this node are gathered on the “Reserves” screen of the Meshwork area (Meshwork). The demo edition has no reserve: the menu item is unavailable.

Reserve groups

A group is formed by the outputs of the domain’s nodes that send to one and the same multicast address and port and have the “Meshwork reserve” field switched on (the “Meshwork reserve” block of the output editor). Members can be UDP, RTP and Pro-MPEG outputs (UDP, RTP, Pro-MPEG and TCP). MPTS outputs are shown in a group but are not reserved.

What a group needs:

• the reserve is switched on on every node with a member — the “Meshwork reserve” field on the “Settings” tab of the “Reserves” screen, followed by a restart of the service;

• the nodes have a node name and a domain set (Server Settings), and all of them are in one main domain: with nodes of another domain no reserve is formed;

• the UDP port of the control channel (43975 by default) is open between the domain’s nodes;

• the output has a multicast address and a sending interface, “Multicast egress interface”;

• the members’ streams have the same “Stream type” and the same scrambled-channel flag: by these the nodes compare the sources.

The reserve’s control channel is protected by the Meshwork domain key, so it must be the same on all members. Nodes with different keys do not accept each other: in the node table this shows as “Authentication failed” and “Mismatch” in the “Key” column.

Same source or not. The nodes continuously compare their members’ streams by timestamps and content. The result is shown in the “Source” column: “Same source”, “Same signal” (the same timestamps, a different encoding), “Labelled”, “Restamped”, “Unproven”, “Descrambling differs” or “Pending” while there is little data. If the stream’s input sets the timestamps itself (a file, RTMP, RTSP, a transcoder, an MPTS multiplexer), they cannot be compared: set the same “Reserve source label” field in the stream editor on all members.

Who sends

In the default mode — the “Only the elected sender sends (reserve-gate)” field on the “Settings” tab, switched on — only the elected node sends in each group, and the other members keep the stream ready. If this field is switched off, the reserve works in watch mode: the members are checked and shown, but all of them send. The field is read when the service starts; set it the same on all nodes of the domain.

The sender is elected by a vote of the domain’s nodes that have the reserve switched on. After starting, a node first goes through quarantine: it neither votes nor competes until it hears the voters (no longer than 15 seconds). A group moves to another member by itself when:

• the sender’s node or its control channel stops responding;

• the sender’s output stops sending;

• the sender’s input has gone silent or switched to a backup, and another member has a better stream.

If the source has vanished on all members at once, the group does not switch. A node that is being stopped normally hands over its groups first. After frequent switchovers, automatic switching of the group is suspended — the group is “frozen” (alert code 72, reserve-frozen, Stream redundancy).

The sender can also be chosen by hand: the “Move to…” button in the group’s row. The current and the new sender switch at one and the same moment of the group’s timeline, without a break and without a repeat; the new member’s output must be open and sending. The output’s “Reserve priority” field (0–9, higher first) sets the preferred member, and “Give back after (s)” sets after how many seconds of such a member being ready the group returns to it (0 — never: the group stays until it is moved).

A domain of two nodes. When only two nodes vote, a restarted node takes a group from the other no sooner than 30 seconds after its own start, and only if the other node has really dropped out of Meshwork. A third node with the reserve switched on also votes and removes this wait; it does not have to hold members.

Every switchover is written to the stream’s event log and is visible on the “Reserve” lane of the statistics charts (The “Reserve” lane).

Output delay

For a switchover to be seamless, the reserve outputs send the stream with a fixed delay relative to the input — this is the group’s timeline. It is set by the “Output delay (ms)” field on the “Settings” tab: from 500 to 2500, 500 by default. Set one value on all members of the group.

A new value applies to an output when it opens anew: at start or after the output is paused for a few seconds and started. A running output continues with the previous delay, and the “Reserves” screen shows which value will apply. If a reserve member lags behind the sender, the sender itself shifts the group’s timeline later — by no more than 300 ms beyond the set delay and no further than 2500 ms; when the lag passes, the delay returns by 5 ms every 5 seconds.

Second path and Pro-MPEG

A member with an RTP or Pro-MPEG output can also send a second path under SMPTE ST 2022-7 (Network requirements for ST 2022-7). The group’s sender sends both paths. Set the second path the same on all members, including “Path 2 multicast egress interface”: the nodes watch the sender on the second path too. If a member’s second path differs, the group shows the problem “Path 2 differs between members” — switching to that member will move or remove the second path at the receivers.

If the stream’s input itself receives two paths, the output delay must be no less than the input’s second-path wait window (the input’s “Path skew (ms)”) plus 400 ms. Otherwise a switchover on a real loss of the input will not be seamless, and the group shows the problem “Reserve delay shorter than the merge wait”.

A Pro-MPEG output takes part in the sender selection only with a particular FEC matrix and without multiplexing or a passphrase (Pro-MPEG output in a reserve group).

OTT groups

An OTT group joins the OTT delivery (HLS, LL-HLS, DASH) of one channel on different nodes of the domain: streams or adaptive bundles with the same group name. The members find and check each other, and an HLS viewer whose node fails keeps watching from another member without leaving the player (Moving viewers to another node).

How to turn it on. The Meshwork reserve must be on for the node (Reserve groups). Then, for a stream on the “Stream” tab of the editor, in the “Meshwork reserve” block, or in the adaptive bundle’s window, turn on the “OTT reserve” field and set “Reserve group name”:

• up to 64 characters A-Z a-z 0-9 _ . -, case-sensitive; the same name on every node that carries the channel;

• with the field empty, the OTT name is the group name: “URL stream name” for a stream, “OTT name” for a bundle. Without either name the member is not announced to other nodes;

• a change takes effect within a second and does not restart the stream’s OTT delivery.

Only an SPTS stream with OTT delivery on can be a member; for an MPTS stream the field has no effect. An adaptive bundle is one member; its rungs are the bundle’s streams with video. The streams’ sources are compared as in multicast groups, so “Stream type” and “Scrambled channel” must match between the members; adaptive bundles are matched by name. For other nodes to reach this node’s OTT server by name, set the HTTP server’s “Host name” (Streaming HTTP server): without it, other nodes get the node’s address, and only over HTTP unless the node’s certificate is issued for that address.

The OTT group card on the “Status” tab shows the group name, the formats it serves (“Serves: …”), the state, the result of the source comparison and the problems. States:

• “Backed up”: a member on another node can stand in for this one;

• “Unprotected”: other nodes have members, but none of them can stand in for this one;

• “Key pending”: the node has no agreed domain key yet;

• “Not a member”: the OTT reserve is on, but the stream or bundle does not take part: no group name, a name that is not valid, an MPTS stream or no domain yet; the reason is in the problems;

• “Duplicate name”: two streams or bundles of this node have the same group name, and neither of them is announced.

For each member the card shows: “Backup”, which is “Can back up” or “Not offered” with the first check that failed; “Viewer move”, how a viewer would move to this member (“Aligned”, “Same timeline” or “Fresh start”); “Live” and “Archive”. A member on a node with a version before 2.1.0.456 does not take viewers: if the other checks pass, the “Backup” column says “Its version cannot take viewers over yet”. If all members on the other nodes are like that, the group stays unprotected.

Problems of an OTT group:

Problem	What it means
“No group name”	Neither “Reserve group name” nor an OTT name: the member is not announced.
“Reserve name not valid”	A name with characters that are not allowed or longer than 64 characters, read from the settings file. Until it is corrected, the stream is not a member; the OTT name does not replace it.
“No backup on another node”	For 60 seconds no member on another node can stand in for this one: other nodes have no such name, or their members fail the checks (details in the problem’s description). The node raises alert 70 (reserve-unprotected).
“OTT formats differ”	The members serve different OTT formats.
“No host name for the OTT server”	The HTTP server’s “Host name” is not set: other nodes get the node’s address, and only over HTTP unless the node’s certificate is issued for that address. This is information, not an error.
“Host name of the OTT server not valid”	The HTTP server’s “Host name” is not a host name: the node does not tell other nodes the address of its OTT server.
“A node’s clock is too far off”	The clock of another member’s node differs from this node’s clock by more than 240 seconds: that member cannot stand in for this one.
“No view key yet”	The node has an OTT reserve member but no domain view key yet: this node’s viewers do not move to other nodes, and it does not accept failover viewers. After 2 minutes, if the key can come from another node, the node raises alert code 78 (reserve-view-key).
“The view key does not reach this node”	Other nodes have the key, but it does not reach this node; check the connection between the domain’s nodes (alert code 78).
“A node has not taken the view key”	A node with an OTT reserve member has not confirmed the current key for 10 minutes (alert code 78).
“Too many view keys held”	The node holds more than four view keys (alert code 78).
“Overloaded for failover viewers”	The node refuses new failover viewers: CPU or network is at the threshold, or the “Failover sessions cap” has been reached (Overload protection).
“Bitrate ladders differ”	For some of this member’s rungs, the member on another node has no rung with the same codec and a matching bitrate: those rungs have no backup. A member none of whose rungs has matched for 60 seconds is not offered: the “Backup” column says “None of its rungs pairs with this node’s”.
“A node has no https address for backups”	The member’s node does not report the HTTPS address of its OTT server: it is not offered as a backup to viewers over HTTPS. On that node, set the HTTP server’s “Host name” to the name its certificate is issued for.
“A node has only private addresses for backups”	The member’s node reports only private addresses: it is not offered as a backup to viewers on public addresses.

Moving viewers to another node

Since version 2.1.0.458 the live HLS master playlist of an OTT group member names up to two other nodes of the group as backup variants of each rung, after its own. A player that supports backup variants moves to the backup node by itself when the viewer’s node stops serving the live window; the viewer does not leave the player. If the node’s stream has stopped or stalled while the backup is live, the node at once answers such a viewer’s playlist request with 404, and the player moves without waiting for its own timeouts. The viewers of a failed node are spread across its backups.

Players. Backup variants are given to players that can use them: AVPlayer (Safari, iOS, tvOS), hls.js and similar players in the browser, ExoPlayer and Media3. Players that cannot move this way (VLC, mpv, Kodi, GStreamer, ffmpeg-based players, set-top boxes without a User-Agent) get a master playlist without backup lines and work as before; how many such master playlists have been served is shown by the “Masters with no backup lines” row of the “OTT view key” block. What each player gets and what has been tested: Players and the OTT/DVR reserve; what to take into account for caches and CDNs: 8.1. OTT/DVR reserve and CDN. The operator’s own application can request backups with the rsv=1 parameter in the master playlist address or decline them with rsv=0.

For hls.js, set short playlist and segment loading timeouts, about one target segment duration, 2–3 seconds: then the player quickly leaves a stalled node.

Which node becomes a backup. A node is named as a backup only if its member can stand in for this node’s member (the “Backup” column says “Can back up”) and, at the moment of the request:

• it has recently served the live window and serves HLS;

• its clock differs from this node’s clock by no more than 240 seconds;

• the nodes share a view key (below);

• the node is not overloaded (Overload protection);

• it has a rung with the same codec;

• the viewer can reach it the same way: a viewer over HTTPS needs an HTTPS address of the backup node, a viewer on a public address needs a public address. So set the HTTP server’s “Host name” on every node (Streaming HTTP server).

View key. The backup node accepts another node’s viewer by a pass bound to the viewer’s session, the viewer’s node, the stream and the time. Passes are signed with a view key shared by the domain’s nodes that have OTT reserve members. Nothing needs to be set up: the key is created automatically, changes every 7 days and is kept only in memory; its state is shown by the “OTT view key” block on the “Status” tab, and the “Rotate view key” button changes it by hand. If the key does not reach a node, the node raises alert code 78 (reserve-view-key).

Limitations of this beta version. Backup variants are given only to single streams whose members on different nodes take the same source (the “Same timeline” move). Adaptive bundles, LL-HLS and DASH are joined into groups and shown, but they have no backups in the master playlist and manifest yet; such viewers are protected only by the redirect on opening (Redirecting viewers). Subtitles stay on the original node after the move. The archive and catch-up master playlist names no backups: the viewer gets other nodes’ archive by redirect.

Redirecting viewers

Since version 2.1.0.459 the node redirects a viewer to another node of the OTT group (a 302 response) as early as the opening of the HLS or LL-HLS master playlist or the DASH manifest. The redirect needs no support from the player: all players follow it, including VLC and set-top boxes, and for DASH it is the only protection of the live stream in this beta version. It works for streams and adaptive bundles with the “OTT reserve” on.

Catch-up from other nodes’ archives. If the viewer asks for a past period that this node’s archive does not have, the node sends the viewer to the node of the group that holds most of the period; the archive depth may differ between nodes. If no node holds the period, the viewer gets 404 (reserve: no recording for this period); before, it was an empty playlist. For HLS the MPEG-TS archive is taken into account, for LL-HLS and DASH the fMP4 archive.

The move to another node’s archive is chosen by the archive range, from the oldest record to the newest. If the start of the requested period falls within this node’s archive range, the node serves the viewer itself, even if its archive has a gap at that moment (for example, while the input was down): gaps within the range are not moved to another node.

New viewers of a stream that is not running here. A viewer who opens a stream that is not running on this node (stopped or paused, in error, its input has failed or there is no picture yet) is sent by the node to a node where this stream is live. A stream that is just starting is not affected: as soon as a segment appears in its live window, this node serves the viewers. A node restarted with the stream paused does not redirect new viewers until the stream has run at least once.

Login first. For streams with the “OTT reserve” on, the viewer’s login is checked before anything else: with a wrong password the response is always 403, even if the stream is not running. Other streams respond as before.

Connection limit. The login’s “Max connections, HTTP/HLS” (User: add and edit) applies on each node separately: a redirected subscriber can open the same number of sessions on another node as well.

The node is chosen in the same way as a backup for the move (Moving viewers to another node), but the player and the move level are not checked, so adaptive bundles and LL-HLS qualify too. The viewer’s network is not taken into account when a node is chosen for catch-up: the other node must be reachable by the viewer the same way; HTTPS needs an HTTPS address.

Overload protection

A backup node accepts a new viewer of another node (a failover viewer) only while it has headroom: the host’s CPU load and the network interface load are below the error thresholds of these resources in the alert settings (Alerter), and the number of failover sessions has not reached the “Failover sessions cap” on the “Settings” tab (0 means no cap; applied at once). A threshold of 0 turns its check off. The node never refuses its own viewers or failover sessions that are already open. On a refusal the player moves on to the next backup, and the other nodes stop naming the overloaded node as a backup for a while.

The “OTT overload guard” block on the “Status” tab shows whether there is headroom, the causes of the overload, the network and session headroom, the thresholds, and the numbers of failover sessions and refusals. While a cause persists, the node’s OTT groups have the “Overloaded for failover viewers” problem. If the node refuses new failover viewers for 30 seconds in a row, it raises alert code 79 (reserve-ott-overload); the alert clears after 60 seconds without refusals.

The “Reserves” screen

The screen shows the groups that have a member on this node; other nodes show their own. If the reserve is switched off on the node, the screen explains how to switch it on and gives an “Open settings” link.

The “Status” tab.

• The tiles “Groups”, “Groups with problems”, “Errors”, “Warnings”, “Peers on the channel” and “Groups by state”.

• The “Control channel of this node” block: the channel state, the clocks (“Synchronised” or “Not synchronised”), the key fingerprint, the voting nodes, the quarantine and “Timing” — takeover, window and delay; the “OTT groups” and “Reports not sent” counters (reports that did not fit into a datagram or were put off to the next round).

• The “Peers” table: a row for each node of the channel — “Channel”, “Key”, “RTT, ms”, “Clock offset, ms”, “Loss, %”, “Last heard” and “Version”.

• The “Groups” list. For each group: the state, this node’s role, the sender and the election number, the members with priority, role, the “Sending” flag and the source comparison result, the “Output of this node” block and the list of problems. Group states: “Alone” (the other members do not report), “Forming” (members found and being checked), “Mismatch” (the members have a different stream type or scrambled-channel flag) and “Conflict” (someone who is not in the group sends to it).

This node’s role in the group: “Sender”, “Standby”, “Competing”, “Alone”, “Not ready”, “Quarantine”, “Waiting for the domain key”, “Sender, unconfirmed”, “Stepped down”, “Released”, “Handing over”, “Taking over”, “Unverified” or “None”.

The “Output of this node” block shows whether the output is open (“Open: this node sends” or “Closed: another member sends”), whether it is aligned to the group’s timeline, the margin, the delay, the group FEC of a Pro-MPEG output and the datagram counters. Growing counters of late datagrams, overflows and send errors are highlighted; a lagging reserve member is marked: switching to it will not be seamless.

The “Settings” tab. The fields “Meshwork reserve”, “Only the elected sender sends (reserve-gate)”, “Control channel UDP port” (1024–65535, 43975 by default), “Control channel interface” (“All interfaces” by default), “Output delay (ms)”, “Give back after (s)”, “Failover sessions cap” (Overload protection) and “Witness interface” (for a node that only votes; not used yet in this version). Switching the reserve on, the mode, the port and the interface are read when the service starts: changes take effect after a restart.

Group problems

Problem	What it means
“Address conflict”	A source that is not in the group sends to it (alert code 71, reserve-address-conflict).
“Two senders”	Two members send to the group at the same time (alert code 67, reserve-two-senders). In watch mode this is a warning: there all of them send.
“Settings differ”	The members have a different “Stream type” or scrambled-channel flag.
“Content contradicts the settings”	A member’s stream does not match its settings: for example, it has video while the stream type is “Audio only”. The settings decide.
“Source not proven the same”	A member’s source cannot be matched for 30 seconds (alert code 68, reserve-source-mismatch).
“Descrambling differs”	The clear parts of the streams match, but the content does not: one of the nodes is not descrambling.
“Own timestamps, no source label”	The member’s input sets the timestamps itself: set the same “Reserve source label”.
“Member not verified”	A node announces a member, but its reports do not reach this node.
“Two outputs of this node”	Two outputs of one node send to one group.
“Multicast egress interface missing”	The member output has no sending interface set.
“This node cannot watch the group”	This node cannot watch the group; details are in the problem’s description.
“MPTS is listed, not reserved”	An MPTS output is shown in the group but is not reserved.
“DVB input, not yet tested”	The stream comes from a DVB input.
“Path 2 differs between members”	A member’s second path goes to a different address or is not set.
“Reserve delay shorter than the merge wait”	The output delay is shorter than the input’s second-path wait window plus 400 ms.
“ProMPEG member not gated: matrix or packets”	The Pro-MPEG output takes no part in the sender selection: “columns × rows” is not a power of two, or a datagram does not carry 7 TS packets. It always sends, beside the group’s sender, and never takes the group over.
“ProMPEG member not gated: multiplexing or passphrase”	The same, for an output with multiplexing or a passphrase.
“ProMPEG FEC matrix differs”	A member’s FEC matrix differs, or its version does not report it: switching to it changes the matrix at the receivers.

The reserve alerts, codes 66–74, 78 and 79, are listed in Stream redundancy.