Hot reserve of multicast outputs

Important

Beta version 2.1. What is described here exists only in beta version 2.1: it is installed from the beta repository and only with a licence; the stable version 2.0.2 and the demo edition do not have it. The beta version comes without warranty, and support is best effort (Beta testing programme).

The hot reserve protects multicast receivers (set-top boxes, TV sets, broadcast receivers) against the failure of a node. Several nodes keep identical outputs to one multicast address and port. One of them sends to the group, and if it fails, another takes the group over, without a break and without repeats, on the same address.

Two words used on this page: a member is a node’s output in a reserve group; the sender is the node that is currently sending to the group. The fields and the screen are described in Reserves; here is what to plan in advance.

Two nodes of a Meshwork domain receive one source and keep an output to the same group 239.0.0.10:1234; node A was the sender and has failed, node B takes the group over at the same point of the timeline; the receivers stay on the same group; a third node only votes.

Node A has failed: node B takes the group over, and the receivers stay on the same address.

When it is needed

  • The receivers take the channel over UDP, RTP or Pro-MPEG multicast and cannot switch to another address by themselves: operator IPTV, hotel TV, a broadcast network.

  • Servers have to be serviced one at a time, without a break on air: a node that is stopped normally hands over its groups first.

OTT viewers need a different mechanism: OTT/DVR reserve. Against loss in the network, the protocol and ST 2022-7 protect (Redundancy: what protects against what).

Nodes

  • How many. Two nodes with members; three nodes with the reserve switched on are better. In a domain of two such nodes, a restarted node takes a group only after 30 seconds, and only if the other has really dropped out. A third node, even without channels of its own, votes and removes this wait (Who sends).

  • Capacity. The reserve node keeps the stream ready: it receives the source and processes it the same way as the sender. Size each node for the full load of its groups.

  • Domain. All nodes are in one Meshwork domain: the one set as the node’s main domain (Server Settings), with the same domain key. No reserve is formed with nodes of another domain.

  • Clocks. The time on the nodes is synchronised (NTP); a node with an unsynchronised clock raises alert code 74 (reserve-clock-unsynced).

  • Version. The reserve is a feature of version 2.1; 2.0.2 does not have it, and there the second server is kept as a cold standby.

Source

The nodes compare the members’ streams by timestamps and content and switch only between members that provably have one source. Therefore:

  • both nodes receive the same source: the same multicast from the satellite receiver, the same stream from the encoder;

  • the streams have the same “Stream type” and scrambled-channel flag;

  • if the input sets the timestamps itself (a file, RTMP, RTSP, a transcoder, an MPTS multiplexer), the sources cannot be compared: set the same “Reserve source label” on all members;

  • the group marks a DVB input with the problem “DVB input, not yet tested”: the reserve was not tested with such inputs in the beta, so test the switchover on your own setup before going on air.

Network

  • Control channel. UDP port 43975 (by default) is open between all nodes of the domain. Over it the nodes vote and watch each other; it is protected by the domain key.

  • Addresses. All members of a group have one and the same multicast address, port and sending interface, “Multicast egress interface”, in the receivers’ network.

  • Receivers follow the group, not the sender. Each node sends from its own address, so after a switchover the group arrives from a different source address. (ASM is a join to a group from any source, SSM only from the listed ones.)

    • ASM receivers (IGMPv2, or IGMPv3 with no source list) work.

    • SSM receivers (IGMPv3 INCLUDE) work only if the join lists the addresses of all nodes of the group.

    • A Perfect Streamer input that receives such a group: leave “SSM source address” and “Path 2 SSM source address” empty.

    No shared address or virtual IP is needed.

  • Switches. IGMP snooping and a single querier (the device that polls IGMP subscribers) in the receivers’ network, as for any multicast (IGMP snooping and querier). Connect the nodes to different core switches: node redundancy does not save you from the failure of a shared switch.

  • Bandwidth. Each node receives the whole source; one node sends the group into the receivers’ network.

Who sends, and the delay

  • Only the elected sender (the default). One node sends to the group, the others keep the stream ready.

  • Watch mode. All of them send, and the reserve only checks the members. Suits a trial run alongside a working setup.

  • Output delay. For a switchover to be seamless, the group’s outputs send the stream with a fixed delay from the input: from 500 to 2500 ms, 500 by default. This is extra channel delay for the receivers; set one value on all members (Output delay).

  • Delay and the input’s second path. If the input receives two ST 2022-7 paths, the “Output delay” must be no less than the second-path window (the input’s “Path skew (ms)”) plus 400 ms: otherwise a switchover on a real loss of the input will not be seamless.

  • Preferred node. The output’s “Reserve priority” sets which member is preferred, and “Give back after (s)” whether the group returns to it after it recovers.

Pro-MPEG and the second path

  • A Pro-MPEG output takes part in the sender election if “columns × rows” is a power of two, there are 7 TS packets in a datagram, there is no multiplexing and no passphrase, and the matrix is the same on all members. Otherwise it always sends, alongside the sender, and does not take the group over (Pro-MPEG output in a reserve group).

  • The second ST 2022-7 path is sent by the group’s sender. Set it the same on all members, including path2-mcast-if: the nodes watch the sender over the second path too (Network requirements for ST 2022-7).

Limitations

  • MPTS outputs are shown in a group but are not backed up.

  • If the source has vanished on all members at once, the group does not switch: node redundancy does not replace backup inputs (1. Source: backup inputs).

  • After frequent switchovers, automatic switching of the group is suspended (alert code 72, reserve-frozen).

  • The demo edition has no reserve.

Before going live

  • On the “Reserves” screen of each node, all nodes of the domain are visible in the “Peers” table with a confirmed key, and the clocks show “Synchronised”.

  • Every group’s source comparison result is “Same source” or “Same signal”, with no problems.

  • A trial switchover with the “Move to…” button in the group’s row passes without a break on the receiver; the switchover event is visible on the “Reserve” lane of the stream statistics.

  • Alerts with codes 66–74 are delivered to the on-call engineer (Stream redundancy).