Building a Meshwork network¶

Configuring Meshwork comes down to making sure each node knows its own name and domain and has at least one reachable neighbour. From there the domain builds up the remaining links on its own. All actions are performed in the web admin panel; changing Meshwork settings requires the administrator role. The exact fields and their location are described in the Web interface section.

Requirements and reachability¶

Network reachability. A node must be able to establish a TCP connection to the administrative web server of a neighbour. The port is taken from the neighbour’s address (“host:port”) or from the public port that the neighbour announces about itself (Node parameters). Meshwork has no port of its own: this is the neighbour’s web server port or, when HTTPS is enabled on the neighbour, its TLS port. If a neighbour behind NAT has set a public port, this is the port forwarded to its web server. The values for a fresh installation are given in Initial settings; the required port of a neighbour must be open to the other nodes.
HTTPS for auto-discovery over the internet. If auto-discovered domain nodes communicate over public (internet) addresses, they need to establish a secure connection (HTTPS) between themselves in order to safely confirm each other’s authenticity over the open network. Therefore enable HTTPS at least on the nodes through which discovery takes place. Enabling HTTPS takes effect after a server restart. If nodes are reachable only over public addresses and none is running HTTPS, direct auto-discovered links are not authenticated: the node is still visible (through relay from a neighbour), but its direct link is flagged with a reason (see Network map and resource catalogue).
Local network — HTTPS not required. If domain nodes communicate over private addresses of a single trusted network (10.x.x.x, 172.16–31.x.x, 192.168.x.x, as well as 127.x.x.x and 169.254.x.x), a secure connection for auto-discovery is not needed — nodes confirm authenticity over plain HTTP as well. A node is recognised as local by its IP address, so on a local network specify neighbours by IP addresses rather than by domain names. HTTPS can still be enabled — it will be used if available. The exception is a node that has an announce interface with a private address set. It announces itself for plain HTTP only. If a node has several networks (for example, an internal one and a public one that carries the default route), without a separate setting it reports its public address to neighbours. Set an announce interface for such a node (Node with several networks).
An explicit link works over HTTP as well — the HTTPS requirement does not apply to it in any case.
Name and secret are mandatory. A node participates in the network only if it has a name and a signing secret set.
Synchronised clocks. Since version 2.0.2.416 a node refuses a neighbour whose clock differs from its own by more than 5 minutes, and such a link is shown as unavailable. Keep time synchronisation (NTP) running on all nodes. When the difference exceeds half the tolerance, both nodes write a warning to the log with the neighbour’s name and the size of the difference. The tolerance is set by the cluster-hello-window-sec parameter of the cluster section of the configuration file (The pss.json settings file): 300 seconds by default, from 0 to 86400; 0 turns the clock check off if the clocks cannot be synchronised.
Name restrictions. A node name is 2 to 16 characters, a domain name is no longer than 16 characters; Latin letters, digits and the characters _, -, . are allowed. Spaces and other characters are not permitted (for example, the domain “Alex Home” cannot be set — use Alex_Home).

Node parameters¶

In the server settings you define how the node presents itself on the network:

Node name — a short unique name by which the others recognise the node. An empty name means the node does not participate in the network.
Note — an arbitrary label for the node (for example, “Moscow, rack 3”), shown in the admin panel; it has no effect on the network operation.
Domain — nodes with the same domain automatically find each other, share the common resource catalogue and replicate alerts. An empty value disables auto-discovery (the node can operate only via the explicit list of neighbours).
Public ports (HTTP and HTTPS) — the ports the node reports to neighbours for reverse reachability under address translation (NAT). If not set, the node reports its own web server ports.
Announce interface — the network interface whose address the node reports to neighbours instead of the web server host name or its public address. It is needed by a node with several networks (Node with several networks). A private address is announced for plain HTTP only; the public ports do not apply to it. The value “Default (HTTP host name or public address)” means the node reports the default address.
Additional domains (super-node mode) — the node is also a member of the listed domains. Resources remain isolated per domain — the node does not mix them. In its main domain the node is a full participant in auto-discovery. In an additional domain it is linked only to the neighbours from its list (Neighbours and secrets): from them it receives the resource catalogue and alerts of that domain, while it sees the other nodes of the domain through them and does not contact them itself. Therefore, for each additional domain, specify at least one neighbour from that domain on the super-node.

Neighbours and secrets¶

Here you define which neighbours the node maintains a direct link with and how the parties verify each other. Requests between nodes are signed with a shared secret, so without matching secrets no link is established. The length of each secret is 16 to 128 characters.

The neighbour list consists of entries of the form “address — name — secret”:

Address of a neighbour — the “host:port” of its admin panel. An entry may have no address: such a neighbour is never called by this node (it only dials in itself) and is identified by name and secret. This is how a node behind NAT that can only initiate a connection is set up.
Name of a neighbour must exactly match the name set on the neighbour itself.
Shared secret of the link with this neighbour must be the same at both ends.
Announce interface (optional) — the interface whose address the node reports to this particular neighbour instead of its general announce interface (Node with several networks). The value “Same as node” means the general one applies.

A neighbour’s announce interface is tied to the neighbour’s name, not to the row of the list:

while the row has no valid neighbour name, the interface selection is unavailable — there is nothing to save the setting under;
when the neighbour is renamed, the setting moves with it;
when the neighbour is removed from the list, the setting is kept and remains in effect if that node stays in the network via auto-discovery.

Such settings for nodes that are not in the neighbour list are shown in the “Announce interface for other nodes” block below the list: there they can be changed or deleted. A new node cannot be added to this block — the setting is made in the neighbour’s row.

The entry of an unreachable neighbour can be deleted not only here: the “Forget node” action on the “Nodes” and “Topology” screens (Nodes) strikes it out of this list too. This is the same configuration edit, made from a monitoring screen, and it cannot be undone.

In addition to the neighbour list, a node has its own signing secret: with it the node signs its outgoing requests. A neighbour verifies the signature with the secret it stores for that node, so the node’s own secret must match the secret recorded about it on each neighbour.

Responses are signed the same way: a node accepts a neighbour’s response only if it is signed with the key the request was signed with — the link secret or the domain key. A response with a wrong signature or from another node does not establish the link. The domain key is passed between nodes encrypted. Nodes of versions before 2.0.2.415 do not sign responses; how to proceed in a mixed domain is described in Upgrading the domain’s nodes. Since version 2.0.2.416 every node message carries its send time and the addressee’s name: a neighbour refuses a message sent again or addressed to another node, as well as a message with a time outside the tolerance (see “Synchronised clocks” in Requirements and reachability).

Note

A practical rule: for a simple site, set one shared secret and use it both as the node’s own signing secret on each node and as the secret in each neighbour entry. Then any pair of nodes authenticates the same way. For stricter separation, you can keep a separate secret per link.

Quick start¶

Option A. Two nodes directly (explicit link)¶

The simplest Meshwork is two servers connected manually.

On each node, set a unique short name (for example, node1 and node2).
On each node, add an entry for the neighbour: the address of its admin panel, its name and the shared secret.
On each node, set its own signing secret. The simplest approach is to keep a single shared secret for both nodes.
Open the “Nodes” screen of the Meshwork area in the admin UI. Within a few seconds both nodes will see each other with the “Alive” status.

A domain need not be set for this, but without a common domain you will get only mutual visibility of the nodes. The shared resource catalogue and alert replication work only between nodes with the same domain. If you need them, set a common domain for the nodes (option B).

Option B. A domain with auto-discovery¶

When there are more than two nodes, it is more convenient to give them a common domain rather than list each link manually.

On each node, set a unique name and the common site domain.
If domain nodes communicate over the internet (via public addresses), enable HTTPS on at least one or two domain nodes and restart them (see Requirements and reachability). On a local network this step can be skipped; for nodes that have no web server host name set, set the announce interface to their internal interface (Node with several networks).
Link the nodes via an “entry point”: on each node, specify at least one already working neighbour (address, name, shared secret), as in option A. The node will find the rest of the domain nodes on its own.
Open the network map. Within a few seconds all domain nodes will appear on it: the explicitly specified ones as permanent, the automatically found ones as auto-discovered.

Nodes behind NAT¶

A node on a private network that cannot be reached directly from outside can take part in the network as an initiator. It establishes a connection with a reachable neighbour itself, and over this single connection data flows in both directions — the node both reports about itself and receives information about others. For such a node it is enough to specify at least one reachable neighbour.

Information about a NAT node spreads across the domain automatically: a neighbour that is in contact with it relays its state to the others. On distant neighbours such a node will be shown as received through relay, but in the “online” state.

To make a node behind NAT reachable for inbound connections as well (for example, to serve as a domain entry point), configure publishing of the public port (see Node parameters) and forward this port on the router/firewall to the node’s administrative web server. On the neighbours’ side, such a node is kept as an entry “name + secret without address”: the neighbour will not vainly try to dial the private address but will identify the node by name and secret when it connects on its own.

Node with several networks¶

By default, a node reports to its neighbours the address at which it should be contacted: the host name set for the web server or, if none is set, the address of its first public interface. For a node with a single network this is enough. But if a node has several networks, the default choice may turn out to be wrong. A typical case: all domain nodes are on an internal network, but one of them also has a public interface that carries the default route. Such a node reports its public address to neighbours, and the neighbours start contacting it over the public network and via HTTPS, even though all nodes are on the same internal network.

To make a node announce the right address, set its announce interface (Node parameters). This is the network interface whose address the node reports to neighbours instead of the host name and the public address:

if the interface address is private (10.x.x.x, 172.16–31.x.x, 192.168.x.x, 127.x.x.x, 169.254.x.x), the node announces it for plain HTTP only — on the web server port, without HTTPS and without the public ports. Neighbours on the internal network contact it over HTTP; HTTPS is not required for the exchange;
if the address is public, it is announced the same way as the node’s public address: with HTTPS when HTTPS is enabled and taking the public ports into account.

The announce interface determines only which address the node reports about itself. Which interface the node’s own connections to neighbours go out through is still decided by the server’s routing table.

The procedure for setting up an internal domain in which one node also has a public network:

On the node with several networks, set the announce interface to the internal network interface.
On the domain nodes that have no web server host name set, also set the announce interface to their internal interface. Without it, a node that has only private addresses reports nothing about itself, and auto-discovery does not find it: it is linked only to those that have specified it explicitly.
Specify the nodes as each other’s neighbours by their internal IP addresses.
Check the network map (Network map and resource catalogue): the node addresses are internal, the NAT indication is not set on the node with two networks, and the links show no failed-authentication reason.

If a node must report different addresses to different neighbours (for example, the internal address to neighbours on the internal network and the public address to a neighbour on the internet), set the announce interface for individual neighbours (Neighbours and secrets): it applies instead of the node’s general announce interface. Note: if the same node announces itself differently in different networks, a third node that receives information about it from both networks may see its address alternate between one and the other. Therefore, different addresses are set for a node on the boundary between networks whose neighbours in the different networks do not exchange information about each other.

The result is visible on the “Nodes” screen (Nodes) in the row of the node itself: the “Public HTTP” and “Public HTTPS” fields show what the node reports to its neighbours. With a private announce interface there is no “Public HTTPS” field, and “Public HTTP” contains the internal address and port. An interface set for an individual neighbour is not reflected in the node’s own row — it is visible in this node’s row on the neighbour for which it is intended.

In the admin panel, the announce interface is selected from the list of the server’s network interfaces — name and IPv4 address; in the configuration file it can also be specified by IP address. An interface that does not exist on the server is rejected on save with a message like cluster-bind-if: 'eth9' is neither a network interface of this host nor an IPv4 address. If the interface disappears later (for example, a VPN goes down), the setting is kept: the node continues to operate and reports the default address until the interface returns, and in the selection list the saved interface is shown by its name alone, without an address.

Upgrading the domain’s nodes¶

Since version 2.0.2.415, nodes verify the signature on responses from neighbours and pass the domain key encrypted (Neighbours and secrets). Nodes of earlier versions do not sign responses, so the upgrade order matters:

if possible, upgrade all nodes of the domain in one go;
if that is not possible, first upgrade the nodes the others connect to (central nodes and nodes with public addresses), and then the nodes behind NAT.

While nodes of different versions run in the domain:

an old and a new node keep seeing each other as running;
the old node’s data (auto sign-in sessions, the shared resource catalogue, alerts) reaches the new node when the old one connects to it itself — usually within 30 seconds;
an old node that cannot connect to the new one (for example, because of NAT) is shown only as running, without its data, until it is upgraded;
full protection against replayed recorded exchanges applies once all nodes of the domain run version 2.0.2.415 or later.

Rolling back to the previous version is safe.

Since version 2.0.2.416 nodes also check the send time of messages, so:

before the upgrade, check that time synchronisation (NTP) is running on all nodes of the domain (Requirements and reachability);
upgrade the whole domain: while it still has nodes of earlier versions, their messages can still be replayed;
a node rolled back from 2.0.2.416 to an earlier version is refused by its neighbours on 2.0.2.416 until they restart. Roll back the whole domain, or restart the 2.0.2.416 nodes after the rollback.

Common problems¶

Nodes do not see each other. Check: both have a node name set; each has a signing secret and it matches the one recorded about it on the neighbour; the neighbour’s address is correct and its admin port is open to the other node.
A neighbour is in the “Dead” state. The node has exceeded the waiting window: check the network and the operability of its web server. For a node behind NAT, make sure it initiates the connection to a reachable neighbour itself, or that port publishing and forwarding are configured. If the log has a clock difference warning for this neighbour, synchronise the time on both nodes (NTP).
A neighbour became unavailable after one node was rolled back. Neighbours on 2.0.2.416 refuse a node rolled back to an earlier version until they restart: restart them or roll back the whole domain.
An auto-discovered neighbour is flagged with a reason, the direct link does not work. The domain cannot confirm the authenticity of the nodes. If they communicate over the internet, enable HTTPS on at least one domain node and restart it; on a local network, check that neighbours are specified by IP addresses rather than domain names, and that the node with several networks has an announce interface set. An explicit link (permanent nodes) works without HTTPS too.
Neighbours contact a node by its public address, although the domain is on an internal network. The node has several networks and by default reports its public address to neighbours. Set its announce interface to the internal interface (Node with several networks).
False NAT indication. If nodes are specified by domain names rather than by IP, the NAT indication may not be determined from the address — this does not interfere with operation. For NAT diagnostics, use IP addresses.
Changing the domain “reset” the map. Changing the domain or the set of additional domains rebuilds membership: auto-discovered links and the shared catalogue are temporarily reset and rebuilt. This is normal.