EPG/XMLTV import

The EPG database is populated from three kinds of sources:

  • EIT from received streams — the “Extract EIT to EPG database” setting of the stream (SPTS and MPTS, Stream modification);

  • EIT from DVB adapters — the adapter setting “Export EPG” (DVB receiver);

  • external XMLTV sources — by a link to a web resource or from a local file.

EIT extracted from streams and from DVB adapters accumulates in a single built-in source, “stream-import”; every external XMLTV source maintains its own database. A channel identifier is unique within its own source; channel names and event texts are stored in several languages.

Source configuration

Sources are configured on the “EPG Settings” screen (EPG Settings). The “Import” tab holds the general parameters: “Enable import”, “Keep history, days” (retention depth for past events) and “Auto-clean orphan channels” (removal of channels that have no events left).

External XMLTV sources are added on the “Sources” tab: name, “Source URL / path”, “Enabled”, “Update interval, s” (once an hour by default), “Content encoding” (auto-detect or force GZip), credentials and Cookies for restricted web sources, “Save downloaded copy” for diagnostics. The source is an http or https address or an absolute path to a file on the node; Cookies are sent in the Cookie header and only to the source’s host. After an enabled source is changed (for example, its address), the data is loaded at once, without waiting for the update interval.

Since version 2.0.2.418, a source has a fetch timeout — the source’s fetch-timeout parameter in the settings file or via the API (from 0 to 3600 seconds, 0 means the former 30 seconds); since version 2.0.2.419 it is set by the “Fetch timeout, s” field (EPG Settings). It limits the wait for every send and every receive of data: the first byte of the response and any pause in the transfer. It is meant for slow sources that build the file on request. The connection is established within 30 seconds at most, and the whole transfer lasts no longer than 30 minutes (or two timeouts, if that is longer). An expired timeout is a failed fetch: it is written to the log and shown in the source error.

Fetch queue and retries

Since version 2.0.2.418, external sources are fetched, parsed and applied one at a time: while one is in progress, the others wait their turn, and the “Sources state” tab shows such a source as waiting, with its position in the queue (Queued: position N); since version 2.0.2.419 it is also marked with a blue dot. Repeated “Update now” presses for a waiting source merge into one fetch. After the service starts, sources begin fetching not all at once but in turn, at an interval: the first after 10 seconds, each next one a second later. So the memory peak when a node with several large feeds restarts stays at the level of one feed, and after the fetch the memory is returned to the system.

A failed fetch is retried not after the full update interval but after 2, 5 and 15 minutes, then every 30 minutes (but no less often than the update interval); the time of the next attempt is shown in the “Next update” field. Since version 2.0.2.419 this field is called “Next retry” while fetches fail, and the “Failed fetches in a row” field is shown next to it (EPG Database). The failure count is reset by a successful fetch, an edit of the source and a service restart. An empty response from a source also counts as an error as of this version.

EPG Database

The accumulated programme schedule is browsed on the “EPG Database” screen (EPG Database). The “Database” tab selects the source and provides channel search and a channel-set filter; the channel schedule is paged by day, and the current programme is highlighted. For each channel the following can be changed:

  • “Channel name” — the name used for XMLTV export;

  • “Time zone” — if event times were not bound to UTC on import;

  • “Icon URL” — the channel icon;

  • “Channel sets” — membership in the sets served to external consumers (EPG for OTT middleware).

Manual overrides may be overwritten on the next source refresh — the web interface warns about this.

The “Sources state” tab shows import progress: the last and next update time, the number of channels and events, errors; the “Update now” button starts a source refresh immediately, without waiting for the schedule.

Since version 2.0.2.420, programme changes in external sources are recognised more reliably: swapping text between two fields of a programme also counts as a change. A feed that has not changed since the last update is not parsed again, but its channels still count as updated, so “Auto-clean orphan channels” does not delete them; before version 2.0.2.420 such channels could disappear. The first update after moving to 2.0.2.420 rewrites nothing in the database, and channel name edits made in the web interface are kept.