DVR archive storage and SSD cache

An appendix for installations with a large archive: how the node reads the DVR archive, how to estimate the load on the disks and choose between HDD and SSD, how to put an SSD cache in front of an HDD array using OS tools, and how to tune the system. Setting up the storages themselves and cleanup are described in DVR storages, and monitoring them in DVR monitor.

All numbers on this page are rules of thumb and examples for estimation, not a performance promise: the real load depends on the audience, the set of channels and the hardware.

1. How the node reads the archive

The node writes the archive to ordinary files in the storage directory and reads them with ordinary file reads. So every archive read goes through the Linux page cache: repeated reads of the same segments are served from RAM, without touching the disk. RAM is the first and free level of cache; nothing needs to be configured on the node for this.

The following are served from memory with almost no disk access:

  • timeshift viewing and viewing from the start of a programme within about half an hour to an hour of the live edge: these segments have just been written and are still in the cache;

  • popular programmes and events watched by hundreds of viewers at once: the disk reads such a segment once, and the other viewers get it from memory;

  • fresh catch-up of yesterday’s prime time, which is watched repeatedly.

The long tail (different channels and days of the archive that each only a few people watch) does not fit in memory and is read from the disks. Linux keeps pages that have been read several times for longer, while one-off views are pushed out by the write stream within half an hour to an hour. So with an archive of tens of terabytes and a wide audience, memory takes off the disks, as a rule, from a quarter to a third of the reads: it speeds up the hottest data but does not replace disk performance.

2. Estimating the load and choosing disks

Write rate, volume and read rate

Three quantities are calculated as follows:

  • Write rate = number of channels being recorded × total bitrate of all variants of a channel. For example, 150 channels at 8 Mbit/s (three quality variants) make 1200 Mbit/s, that is 150 MB/s of continuous writing.

  • Archive volume = write rate × retention depth. 150 MB/s is about 13 TB per day, 7 days is about 91 TB, 14 days is about 182 TB of usable space, not counting RAID.

  • Read rate = concurrent archive viewers × average viewing bitrate. The number of archive viewers is the peak share of concurrent viewers multiplied by the share of those who watch the archive rather than the live broadcast. For example, 10,000 subscribers, 30 % in front of screens at the peak and 30 % of them in the archive make 900 viewers; at 4 Mbit/s on average that is 3.6 Gbit/s, about 450 MB/s of reading.

The write rate is added to the read rate from the disks: the disks serve both at the same time. Part of the reads is taken off by RAM (1. How the node reads the archive).

HDD and SSD

Each archive viewer is a separate sequential read stream, and hundreds of such streams together look to a disk like random access in large blocks.

  • HDD (NL-SAS or SATA 7200 rpm) with many concurrent streams delivers, as a rule, 60–100 MB/s per disk if the readahead is large (4. System tuning). The cost per terabyte is lowest for HDD.

  • NVMe SSD delivers gigabytes per second, and random access does not hinder it. A terabyte costs several times more. Write endurance is given in DWPD: how many full disk capacities per day can be written over the whole warranty period. For an archive it is calculated like this: daily write volume ÷ total disk capacity. 13 TB per day on eight 15.36 TB disks is about 0.1 DWPD, with a margin even for 1 DWPD disks.

An example for a 7-day archive (about 91 TB): RAID 6 of ten 20 TB HDDs gives 160 TB of usable space and on the order of 500–800 MB/s of total reading. That is enough for hundreds of concurrent archive viewers together with recording. When there are more than a thousand archive viewers, such an array works at its limit: you need more disks, an SSD cache (3. SSD cache with OS tools) or an SSD array.

Choosing RAID and general recommendations

  • RAID 6 is the main choice for large HDD arrays: recovery after replacing a 20 TB disk takes a long time, and RAID 6 survives the failure of a second disk during that time. RAID 5 is not recommended on disks of this size.

  • RAID 10 gives more performance on a mixed load, but only half of the capacity.

  • For SSD arrays, RAID 5, RAID 6 or RAID 10 are suitable: their recovery time is much shorter.

  • One storage per disk or array (DVR storages).

  • Guidelines for the choice:

    • up to about a thousand concurrent archive viewers: an HDD array (RAID 6 or RAID 10) and 256–512 GB of RAM for the page cache;

    • more viewers, or an archive of 30–90 days with active catch-up: HDD for capacity and an SSD cache for the last day or so, or distributing the channels across several nodes;

    • an SSD-only array: when rack space, power consumption or a very high load matter more than the cost per terabyte.

3. SSD cache with OS tools

An SSD cache does not get in the way of the node and requires nothing from it: the node reads and writes files, and the cache sits below the file system (block level) or inside it. The order by strength of effect: first RAM (the page cache), then SSD for the hot data of the next one to two days (usually 2–8 TB of NVMe), then HDD for capacity.

Writethrough and writeback

  • Writethrough is the recommended mode for an archive. Writes go straight to the HDD, and the SSD holds frequently read blocks. An SSD failure loses no data: the array keeps working without the cache.

  • Writeback keeps fresh writes on the SSD too, which speeds up timeshift and fresh catch-up. But data not yet moved to the HDD is lost together with the SSD, so in this mode the cache is placed only on a mirror of two SSDs (RAID 1), and it consumes the SSD write endurance.

  • dm-writecache caches only writes and does not speed up archive reads; it is not suitable for an archive.

lvmcache: RHEL, AlmaLinux, Rocky Linux, Oracle Linux

lvmcache (dm-cache) is the main option for RHEL-based systems, where it is supported out of the box. The cache is a logical volume on the SSD, attached to the archive volume in the same volume group. The default policy (smq) moves blocks that are read repeatedly to the SSD, so one-off views do not clutter the cache. The cache can be attached and detached on a running volume, without stopping recording.

In the example the archive is on the volume vg_dvr/dvr (an HDD array) and the SSD is /dev/nvme0n1. Add the SSD to the volume group and create the cache volume on it:

sudo pvcreate /dev/nvme0n1
sudo vgextend vg_dvr /dev/nvme0n1
sudo lvcreate -n dvr_cache -L 3T vg_dvr /dev/nvme0n1

Attach the cache in writethrough mode:

sudo lvconvert --type cache --cachevol dvr_cache --cachemode writethrough vg_dvr/dvr

Check the mode and the hit statistics:

sudo lvs -o name,size,cache_mode,cache_policy,cache_read_hits,cache_read_misses vg_dvr

The ratio of cache_read_hits to the sum of hits and misses is the share of reads served by the SSD. To detach the cache: --splitcache keeps the cache volume for reattaching, --uncache deletes it; the archive data stays on the HDD in both cases. Old LVM versions (for example in RHEL 7) have no --cachevol: there the cache volume is first turned into a pool (lvconvert --type cache-pool) and attached with --cachepool.

sudo lvconvert --splitcache vg_dvr/dvr

bcache: Debian and Ubuntu

bcache is part of the Linux kernel and is common on Debian and Ubuntu; it is not shipped in RHEL. It is created on empty devices: it cannot be enabled on an existing array holding an archive without moving the data. The file system is created on the /dev/bcache0 device itself.

In the example the HDD array is /dev/md0 and the SSD is /dev/nvme0n1. Create a device with a cache (in writethrough mode by default):

sudo apt install bcache-tools
sudo make-bcache -B /dev/md0 -C /dev/nvme0n1

The main setting for an archive is sequential_cutoff. By default bcache bypasses the cache for sequential reads and writes larger than 4 MB, and each archive viewer is exactly a sequential stream, so with the default setting the cache stores almost nothing. Turn the threshold off:

echo 0 | sudo tee /sys/block/bcache0/bcache/sequential_cutoff

This value is not kept after a reboot: set it at boot, for example with a udev rule or a systemd service. Check the mode and the hit ratio:

cat /sys/block/bcache0/bcache/cache_mode
cat /sys/block/bcache0/bcache/stats_day/cache_hit_ratio

ZFS: ARC and L2ARC

In ZFS (OpenZFS packages) the role of the page cache is played by its own in-memory cache, ARC, and an SSD is attached to the pool as a second cache level, L2ARC. L2ARC caches only reads; an SSD failure loses no data. Attach the SSD to the pool dvr:

sudo zpool add dvr cache /dev/nvme0n1

By default L2ARC does not accept data read by readahead, that is, the sequential streams of archive viewers. For an archive this must be turned off and also saved for subsequent boots:

echo 0 | sudo tee /sys/module/zfs/parameters/l2arc_noprefetch
echo "options zfs l2arc_noprefetch=0" | sudo tee -a /etc/modprobe.d/zfs.conf

A few more settings for an archive:

  • recordsize=1M for the dataset with the archive: the archive consists of large files that are read whole;

  • atime=off: no access time is written on every read;

  • zfs_arc_max: the limit of memory for ARC; leave some memory for the node too;

  • l2arc_write_max: the L2ARC fill rate; the default value is small, and with a large archive the cache fills slowly.

ARC and L2ARC statistics are shown by arc_summary, and the load on the pool devices by zpool iostat -v dvr 5.

4. System tuning

  • Readahead. For an HDD array set a large readahead, for example 4 MB (8192 sectors of 512 bytes); it is kept only until reboot, so set it at boot too (with a udev rule). For an LVM volume lvchange --readahead sets the same.

    sudo blockdev --setra 8192 /dev/md0
sudo blockdev --getra /dev/md0

  • Memory for the cache. The page cache takes all free memory, so leave the node a reserve: do not run other memory-hungry processes on it and do not let the system go into swap. How the node returns unused memory is described in Memory.

  • File system. XFS (the default on RHEL-based systems) and ext4 are suitable. Mount the storage with noatime so that reading the archive does not generate access time writes.

  • Monitoring. The load on the disks is shown by iostat -x 5 (the read and write columns in MB/s, the wait time, the device utilisation). The size of the page cache is shown by the buff/cache line in the output of free -h, SSD cache hits by the commands from the sections above. The read and write latencies of the archive per stream are shown by the “DVR monitor” screen (DVR monitor), and during long reads and writes an alert with code 50 (dvr-rw-time, List of alerts) is raised.