Optimizing Frigate NVR storage with mergerfs

Introduction

I use Frigate for real-time object detection with a Google Coral Edge TPU and as my Network Video Recorder (NVR). However, storing just a week's worth of footage from my cameras requires about 2TB of space on my NAS. While a NAS offers an economical storage option, accessing footage stored on HDDs over a network introduces latency. This lag noticeably impacts the performance of Frigate's UI, especially when playing back recent clips. The ideal setup would store recent footage on an SSD for quick access, while using a HDD NAS for older recordings. However, Frigate's limitation of supporting only a single mount point means this has to be solved outside of Frigate.

The solution: mergerfs UnionFS

In this guide, I'll walk you through setting up mergerfs a powerful tool that allows you to transparently overlay two file systems as one. This setup, combined with a cron job to manage file aging, provides the perfect balance of speed and capacity.

Prerequisites:

Before we begin, ensure you have:

• A 'fast' file system (e.g., an SSD) mounted
• A 'slow' file system (e.g., an HDD NAS) mounted

In my setup, I use:

• SSD mount: /mnt/frigate-ssd
• NAS mount: /mnt/frigate-nas

Step 1: Setting up mergerfs

First, we'll install unionfs-fuse. There are other alternatives like mergerfs that I wasn't able to get it up and running, but others have reported success. Avoid using overlayfs as it doesn't support writing to both underlying file systems which makes it suboptimal for our use cases where we want Frigate to age out old files that are already on the NAS.

sudo apt-get update
sudo apt-get install mergerfs

Then, we make our new mount that frigate will be pointed to. Under the hood it will consist of two file systems, but Frigate won't know that. Mine new mount will be at /media/frigate.

sudo mkdir -p /media/frigate

Now we create the new filesystem. Make sure to update the paths if yours are different than mine.

sudo mergerfs -o allow_other,use_ino,cache.files=partial,dropcacheonclose=true,category.create=ff,moveonenospc=true /mnt/frigate-ssd:/mnt/frigate-nas /media/frigate

Here's what the various parameters mean:

• allow_other: Allows access to other users
• use_ino: Tries to use the underlying inode numbers for files
• create=ff: First write to the first mount (the SSD)
• moveonenospc: Write to NAS if and only if SSD doesn't have enough space.

Step 2: Passing the mount through to the unprivileged LXC

Edit the config for the LXC on the Proxmox host:

nano /etc/pve/lxc/127.conf

Add the mount, and save

mp4: /media/frigate,mp=/media/frigate

You'll likely need to chown the mount so that it can be written to by the LXC.

chown -R 100000:100000 /mnt/frigate-ssd/

Step 3: Testing the Setup

Verify that:

1. The new mount shows files from both the SSD and NAS
2. You can create files on both the SSD and NAS
3. Deleting files from /media/frigate removes them from their original locations

If all of those things work, you should be good to go.

Step 4: Aging out files from the SSD to the NAS

Using the above setup, Frigate will always write to the first mount in the command, the SSD at /mnt/frigate-ssd. However, since Frigate is completely unaware that there are actually two file systems, we will need to move older files so that the SSD does not fill up and run out of room for new footage, and so that older footage remains archived on the NAS.

I decided to store 24 hours of footage on the SSD, and move older files to the NAS. You'll want to make sure your SSD big enough to store that much footage.

Now, we'll setup a script to move the files, and a cron job to run the script every hour. This means that every hour, any footage that is older than 24 hours is moved. This means that every hour, the oldest hour will be moved from the SSD to the NAS.

Create the script:

sudo nano ~/frigate-archiver/frigate-archiver.sh

Let's write our script, it will find files that are more than 1440 minutes old and move them from the SSD to the NAS. Then it will delete any empty directories.

#!/bin/bash
# Source and destination directories
SRC_DIR="/mnt/frigate-ssd/recordings"
DEST_DIR="/mnt/frigate-nas/recordings"

# Find and move files older than 24 hours
find "$SRC_DIR" -type f -mmin +1440 -print0 | while IFS= read -r -d '' file; do
    rel_path="${file#$SRC_DIR/}"
    dest_file="$DEST_DIR/$rel_path"
    dest_dir="$(dirname "$dest_file")"
    mkdir -p "$dest_dir"
    mv "$file" "$dest_file"
done

# Remove empty directories in the source
find "$SRC_DIR" -type d -empty -delete

Make the script executable:

sudo chmod +x ~/frigate-archiver/frigate-archiver.sh

Set up a cron job to run the script hourly:

0 * * * * ~/frigate-archiver/frigate-archiver.sh

Wrap up

By implementing this mergerfs setup, you can significantly enhance your Frigate NVR experience. You'll enjoy quick access to recent footage while maintaining a cost-effective long-term storage solution. Remember to monitor your setup after 24 hours to ensure everything is working as expected.