Operating Proofroom client stacks

This guide is for the operator: whoever runs the client’s server. That is the client’s own IT, or a reseller or we on the client’s behalf if the client chooses. Proofroom runs on the client’s own server. Each client gets a stack: its own containers, database, files, keys, domain and licence. proofroomctl creates, updates, backs up, restores and removes stacks; you never need docker compose by hand.

AI assistants (MCP): see AI assistants (MCP).

1. What a stack is

Directory /srv/proofroom/stacks/<name>/
Files compose.yml (from the release, replaced by update; do not edit), .env (image, port, memory, database passwords), app.env (the app’s settings, including MASTER_KEY), licence/licence.json, backups/, db-init/
Containers compose project proofroom-<name>: app and db
Data Docker volumes proofroom-<name>_pgdata (database) and proofroom-<name>_evidence (evidence files, encrypted with MASTER_KEY)
Address the app listens on 127.0.0.1:<port> (from 41000), behind Caddy (or your own proxy) at the client’s domain
Memory app 1 GB, database 512 MB by default (--app-memory, --db-memory at create; resize changes them later)

A stack name is 2 to 31 lower-case letters, digits or dashes, starting with a letter. proofroomctl only ever touches proofroom-* containers, networks and volumes, /srv/proofroom, /etc/caddy/proofroom.d/ and the patch report’s script and units (section 10). Other applications on the same server are left alone. Every command logs what it changes. Exit codes: 0 done, 1 failed, 2 wrong usage.

See before you change: simulate. Put -s (or --simulate, or --dry-run as in install.sh) before any command: sudo proofroomctl -s update acme. It prints each change the command would make as a numbered step (files, systemd units, containers, the database, Caddy, calls to the licence server that change something), with passwords, keys and tokens shown as ****, and changes nothing. The checks that only read run for real (the system, free memory and disk, where the domain points, the licence file, versions, whether another command is running), so the plan is the one for this server, and the exit code is the one the command would end with: 0 it would succeed, 1 it would fail, after the same message.

Every command and option is described by sudo proofroomctl help <command>, by man proofroomctl (installed by init and each update) and in the command reference, all written from the same source. On a minimised Ubuntu image without man, proofroomctl help has the same text.

The commands in this guide are run from an ordinary admin account with sudo. Signed in as root on a server without sudo (Debian)? Leave out sudo.

2. The server

  • Ubuntu 24.04 or Debian 12, and an account that can use sudo (root also works).
  • Docker Engine with the compose plugin: curl -fsSL https://get.docker.com | sudo sh (or your distribution’s docker.io and docker-compose-v2 packages), plus curl and openssl.
  • Ports 80 and 443 free (Caddy takes them), unless you use your own reverse proxy.
  • Memory: 8 GB recommended (2 GB is the minimum, enough for one small stack). About 1.5 GB per stack of the default size, plus 1 GB for the system; init tells you how many fit, and resize gives a busy stack more (§5).
  • For each client, a DNS record for its domain pointing at the server, before you create its stack (Caddy gets the TLS certificate as soon as the site is added).

3. Getting proofroomctl

The release image is public, so no registry login is needed:

sudo docker run --rm --entrypoint cat ghcr.io/proofroom-team/proofroom:<version> /app/release/proofroomctl | sudo tee /usr/local/sbin/proofroomctl.new > /dev/null \
  && sudo chmod 755 /usr/local/sbin/proofroomctl.new && sudo mv /usr/local/sbin/proofroomctl.new /usr/local/sbin/proofroomctl
sudo proofroomctl help

The download goes to a new file first, so a failed download never leaves you without a working proofroomctl.

Each release carries its own proofroomctl. After a successful update, proofroomctl puts that release’s copy in place of itself (§6), so you get it once, here. (From a private registry, log in first: sudo docker login <registry host> -u <user> --password-stdin < <token file>.)

If your security team asks what is in the image, each release has a CycloneDX SBOM: SBOM and vulnerability scanning.

4. Preparing the server: init

sudo proofroomctl init

For a private registry: sudo proofroomctl init --registry R --registry-user U --registry-token-file F.

It checks the server (operating system, Docker, memory), logs in to the registry if given a token, installs Caddy from Caddy’s own package repository if it is not there, writes /srv/proofroom/ctl.env and installs the daily patch report (section 10) and updates by button (section 6). Run it again at any time; it changes only what is missing. sudo proofroomctl init --refresh installs or refreshes only the patch report and updates by button, on a server set up before they existed.

Option
--registry R the image repository (default ghcr.io/proofroom-team/proofroom)
--registry-user U, --registry-token-file F log in; - reads the token from standard input. The token never appears on a command line
--own-proxy do not install or touch Caddy. For each stack, proxy the client’s domain to http://127.0.0.1:<port> (printed by create) with a request body limit of at least 110 MB, and terminate TLS yourself
--licence-server URL where stacks check in (default https://licence.proofroom.com); "" for servers without internet access
--update-url URL where the server reads the newest release’s version every 4 hours (default https://proofroom.com/releases/latest.json); "" turns the check, the dashboard notice and Update now off on this server

If Caddy was already installed, init adds one line to /etc/caddy/Caddyfile (import /etc/caddy/proofroom.d/*.caddy) and keeps the rest; a copy from before is kept as Caddyfile.before-proofroom. Every Caddy change is checked before Caddy reloads, and undone if Caddy refuses it, so your other sites keep running.

5. A new client: create

You get a licence file from us (<id>.proofroom-licence.json). It names the client’s domain and holds the stack’s check-in secret: keep it like a password.

sudo proofroomctl create acme --domain compliance.acme.example --licence ~/lic_….proofroom-licence.json --admin-email it@acme.example

It checks the licence (signature and domain) with the release’s own image, generates the stack’s passwords, AUTH_SECRET and MASTER_KEY, picks a port, starts the stack, waits until it is healthy, creates the first admin and prints their one-time setup link (valid 24 hours; send it to them securely), and adds the Caddy site. A lost or expired link: sudo proofroomctl admin-link NAME --email E prints a new one (it also recovers an admin locked out of their authenticator). Add --name "Full Name" to set the person’s display name, on a new or an existing account; names can also be changed in Admin → Users, and each person changes their own on My profile. Options: --version V (default: proofroomctl’s own version), --app-memory 1g, --db-memory 512m, --master-key-from FILE (section 8).

A firm licence (one licence whose users are shared by an audit firm’s client sites) comes as <id>.proofroom-firm-licence.json, for every client site under the firm’s domain. create with it registers the new site with the licence server first: the domain must be one level below the firm’s (client7.auditfirm.com) or listed on the licence, and seats must remain. The site gets its own install secret; the firm token in the file stays with you, not on the stack. It needs Proofroom 1.10 or later and a server that can reach the licence server.

Settings such as single sign-on go in the stack’s app.env (each setting is described in the install guide); then sudo proofroomctl restart <name>. Email is set in Admin → Settings → Email, or in app.env, which wins when it sets MAIL_TRANSPORT. If you later remove the MAIL_* lines, any settings saved earlier on the page are used again; the page names them while app.env is in use.

More memory for a stack. A stack with many people, evidence or integrations may need more than the default:

sudo proofroomctl -s resize acme --app-memory 2g --db-memory 1g   # what it would change, and whether it fits
sudo proofroomctl resize acme --app-memory 2g --db-memory 1g

It sets the limits (and the app’s heap, three quarters of the app’s limit) and restarts the stack’s containers, which takes about a minute. It refuses, changing nothing, when the extra memory is more than the server has free (its MemAvailable, less 256 MB kept for the system), and puts the previous limits back if the stack does not come up healthy on the new ones. The app needs at least 512m and the database 256m.

Looking at stacks:

sudo proofroomctl list             # every stack: version, domain, port, health, licence state, last backup
sudo proofroomctl status acme      # one stack in detail, and the server's patch report (section 10)
sudo proofroomctl logs acme --since 1h

6. Updates

A stack updates in one of two ways, and both do the same thing: an admin presses Update now in the app, or you run sudo proofroomctl update on the server. Either way it can go straight to the newest release; there is no need to install the releases in between.

Before you update, take a backup (sudo proofroomctl backup acme) and copy it and the stack’s app.env off the server (§7). The update takes its own backup first and restores it if the update fails, but that copy is on the same server.

What an update does

  1. Checks there is room: it stops, changing nothing, only when the disk Docker keeps its images on is clearly short of the new image’s download (its size in the registry) and 512 MB. When removing old Proofroom images no stack uses would make the room, it removes them first; never the image the stack runs or the one before it. When the size can’t be read, it says so and goes on.
  2. Takes a verified backup.
  3. Switches the stack to the new image and the new release’s compose.yml.
  4. Waits up to five minutes for it to be healthy (database changes run at start; the release notes say which releases have them).
  5. If it doesn’t come up healthy, puts the previous image back and restores the backup it just took, so the stack is as it was. It stops with an error and the app’s last log lines.
  6. When it is healthy, installs the release’s own proofroomctl and its systemd units (the patch report, the update check and Update now), so changes to them arrive with the update. If that step fails, the update still stands and one line says to run sudo proofroomctl init --refresh.
  7. When it is healthy, removes the Proofroom images no stack on the server needs. Each stack keeps the image it runs and the one before it, for a rollback. Only Proofroom’s own images go (ghcr.io/proofroom-team/proofroom, the old name ghcr.io/hriday/proofroom and your registry, if you set one), one by one by tag or ID; never the database’s or anything else on the server, and never docker image prune. A failed update removes nothing.

proofroomctl never replaces itself with an older version, and never after a rollback. If it can’t replace its own file (for example, a read-only file or a link), it says so and prints the command to do it by hand:

sudo docker run --rm --entrypoint cat ghcr.io/proofroom-team/proofroom:<version> /app/release/proofroomctl | sudo tee /usr/local/sbin/proofroomctl.new > /dev/null \
  && sudo chmod 755 /usr/local/sbin/proofroomctl.new && sudo mv /usr/local/sbin/proofroomctl.new /usr/local/sbin/proofroomctl

Update now (Admin → Updates)

Admin → Updates shows the version the stack runs, the newest release with its release notes, when the server last checked, and Update now. Admins also see a notice on the dashboard when a newer version is out.

  • The check: every 4 hours (and a few minutes after each boot) root fetches one small file, https://proofroom.com/releases/latest.json, into /srv/proofroom/host/update.json. The request has no query string and sends nothing about the server or its stacks. To check now: sudo systemctl start proofroom-update-check.service.
  • Turning it off: Admin → Updates → Check for updates, per stack. The server makes no request when every stack on it has the check off. sudo proofroomctl init --update-url "" turns it off for the whole server.
  • Pressing Update now: the app writes one empty file in the stack’s own request folder (/srv/proofroom/stacks/<name>/update-request/update). Root removes it, fetches latest.json itself, checks the release is newer than the stack runs, and runs the same update as by command. The app never chooses what root runs.
  • Following it: the page shows each phase through the app’s restart, then the outcome: “Updated from X to Y” with the backup taken first, or that the update failed and the stack went back to the previous version with its data. The audit log records who asked and each outcome.
  • When there’s no button: the check is off, the last check failed or is over 24 hours old, there is no newer release, or an update is already waiting or running. Root also refuses a request made under a minute after the last one, or while another proofroomctl command holds the server’s lock. Nothing changes, and the page says why.
  • If an update is interrupted (a reboot, the 45-minute limit, an error), the status ends “interrupted” and says what changed. If it stopped while the new version was starting, it was not rolled back: the status names the image the stack is set to and the command to restore the backup taken just before (sudo proofroomctl restore <name> <backup>).
  • One stack, one folder: each stack’s request folder is mounted into that stack only, so one stack can never update another. Its status folder is read-only to the app.
  • Logs: journalctl -u proofroom-update@<name> and journalctl -u proofroom-update-check.

By command

sudo proofroomctl update acme --version 1.8.0  # a given release (the newest is in the release notes)
sudo proofroomctl update acme                  # the release this proofroomctl came with

Without --version it never moves a stack back to an older release than it runs. After an update by button the server’s proofroomctl is the new release’s, so that is usually the newest.

Old images and disk space

Before 1.14.3 an update kept every image it pulled, so a server updated often can fill its disk. To see what is there and free it:

sudo proofroomctl images              # every Proofroom image, its size and whether a stack keeps it
sudo proofroomctl -s images --clean   # what --clean would remove, changing nothing
sudo proofroomctl images --clean      # remove the ones no stack needs (each keeps the one it runs and the one before)

An image Docker will not remove (a stopped container still uses it) is left, with a note. If a stack names an image that is not on the server, nothing is removed until that is sorted out (the note says which). Updates do the same clean-up on their own from 1.14.3 on.

Coming from an older version?

Every stack checking in runs 1.7.1 or later, which updates as above. If you restore or run something older: The release notes have the steps under each release’s “Upgrading”.

7. Backups

  • Every night at 01:30 UTC each stack’s app takes a verified backup (database, evidence files and a manifest) into /srv/proofroom/stacks/<name>/backups/; the newest 14 are kept. sudo proofroomctl backup <name> takes one now; update always takes one first.

  • Copy them off the server. They are on the same disk as the stack. The simplest way: in each stack, Admin → Backups → Off-server copy (below). Or copy them yourself, for example nightly from another machine:

    rsync -a --rsync-path="sudo rsync" <you>@<server>:/srv/proofroom/stacks/ ~/proofroom-backups/   # backups/ plus each stack's .env and app.env
    sudo rclone copy /srv/proofroom/stacks/acme/backups remote:proofroom/acme  # or to object storage, from the server
    
  • Off-server copy (Admin → Backups). Give the stack an S3-compatible bucket you own (AWS S3, Cloudflare R2, Backblaze B2, Wasabi, MinIO and others): endpoint (https://, without the bucket), bucket, region (auto or us-east-1 where the service has none), access key, secret and prefix. After each nightly backup the app encrypts that backup’s files (AES-256-GCM, with a key derived from the stack’s MASTER_KEY) and uploads them, with the stack’s licence file, also encrypted, and copy.json last:

    <bucket>/<prefix>/<stack's domain>/<backup name>/db.dump.enc
                                                     evidence.tar.gz.enc
                                                     manifest.json.enc
                                                     licence.proofroom-licence.json.enc
                                                     copy.json   # written last: a copy without it is incomplete
    

    copy.json is not encrypted: it names the backup, the key’s fingerprint, and each encrypted file’s size and SHA-256, so a download can be checked before anything is decrypted. Where the service supports it, the app also asks it to encrypt at rest (SSE-S3, a setting on the page).

    The domain in the path lets several stacks share one bucket and prefix. The copy goes straight from the server to your bucket; Proofroom, the company, never receives it. The secret is encrypted with the stack’s master key and never shown again. Test connection writes proofroom-connection-test.txt under <prefix>/<domain>/ and removes it (a key that may only write passes; delete the file yourself). Upload the newest backup now and Back up now do not wait for the night. Each upload is in the audit log (backups.uploaded, backups.upload_failed), and a failure shows on the page and the dashboard in plain words until an upload succeeds. A failed upload never touches the backup on the server.

    Set the bucket up so a stolen key or a mistake cannot remove your backups:

    • a private bucket (the files are encrypted before they leave the server; encryption at rest is a second layer);
    • a key that may only write (s3:PutObject, plus s3:AbortMultipartUpload) to <prefix>/*;
    • versioning or object lock, so an overwrite or delete keeps the old copy;
    • a lifecycle rule for retention, for example delete objects under <prefix>/ after 30 days, and abort incomplete multipart uploads after 7 days (AbortIncompleteMultipartUpload), so an upload cut off half-way does not leave parts you pay for. Proofroom never deletes backups in the bucket; BACKUP_KEEP applies only to the server’s own folder.

    app.env (with MASTER_KEY) is not uploaded, and the copy cannot be decrypted without it: keep it off the server separately (next point). Restoring from the copy: section 8, “Restore from the off-server copy”.

  • The key. Evidence in a backup is encrypted with the stack’s MASTER_KEY, in app.env. Without it a backup’s files cannot be read. Back up each stack’s app.env separately and securely (a password manager or an encrypted store), not only next to the backups.

  • Test a restore now and then (section 8, --into), then remove the test stack.

8. Restoring and moving a client

Restore a stack in place (asks you to type the stack’s name; --yes skips that):

sudo proofroomctl restore acme 20261021T013000Z          # a backup in the stack's backups/, or a path to a backup directory

It first takes a safety backup of the current data (if that fails, nothing is changed), then stops the app, replaces the database and the evidence with the backup’s, and starts the app. Safety and pre-update backups never delete older ones, so restoring the oldest backup is safe; the nightly job keeps the newest 14.

Start a new stack from a backup, on the same server (to test a restore, or to move a client to a new domain):

sudo proofroomctl restore acme 20261021T013000Z --into acme-test --domain test.acme.example --licence ~/<licence file for that domain>

The new stack gets the source stack’s MASTER_KEY and version. With a firm licence file, the new site is registered on the firm licence. On a site with a firm licence, restoring (in place or --into) or importing data from another site on the same firm licence counts as one of the firm’s 3 releases, reactivations and imports a year, counted by the licence server (so it needs the server, and is refused, with nothing changed, when none are left); so does a backup taken before 1.10, which does not say which site it came from. Restoring a site’s own backup, or importing a client’s export from their own Proofroom, is free. A counted restore that fails gives its count back.

To move a client to another server: on the new server, create the stack with --master-key-from <copy of the old stack's app.env>, copy a backup directory over, then sudo proofroomctl restore <name> /path/to/<backup directory>. Then remove the old stack. A site on a firm licence keeps its registration when it moves: give create a licence file made from the old stack, { "licence": <its licence/licence.json>, "installSecret": "<LICENCE_INSTALL_SECRET from its app.env>", "licenceServer": "https://licence.proofroom.com" }, not the firm file (which would register a new site, and the domain is still taken by the old one).

Restore from the off-server copy

When the server is lost, or to check that the copy in the bucket works, restore from it on a server prepared with init (a new one, or the same). You need the stack’s app.env that you kept off the server (it holds MASTER_KEY), and its domain’s DNS record pointing at the server. The licence is in the copy, encrypted with the backups; there is no need to ask for a new one.

  1. Download the backup. Each backup is a folder <prefix>/<domain>/<backup name>/ in the bucket; take the newest whose copy.json is there (a folder without it was cut off). With the AWS CLI, or rclone, for any S3-compatible service:

    aws s3 ls s3://acme-backups/proofroom/compliance.acme.example/ --endpoint-url https://<endpoint>
    aws s3 cp --recursive s3://acme-backups/proofroom/compliance.acme.example/20261029T013000Z/ /root/copy/20261029T013000Z/ --endpoint-url https://<endpoint>
    # or: rclone copy remote:acme-backups/proofroom/compliance.acme.example/20261029T013000Z /root/copy/20261029T013000Z
    
  2. Check the download against copy.json: each file’s size and SHA-256 (the next step checks them again, and the key’s fingerprint, before it decrypts anything):

    cd /root/copy/20261029T013000Z
    python3 -c 'import json; [print(v["sha256"] + "  " + k) for k, v in json.load(open("copy.json"))["files"].items()]' | sha256sum -c
    
  3. Decrypt it with the stack’s app.env. It writes /srv/proofroom/restored/<backup name>/: a backup that restore takes, with the licence file beside it. Nothing is written unless the whole copy decrypts; a copy made with another MASTER_KEY is refused, naming both keys’ fingerprints.

    sudo proofroomctl backup-decrypt /root/copy/20261029T013000Z --master-key-from /root/acme-app.env
    
  4. Create the stack (on a new server) with the same key and the licence from the copy, then restore:

    sudo proofroomctl create acme --domain compliance.acme.example --licence /srv/proofroom/restored/20261029T013000Z/licence.proofroom-licence.json --master-key-from /root/acme-app.env
    sudo proofroomctl restore acme /srv/proofroom/restored/20261029T013000Z --yes
    

    On the server the stack already runs on, skip create: sudo proofroomctl restore acme /root/copy/20261029T013000Z decrypts the copy with the stack’s own key and restores it, in one step. Put -s before either command to see what it would do first.

  5. Sign in and check the data. Admin → Backups comes back with the backup (its secret is sealed with the same key): press Upload the newest backup now to check the copy works from here. Remove /srv/proofroom/restored/<backup name> and /root/copy when you are done.

If the old server’s site is still registered on a firm licence, the licence file in the copy keeps that registration: release the old site only after the new one runs.

Moving an audit to an audit firm’s Proofroom

An audit firm may work on a client’s audit in its own Proofroom. The client’s admin makes a password-protected export (Admin → Export) and sends the file and the password separately. On the firm’s server:

sudo proofroomctl create acme-2026 --domain acme-2026.audits.firm.example --licence ~/<licence file>   # no --admin-email: the site stays empty
sudo -i   # a root shell for the import (the stack's folder is root's); exit when done
cd /srv/proofroom/stacks/acme-2026
cp /home/<you>/acme-export.zip /home/<you>/pw.txt backups/ && chown 1000:1000 backups/acme-export.zip backups/pw.txt && chmod 0600 backups/pw.txt
docker compose -p proofroom-acme-2026 stop app
docker compose -p proofroom-acme-2026 run --rm app node_modules/.bin/tsx scripts/import.ts --archive /backups/acme-export.zip --password-file /backups/pw.txt
docker compose -p proofroom-acme-2026 up -d app
rm backups/acme-export.zip backups/pw.txt

The import prints setup links for the imported admins; invite everyone else again from Admin → Users. Integrations come back switched off and without secrets: leave them off, so the copy never calls the client’s systems. Passwords, authenticators, sessions, API keys, integration secrets, the email settings and the TypeSafe API key don’t travel; every record, evidence file and the audit log do. Work done in the firm’s copy doesn’t flow back to the client’s. On a firm licence, importing a client’s export from their own Proofroom is free; only an import of a site that was on your own firm licence counts toward the firm’s limit of three releases, reactivations or imports a year.

9. Licences

Each stack checks in with the licence server once a day; renewals, extra users and grace extensions arrive that way, with nothing to do on the server.

sudo proofroomctl licence acme                   # the stack's licence state, licensed and active users
sudo proofroomctl licence acme ~/<file>         # install a licence file by hand (servers without internet access)
sudo proofroomctl usage                          # licensed and active users for every stack; --csv for a spreadsheet

Admins buy, add users and renew on the stack’s Admin → Licence, paid through our payment provider; the stack picks up the paid licence within seconds of payment, and nothing is needed on the server. A stack without access to the licence server cannot buy itself: the billing contact gets the licence file by email after any payment, to install with sudo proofroomctl licence <stack> <file>.

On a site whose licence a vendor (reseller) manages, Admin → Licence names the vendor instead and offers no buying: the vendor changes the site’s users and renews from its own vendor area, and the change arrives with the daily check-in, or at once when an admin presses Check now. Up to 5 of the vendor’s own staff accounts are free on the site, so the vendor can help with set-up; they are marked Vendor on Admin → Users. A vendor hosting its clients runs create with each client’s install code, one stack per client.

proofroomctl create (and licence) writes licence/profile.json beside licence.json when the licence file came from a trial with profile answers. It holds hints for the company profile only; deleting it changes nothing about the licence.

Up to 3 auditor accounts per open audit are not counted; an auditor on no open audit counts (activeUsers is the counted number; totalUsers and freeAuditors are in sudo proofroomctl licence <name>). On a firm licence, licence also shows firm=true, the seats used across the firm’s sites (poolUsed) and whether the site was released. To free a firm site’s seats, an admin presses Release licence on its Admin → Licence: the site becomes read-only (sign-in, viewing and exports still work).

10. Patch report

Every stack’s server needs its security updates installed, and an auditor asks for proof (PCI DSS 6.3.3: critical patches within a month of release). init installs a daily patch report that records it. It only reads: Proofroom never installs, upgrades or reboots anything on the server, and never changes its settings. It shows each problem with the command that fixes it, for you to run.

  • What it records: the operating system and kernels, pending updates (and which are security updates), every security update from the day it was published to the day it was installed, whether automatic updates are on, and whether a reboot is pending. It is written to /srv/proofroom/host/status.json, with 90 days of history in /srv/proofroom/host/history/. Nothing is sent to us: each stack’s app reads the folder read-only at /host.
  • When: the proofroom-host-report.timer runs /usr/local/lib/proofroom/host-report.sh once a day, at a time between 01:00 and 05:00 that stays the same on each server (a run missed while the server was off happens at the next boot). Run it now: sudo systemctl start proofroom-host-report; its log: systemctl status proofroom-host-report.
  • Check now: an admin can also run it from Admin → Server health. The button writes one empty file, /srv/proofroom/host-request/request, and nothing else; proofroom-host-report.path sees it and starts the same service, which removes the file and writes a new report, usually within a minute. It runs at most once every 5 minutes: a press sooner says when the last report ran. The folder is the only one the app can write to (mounted at /host-request; root’s, group 1000, mode 770). The script removes whatever is at that name without following it (a file, a link, an empty folder); only a folder with something in it is left, and the button stops working until you remove it by hand. If the path unit gave up, sudo proofroomctl init --refresh or update resets and restarts it. Without the folder (a server not yet updated), the page shows the command instead of the button.
  • After patching: press Check now on Server health, or run sudo systemctl start proofroom-host-report.service, so the page and the next evidence show the updates installed.
  • First report: when init or update installs or changes the timer, it also starts the first report in the background, so Server health has data within minutes. If that start fails, one line says so and the timer runs it that night.
  • Updates: update installs the release’s copy of the script when it changed, and the units (the service, the timer and the path unit) when missing or changed, as the new release’s proofroomctl writes them (section 6). A server set up before the patch report, or before Check now, gets them with sudo proofroomctl init --refresh (or on its next update). A problem with the patch report prints one line and never stops init or update.
  • Updating from 1.2.x: the first update runs the old proofroomctl, which does not know the patch report. Afterwards, run sudo proofroomctl init --refresh once (or update again) to install the daily report.
  • Looking at it: sudo proofroomctl status (or sudo proofroomctl status <name>, with the stack) prints a summary and each warning:
Warning Fix
Security updates open more than 30 days after publication sudo apt-get upgrade
Automatic updates are off sudo apt-get install unattended-upgrades && sudo dpkg-reconfigure -plow unattended-upgrades
A reboot needed for more than 7 days reboot at a quiet time (the stacks start again on their own), then sudo proofroomctl status
The report has not run for more than 48 hours systemctl status proofroom-host-report
“none yet” the first report is still running, or the timer was never installed: sudo proofroomctl init --refresh, then sudo systemctl start proofroom-host-report

11. Troubleshooting

Symptom Look at
create says the stack did not become healthy sudo proofroomctl logs <name>; the stack is left in place. Remove it with sudo proofroomctl remove <name> --confirm <name> --no-export and try again
“the licence was refused” the message says why: another domain (the licence names exactly one), or a changed file (download it again)
“Caddy did not accept the change” the Caddyfile or another site in it does not validate: caddy validate --config /etc/caddy/Caddyfile says why. (DNS is not checked then; a domain whose DNS is not ready yet gets its certificate once it is)
The site shows a certificate error DNS for the domain does not point at this server yet, or ports 80/443 are blocked; Caddy retries on its own: journalctl -u caddy
“another proofroomctl command is running” one changing command runs at a time; wait for the other to finish
A restore or rollback says its restore failed the data is as it was before (the database restore is all-or-nothing). Read the message above it: a “master key fingerprint mismatch” means the stack’s MASTER_KEY is not the one the backup was taken with (section 8)
A port is in use proofroomctl skips ports in use when it creates a stack; another program that later takes a stack’s port stops that stack starting
The app is read-only the licence expired or is not valid, or a firm site was released: sudo proofroomctl licence <name>, and contact us
“a firm licence needs the licence server” the server was prepared with init --licence-server ""; firm sites need to reach the licence server (init --licence-server https://licence.proofroom.com)
“Another site on the firm licence already uses this domain” release that site first (its Admin → Licence), or ask us if its server is gone
Admin → Updates says “not set up on this server yet” the stack has no update folders, or root has no units: sudo proofroomctl init --refresh
Update now waits and nothing happens systemctl status proofroom-update@<name>.path; if it gave up, sudo proofroomctl init --refresh resets it

12. Removing a client

sudo proofroomctl remove acme --confirm acme

It takes a final full export of the client’s data (final-export.zip) and copies the stack’s .env, app.env and backups into /srv/proofroom/removed/acme-<time>/, then deletes the stack’s containers, volumes and Caddy site. Give the client their export; delete that directory when you no longer need it. Without a working app the export cannot be taken and remove stops; --no-export removes the stack anyway. A site on a firm licence keeps counting toward the firm’s seats until it is released: press Release licence on its Admin → Licence before removing it.