Backups

Every Joinery site can back itself up: on a schedule, encrypted, uploaded to an S3-compatible bucket, with retention enforced. No agent, no SSH, no management node. server_manager is a fleet layer on top of this, not a prerequisite for it — a site running server_manager backs itself up through exactly the same path as a standalone install, because no site's recovery may depend on another machine being alive.

Configured at Admin → System → Backups (/admin/admin_backups).

Two parties, two profiles

A site can be backed up by more than one party. It backs itself up, and a management node managing it may take its own copies. Those are not two ways of doing one thing — they are two parties' backups, on two schedules, onto two shelves, answerable to two people. They open with the same key, because that key belongs to the machine and its administrator, not to whoever asked for the run.

A profile (includes/BackupProfile.php) is the unit that keeps them apart:

The credential a run is handed

A manager run needs a credential to write with, and it is chosen at build time from three, strongest first:

• A key minted for that run. Where the target's provider can pin a key to one bucket, one name prefix, one capability and a lifetime — Backblaze B2 does, in one call — the node is handed a key created when the job is picked up, allowed only to add objects under its own prefix, expiring with the run. A key read off a machine somebody else administers then opens that machine's own directory for an hour, rather than the fleet's whole backup storage indefinitely. Off until an operator turns it on, per target, on the Remote Backup page — minting needs a master key the provider will let create keys, and a target switched on without one fails every run rather than falling back.
• The target's write-only node credential. A second stored key allowed to add objects and not to delete any. Shared across the fleet, which is what minting improves on.
• The target's main credential. Where neither of the above is configured.

Restores need no minted key at all: the management node signs one object key per artifact and the node receives the signature, so no bucket credential of any kind travels for a read.

Two things to settle before switching minting on for a real fleet, neither of which has been exercised yet:

• Key lifetime and cleanup. Nothing deletes a minted key early; each one is expected to expire on its own. Whether the provider removes expired keys, and what its per-account key ceiling is, decides whether a fleet of forty nodes backing up nightly accumulates keys faster than they lapse.
• Cost per pickup. Minting adds three provider round trips to the job hand-out request (authorize, look the bucket up, create the key), and the bucket is looked up afresh every run. Fine for a fleet of tens; worth caching the bucket id if job pickup ever starts timing out.

The reason is that sealing cannot fail visibly. Encrypting to a public key succeeds whether or not anybody holds the private half, so a substituted key produces archives that report themselves encrypted, upload normally and show green on every dashboard — while only whoever substituted the key can read them, and only a restore attempt would ever reveal it. A key that arrives over a wire is therefore a key nobody on the receiving machine can verify, whatever sent it. The cost is accepted deliberately: opening a machine's backups needs that machine's recovery key, and no single key opens a fleet.

Neither profile owns the site's backups. They are peers. A site admin who wants copies of their own as well as the management node's just sets their profile up; a management node keeps taking its own whatever the site does. Two backups a night of one machine is a supported configuration, not a misconfiguration to be detected.

The asymmetry in the last row is the safety argument: *the site profile depends on nothing. A management node that is down, retired or hostile costs a site nothing it was relying on.

Everything a run touches that could collide with another run is derived from the profile: the working directory (backups/ for the site, backups/manager/ for a management node's), and therefore the lock, the tar snapshot, the chain manifest, the envelope scratch and the local sweep. Sharing a snapshot alone would corrupt both chains — each run advances it, so each profile would treat the other's work as already archived.

Two locks are held. The per-profile lock is correctness: two runs of one profile share a snapshot and a manifest. The machine-wide lock is courtesy and I/O: a run that finds the other profile working reports itself skipped and waits for its next tick.

What a backup is

Two shapes, chosen by How backups are taken.

Incremental (default). A chain: one full, then runs that carry only what changed. Measured on a real site, the first run's file archive was 193 MB and the next was 37 kB. The site is archived as two parts, each with its own incremental history: data, the site directory less public_html/ (uploads, config, static files), and code, public_html/.

{path_prefix}/{slug}/{profile}/chain-{YYYYMMDD_HHMMSS}/
    manifest.json           the restore contract — order, levels, hashes, sealed keys
    data-0000.tar.gz.enc    the site less public_html: the full
    code-0000.tar.gz.enc    public_html, rooted at public_html: the full
    db-0000.sql.gz.enc
    meta-0000.tar.gz.enc    shape.json + virtualhost + a note of the run
    data-0001.tar.gz.enc    an incremental
    code-0001.tar.gz.enc    a full again when an upgrade swapped the code
    db-0001.sql.gz.enc
    ...

The manifest (version 2) records a level on every artifact of a kind that increments — data, code, and a physical database backup, pgdata — so one kind can start over inside a chain while the others carry on; a run's own level is 0 exactly when every kind in it is. A restore takes each kind from its newest level 0 at or before the run, forward to the run (BackupChain::restore_plan(); restore_chain.sh applies the same rule). A version 1 manifest is a chain written before the split: one files-NNNN.tar.gz.enc a run, the whole site directory, and one level per run that is the files archive's. Every reader reads versions 1 and 2, and refuses a version it does not know, by name, rather than restoring part of a chain.

Full every time. One self-contained archive per run:

{path_prefix}/{slug}/{profile}/{project}-{YYYYMMDD_HHMMSS}.tar.gz.enc            the archive
{path_prefix}/{slug}/{profile}/{project}-{YYYYMMDD_HHMMSS}.tar.gz.enc.keys.json  its envelope

A database-only backup is the same shape with {database}-{YYYYMMDD_HHMMSS}.sql.gz.enc as the archive. All three families carry the same stamp, which is what a management node's retention sorts backup storage by.

Nothing an engine produces lands on this disk. The data and code archives, the standalone archive and the database dump are each one pipeline — tar | openssl or pg_dump | gzip | openssl — whose stdout the runner hands to S3Signer::put_stream(), which uploads it as it flows. What a run holds on disk is the metadata artifact (kilobytes), the chain manifest, the snapshot, and, for a standalone archive, the compressed database dump for the length of the tar (it is a member of that archive, staged beside the output and removed with the staging directory). A standalone archive is built from the live tree — a second -C into the site root with the members renamed under project_files/ — so no copy of the site is ever made. On the smallest nodes this is the difference between a site that can be backed up and one that cannot: peak disk during a run is the site itself.

The engine's verdict arrives after its bytes. tar reports a file that changed while it was being read as exit 1 (accepted — the normal case on a live tree) and a real failure as 2 or more; pg_dump reports its own. So each engine in stream mode (--archive -) writes its statuses to a report file (--report FILE) once the stream has been fully produced, and the runner completes the upload only when the process exited 0, the report says tar 0 or 1 (or pg_dump 0), openssl 0, and at least 64 bytes went up — an openssl envelope around an empty stream is 32 bytes, and a backup of nothing is never recorded as a backup. Anything else aborts the upload: nothing partial or empty is ever in backup storage, and a chain run that fails this way is discarded under the ordinary rule (snapshots and manifest put back as they stood, the run's other objects deleted where the credential can delete).

Everything is AES-256-CBC (PBKDF2, random salt). slug defaults to the project directory name — the same value a management node would use for this site — so a standalone site that later joins a fleet keeps one location instead of starting a second pile beside the first. profile separates the parties, so a listing can always say whose backup an object is and each party's retention addresses only its own backup storage.

What a backup carries besides files and database

A backup has to be able to rebuild a site on hardware that is not the hardware it came from, so it records the facts about that hardware which a restore has to settle. shape.json, written by reconcile_site.sh --print-shape into the archive (full mode) or the meta artifact (chain mode):

{prefix}/{slug}/{profile}/
    chain-{id}/
        objects-0003.json.gz          the run's index — plain gzipped JSON, like the manifest
    {slug}-{stamp}.objects.json.gz    a standalone full's index, same stamp as its archive
    objects/
        {epoch}/
            envelope.json             the epoch's sealed data key
            {fbb_stored_name}.enc     one object per offloaded blob

Epochs are the key model. One data key with one envelope at objects/{epoch}/envelope.json — the ordinary envelope, sealed to the same two recipients as every chain — encrypts every object stored while the epoch is current, in the same aes-256-cbc-pbkdf2 form as an archive, so the Opening a backup with no Joinery anywhere procedure opens an object unchanged. A new epoch starts when there is none, when the recovery recipient changed, or when the site key cannot open the current envelope (degrade, never fail every run). On recovery-key rotation the older epochs are re-sealed, not re-encrypted: each envelope the site key opens is rebuilt with the new recovery recipient added and uploaded again under its name, so the new key opens everything and the old key still opens what it always did. An epoch the site key cannot open stays sealed to the retired key alone; the run writes those down (objects/retired-epochs.json), names them in its message (so a management node's job result and this site's history say so), and Recovery Readiness says so on the recovery-key card — "N objects (X GB) open only with a retired recovery key" — with no automatic re-copy. Keep that key.

Enabled profiles, and the hold. A profile stores offloaded files when it is enabled (BackupProfile::enabled()): the site profile when a target is enabled, the recovery key proven and the backup type includes files; the manager profile when this machine has joined a management node and a manager run carrying the object store has been here (the objects/enabled marker). An offloaded file's local bytes stay on this server until every enabled profile holds it — the site profile by the store the offload tick makes on the way out (docs/cloud_storage.md § An offloaded file is in backup storage before its local copy goes), the manager profile by its held.json, which the management node's runs write. Only then are the original and its variants released. With no profile enabled nothing is held, and the file store alone serves the file.

The run. Before the archive: the epoch; what backup storage holds (the site profile lists its objects/ prefix; the manager profile reads the newest manager index through the presigned objects_index_url in its request, union held.json); the exclude list; then the store — every cloud blob not held, one at a time under the offload engine's per-row lock, local originals first, then catch-up: a blob with no local bytes is fetched from the file bucket, encrypted and uploaded, both temporaries deleted. One budget covers the whole step, 2 GB or 20 minutes (BackupRunner::OBJECT_STORE_BUDGET_*, constants); what it leaves is indexed stored: false and taken next run, and the Backups page says "N files (X GB) still to copy from the file store" until it reaches zero. Then the engines with the exclude file, the index, the commit, and — after the run is offsite — held.json and the release. A run whose request did not ask for objects (a management node not running the object store, a database-only backup) stores nothing and holds nothing.

The manager run request carries three fields from a management node running the object store: objects: true, objects_index_url (a presigned GET for the newest manager index, or absent when there is none) and epoch_envelope_urls ({epoch id: presigned GET} for every envelope in the listing, what the re-seal reads). A node whose management node sends none of them behaves as if the object store did not exist.

Node disk. Every step is bounded:

What each backup storage protects against. The object store protects against a deleted bucket, a revoked key and an accidental rm in the file store. It does not protect against losing the account backup storage is on: when the site's backup target and the file store share an access key, the Backups page and the cloud-storage page both say "Backup storage and the file store share one account; losing it loses both. A management node's copy would survive it." The same-key test is the whole rule.

On the pages. The Backups page's Offloaded files box: how many files live in the file store; per enabled backup, "Offloaded files in backup storage (this site's backup): N objects, X GB; last indexed at the run of …"; what is still to copy from the file store; "M files (X GB) waiting for the management node's backup before their local copy is released" (every cloud row whose original is still on disk — one stat per row on page load, no column, no cache; BackupObjectsStatus); the same-account line; and the daily file-store check with Bring them back (§ The file store is checked too). The cloud-storage page's Status block has the waiting count and size. The node's Backups tab in the reads the backup storage listing: "Offloaded files in backup storage: N objects, X GB by this management node; last indexed at the run of …", with Bring them back. And because a count on a page is read by nobody, BackupObjectsNotice stands on every admin page when the waiting bytes pass 2 GB or anything waits for a backup that has not succeeded in 7 days: "N files (X GB) are waiting for the management node's backup, which last succeeded D days ago. They stay on this server until it does." It clears itself when nothing waits.

What a restore reconciles

Both restore paths — the archive and the chain — end in reconcile_site.sh, which makes the restored site agree with the machine it landed on. It reports each value it changed and refuses rather than papering over a mismatch it cannot fix.

The identity, in both places it lives:

php maintenance_scripts/sysadmin_tools/backup_envelope.php open \
    --sidecar manifest.json --private ~/recovery.key --key-out /tmp/k
bash maintenance_scripts/sysadmin_tools/restore_chain.sh {project} \
    --artifacts {downloaded chain dir} --key-file /tmp/k [--seq N] [--domain d]

Every artifact is checked against its recorded size and hash before anything is written, so a truncated download fails while the live site is still intact. --seq N restores as at run N; the default is the newest. --dry-run reports the plan and needs no key. --domain names the domain the restored site is to answer to; without it the site keeps the domain this machine's config already names. --skip-ssl is passed to the reconcile, which then arms no certificate retry.

Extraction replays the chain the way it was taken: anything in the target that the archive does not list is removed, at any depth, including a directory the files engine never archives (vendor/, say) when the source had none of that name. A directory the source had but the archive left out (logs/) is kept.

--adopt-secret-key is for a chain from another machine. This machine keeps its own config/Globalvars_site.php and takes the chain's secret_box_key into it: read from the chain's copy of the file by pattern (never run), required to be 32 base64-encoded bytes, written with var_export, and linted before it replaces the file, which keeps its owner and mode. Every value the source sealed then opens here, and the database password, the paths and the rest stay this machine's. It is refused before anything is written when this machine has no config of its own, and after the files when the chain's config carries no usable key (this machine's key is then left in place and the database is not loaded).

Offloaded files. The archives carry no file the site offloaded to its file bucket; those are in backup storage under objects/{epoch}/, named by the run's index (objects-NNNN.json.gz beside the run's archives). Download that tree with the site's own credential and hand it over:

bash maintenance_scripts/sysadmin_tools/restore_chain.sh {project} \
    --artifacts {downloaded chain dir} --key-file /tmp/k --objects {downloaded objects dir} \
    [--objects-mode missing|all] [--epoch-key epoch-20260901_000000=/tmp/e]

The step runs after the database is loaded, because the blob row is what says where each file belongs and how big it is. It runs the restored tree's own utils/restore_objects.php: each object is checked against the index's size and hash before it is decrypted, decrypted into its placement, checked against its row (size, and fbb_sha256 where the row records one), put in place, and only then is the row set to local. missing (the default) brings home only what the file bucket cannot serve — the bucket is HEADed per file — and all brings every offloaded file home, for a site leaving its bucket. Nothing on disk is overwritten (a file already at the placement is adopted when it matches its row and refused by name when it does not) and nothing in any bucket is deleted; running it again finishes what an interrupted run left. Variants are regenerated on demand. Each epoch's envelope opens with the machine's own backup_site_key; for an epoch it does not open, recover the key with backup_envelope.php open --sidecar {epoch}/envelope.json --private … and pass it as --epoch-key. The script can be run on its own, and --dry-run says how many files would come home and from which epochs:

php public_html/utils/restore_objects.php --index {chain dir}/objects-0003.json.gz \
    --objects {objects dir} [--mode missing|all] [--dry-run]

From the dashboard: the node's Backups tab lists every run in the node's backup storage, newest first, with the last backup, the last full backup and the oldest backup held stated above the list — and, when a run in backup storage carries an offloaded-files index, an Offloaded files row with Bring them back, which runs the restore_objects loop below in missing mode against the newest such run without restoring anything else; each row's Restore button runs the restore_chain job for that run. That job recovers the chain key on the node from the node's own backup_site_key, so no recovery private key travels in a job record. Once it completes, the run's offloaded files follow in missing mode as the paged restore_objects jobs described under Restoring a managed node below. A chain taken by a machine that no longer exists is restored from a shell with the recovery key, as above.

Checking a restore: the census

A restore onto another machine is checked by counting what each side holds and comparing the counts. maintenance_scripts/sysadmin_tools/site_census.php (no arguments, run as root so it can read every directory) prints one CENSUS= line with a JSON object:

• tables: the rows in every table (a table outside public is named schema.table);
• files: for each top-level directory of the site, the files, directories and bytes, and a digest over every path with its size, so a renamed file or two sizes that changed by amounts that cancel still show; loose files at the site root are counted under .;
• secrets: whether the key canary opens (ok, dead, absent, or nokey when the site has no secret_box_key), and how many registered sealed secrets are stored and how many of them are dead, by where they live.

php -r '
$env = json_decode(file_get_contents($argv[1]), true);
$sk  = base64_decode(trim(file_get_contents($argv[2])));
$kp  = sodium_crypto_box_keypair_from_secretkey_and_publickey(
           $sk, sodium_crypto_box_publickey_from_secretkey($sk));
foreach ($env["recipients"] as $r) {
    $key = sodium_crypto_box_seal_open(base64_decode($r["sealed"]), $kp);
    if ($key !== false) { file_put_contents("data_key.txt", $key); exit; }
}
fwrite(STDERR, "no recipient in this envelope opens with that key\n"); exit(1);
' {archive}.keys.json recovery_private.b64

openssl enc -d -aes-256-cbc -pbkdf2 -pass file:data_key.txt \
    -in {archive}.tar.gz.enc -out {archive}.tar.gz

php maintenance_scripts/sysadmin_tools/escrow_keypair.php generate --private-out ~/recovery.key
php maintenance_scripts/sysadmin_tools/escrow_keypair.php unseal   --private ~/recovery.key

It exists for one adversary: the party that chooses where a restore's bytes come from. When a management node runs this machine's backups it also owns the bucket and signs the download, and the person approving a restore approves a name and a date — they never see the bytes. The ledger is the only thing on the machine able to say whether those bytes are the ones it uploaded under that name. Two attacks fail against it, and the second is the one that carries it:

• Forgery — an artifact whose content is simply made up. The hash does not match.
• Replay — this machine's own genuine month-old archive, served under a fresh-looking name. Every signature verifies and every envelope opens, because it really is this machine's backup; sealing does not touch this at all. The name it is offered under has no record, so it is refused.

The address is chosen for two properties that are invisible from the code that reads it. config/ is a named volume on a container node, so the ledger survives a container rebuild — a ledger under /var/lib would not, and since a ledger only records what has been uploaded since, restore would stay broken for as long as the current chain is old. And restore_project.sh drops config/backup-ledger from a staged archive the same way it drops Globalvars_site.php and backup_site_key: they are the machine's, not the backup's, so the first restore cannot overwrite the record that vouches for the second.

A backup taken by an unprivileged process cannot write the ledger; the run reports that rather than hiding it, because an unledgered artifact is one the machine will refuse to restore from over the agent channel. On a managed node the backups that matter are taken by the root agent, so they are ledgered as a matter of course.

The ledger is 0700/0600, and both the platform and the agent refuse one that group or other can write rather than reading it. Anything that can write the ledger can vouch for any bytes it likes, so a loose ledger does not weaken the check — it makes the check report success. The test is on the mode, not on the owner: backups legitimately run as root on a managed node and as the site user elsewhere. fix_permissions.sh pins the directory out of its sweep to match.

Retention

• Cloud — keep backup_retention_days days of restore points (default 28): every restore point that started inside the window, and the newest one that started before it, which is what a restore to the window's first day replays from. The window is days rather than a count because chains differ in length (the configured interval, a key rotation, a lost snapshot), so a count of chains covers an unpredictable span. The rule is BackupRunner::surplus(), shared with the management node's retention. Older restore points are deleted, driven by this site's own run history rather than by a bucket listing, so it can only ever delete objects this site recorded writing. Retention runs last in a backup, and only after an upload is confirmed: a run that failed must never be the run that decides an older backup is surplus.

bash maintenance_scripts/sysadmin_tools/restore_project.sh {project} /backups/{archive}

php maintenance_scripts/sysadmin_tools/backup_envelope.php open \
    --sidecar {archive}.keys.json --private ~/recovery.key --key-out /tmp/k
bash maintenance_scripts/sysadmin_tools/restore_project.sh {project} {archive} --key-file /tmp/k

Bring the backup back. Every node deletes its local archive once it is safely uploaded, so the normal state of a machine is that its backups are all offsite — and a restore takes the name of a file it expects to find in its own backup directory. Bring back to node on the node's Backups tab (or Prepare, for a chain) fetches it. No bucket credential is sent: the management node signs one object key, for no longer than the job's own claim budget, and the node receives the signature. A node's stored credential is write-only by design, because a node that could read backup storage is a node whose compromise reaches every other node's backups. Everything fetched is checked against the node's own upload ledger before it lands, and lands 0600 — created that way, not chmod'd afterwards, because on a container node the backup directory is inside the site tree and a descriptor opened during a multi-gigabyte transfer stays open. The transfer is capped at the size the ledger recorded, so a response that declines to say how big it is cannot run the node's disk to zero; and a failed download reports its HTTP status without its body, because the plane picks the URL and reads the transcript.

Ask for the restore. The management node dispatches it as a primitive. It sends a name and a profile — no path, no key, no bucket, no domain. A chain restore also carries a project, and the node treats it as a claim to check rather than an instruction: restore_chain.sh spends that value twice, on the tree it replaces and on the database name it loads over, so the node uses its own project and refuses a job naming any other.

Approve it on the node. The node's agent claims the job and runs nothing. It composes its own statement of what it would do — which project, which database, which archive, the archive's real age, size and fingerprint — from its own records, seals a one-time challenge to the backup recovery public key it already holds, binds it to that job and that statement, and stages it for the node's own site. The node's Backups page shows the pending approval; the operator opens the challenge there with their recovery key, in their browser, and answers. The agent verifies the answer against what it sealed and only then restores.

What a database restore leaves behind, and what it does not roll back. A database restore replaces the database and nothing else, so a dump taken before an upgrade lands under files that are still on the newer release: the site's recorded system_version and its settings go back to the older value while the code on disk stays put. This is self-correcting — the next upgrade re-runs the migrations, which are idempotent — but it means the version a freshly restored node reports from its database can trail its files until then. A restore also leaves the archive it was given in the backup directory; the local sweep expires it on the ordinary keep_local_days schedule.

Bring the offloaded files home. The archives carry no file the site offloaded to its file bucket, so once the chain restore completes the plane brings those home too, in missing mode — only what the file bucket cannot serve. The node never lists its backup storage and holds no read credential, so every object arrives by a link the plane signs, and a job is bounded, so the work is paged and the plane drives the loop (FleetObjectRestore): a survey job (the restore_objects primitive with the run's index link and no object links) has the node read the index — ledger-checked like every artifact — HEAD its file bucket per offloaded file, and answer with the names it would bring home (RESTORE_OBJECTS_WANT, at most a thousand, RESTORE_OBJECTS_MORE when there are more). The plane signs a page of links from that answer — object links plus the envelope of each epoch those objects are sealed under, filled to the job's byte ceiling and never more than 150 — and each page's result issues the next, so no link waits in a queue past its expiry. When the pages are done and the survey said there was more, a fresh survey names the next thousand; a survey that names nothing ends the loop. The node opens each envelope with its own key, fetches objects one at a time, checks each against the index before decrypting it and against its row after, and sets the row to local only once the file is in place. restore_objects is an operate primitive: it overwrites nothing and deletes nothing in any bucket, so no page needs an approval. A failed page ends the loop with its reason on its own job; a restore whose result is read long after it finished starts none. The job's record names the survey it pages and its slice, and the survey's result holds the names, so a store of ten thousand files is stored once, not once per page.

The management node is not in that path at all — not as a gate, and not as a relay. The challenge and the answer live entirely between the node's own site and its own agent, and the restore vocabulary declares no parameter through which an approval answer could travel, so relaying one is impossible by wire format rather than by care. A management node can dispatch a restore and can do nothing whatsoever to get it approved.

The costs are deliberate and worth stating. Restoring in place requires the node's site to be up. A node whose site will not boot is rebuilt by hand: a fresh site installed on a new machine, then the backup restored onto it from a shell with the recovery key (Rebuilding a site on new hardware). No approval is asked there, because holding the recovery key is the proof. An unanswered challenge expires and the job is refused, so a restore nobody is watching fails rather than pinning the node. And a support-driven restore requires the customer to be reachable: there is no unattended destructive path, including for us.

The archive's age is on the approval screen as an age, above the key box, because it is the one fact no automatic check can substitute for — a replayed archive is genuine, signed and openable, and only its date is wrong.

Verifying backups

A backup that has never been opened is a hope. Backup storage listing proves a backup is present; restore_chain.sh --dry-run proves it is intact against its manifest; Prepare proves the key this machine holds opens its envelope. Verification is the step that proves a backup is recoverable, without restoring it, and records the proof where people look: the site's own Backups page, the node's Backups tab on its management node, and the node's card on the management dashboard. The word every page uses is verified restorable, with a date, and the level it was proven at:

The node's history row is the authority. A verify stamps the run it opened (bkh_verify_time, bkh_verify_level 2 or 3, bkh_verify_outcome pass or fail, bkh_verify_message in plain words) whichever profile took the run. A verify that was skipped — not enough disk, a throwaway database that could not be created, a machine busy with a backup for too long — stamps the message only, because nothing was proven either way: the last real outcome stands and the skip's reason rides beside it. A request refused before anything was read is noted the same way. The management node's copy (mgn_backup_verify_*) is refreshed from the job's result lines and from the node's status report, so a verify the site ran itself is seen from the management node too.

A failed verify is surfaced exactly like a failed backup, in the node's own words, and never triggers anything automatic — not a re-run, not a deletion, not a fresh full. A failing verify is a person's problem to look at.

The engine

utils/verify_backup.php is the script a node runs (as the verify_backup agent primitive, or from the site's own scheduled task and Backups page). It takes JSON on stdin and nothing on argv: the chain, the profile, the level, a signed link to the manifest and a signed link to every object under the chain, keyed by bare name, optionally the run, and — when the run carries offloaded files — a signed link per epoch envelope the run's index names (epoch_envelope_urls, keyed by epoch id) and, for a rehearsal, the sample (object_urls, keyed by the object's name in the index, at most 20). Whoever signs the links reads the run's index from backup storage to pick them: the site's own launcher (BackupVerifyLauncher::object_links()) and the management node's builder alike. Everything about what may be fetched is BackupStaging's, shared with Prepare, so a verify can never fetch something a Prepare would refuse: the manifest is read on the machine, the artifact list comes from BackupChain::restore_plan(), every fetch is checked against the upload ledger, and the chain key is recovered from the machine's own config/backup_site_key. Offloaded files are not ledgered — their index is — so BackupStaging::fetch_objects() checks each sampled object against the index's recorded size and hash instead, and lands it under objects/{epoch}/ in the working directory. BackupVerifier is the level 2 and 3 engine over an already-staged directory (read_all, rehearse, disk_needed, sample_objects). A management node sends the two link maps with every verify of a run whose index names offloaded files; a run that carries offloaded files but whose request links no envelope for an epoch they need fails by name, because the objects under it cannot be opened here.

The script works in {site}/backups/{profile}/verify-{pid}/, under the same two locks as a backup run (so it never reads a chain a run is writing), checks free disk before downloading anything (the set for level 2; the set plus twice each tree archive's full plus three times the dump for level 3), and removes its working directory, scratch tree and throwaway database on every exit path, including a fatal. The local sweep removes any verify-* directory older than a day that a killed process left behind.

It answers with one key per line:

VERIFY_RESULT=pass|fail|skipped
VERIFY_LEVEL=2|3
VERIFY_RUN=<chain_id>/<seq>
VERIFY_RUN_TIME=<manifest run time, UTC>
VERIFY_ARTIFACTS=<n read>
VERIFY_BYTES=<bytes read>
VERIFY_FILES=<entries listed (2) or files restored (3)>
VERIFY_OBJECTS=<offloaded files proven recoverable: stored, epoch envelope opened>
VERIFY_OBJECT_BYTES=<their bytes in backup storage>
VERIFY_OBJECTS_SAMPLED=<n opened and compared>     (level 3 only)
VERIFY_TABLES=<n>                            (level 3 only)
VERIFY_ROWS=usr_users:<n>,<table>:<n>,…      (level 3 only)
VERIFY_DURATION=<seconds>
VERIFY_REASON=<one line>                     (fail or skipped only)
VERIFY_NEEDS_BYTES=<n>                       (skipped for disk only)
VERIFY_FREE_BYTES=<n>                        (skipped for disk only)

Exit 0 on pass or skipped, 1 on fail, 2 on a request it could not understand. A skip's reason is disk (with both numbers, so the card can say "needs N free, has M"), createdb, or busy. An object retention deleted from under a verify in flight fails with reason gone, naming the object — an archive, an epoch envelope or a sampled offloaded file alike; the next pass verifies the newer backup, and the retention pass never holds a deletion for a verify — a verify must not be able to keep a backup alive.

The site's own backups

A site verifies its own backups the same way: the Backup verification scheduled task (tasks/BackupVerify.php, switched on together with Backup by BackupNightly) opens and reads the site's newest own backup every backup_verify_every_days (30 by default; 0 never), signing the links from the site's own target through S3Signer::presign_get — no credential reaches the script. It is due when the interval has passed since the last verify (pass or fail) and a newer backup exists, or when nothing has ever been verified. A site that takes no backups of its own has nothing to verify and the task says so, once, as a skip. The Backups page's Verify the newest backup button runs the same thing on demand, and Rehearse a restore runs level 3; both run in the background like Run a backup now (the request crosses on the detached process's stdin, handed over on an explicit descriptor — BackupVerifyLauncher::detach), and the result lands on the run's row under Recent backups and in the Status box's Last verified restorable line. A verify that proves nothing either way — skipped for disk or a busy machine, or refused before it read anything — leaves its reason as the run's message with no outcome (BackupVerifier::stamp_history / note_history); the row says not verified with the reason, or since then: beside a proof that still stands, and the Status box says Last attempt: when it is about a backup no older than the last proof.

BackupVerifyLauncher is the site side (newest_run, due, request, run, start).

The file store is checked too

Verification proves backup storage holds what the index says. The other side — does the file bucket still hold every offloaded file it is serving? — is asked daily by the offload tick (CloudStoreInventory, docs/cloud_storage.md § The file store is checked daily): every cloud blob is HEADed in its bucket, a slice per tick, and a file the bucket lacks or holds at the wrong size is written down by name. The Backups page's Offloaded files box and the cloud-storage page's Status block both say when it last looked and, when anything is missing, "N offloaded files are missing from the file store; the backup holds M of them" — M counted against the enabled profiles' held sets — with Bring them back beside it:

• A site with a backup target of its own runs it here, in the background (BackupObjectRestoreLauncher::start_newest() → utils/bring_back_objects.php, the request on stdin as a verify's is). The launcher reads the newest run's offloaded-files index off the site's own backup storage, surveys it in missing mode (the bucket is HEADed per file, so a file the bucket has recovered by then is left alone), and brings the rest home a page at a time — each page's object and envelope links signed here with the site's own credential (S3Signer::presign_get) and opened with its own key; the engine is the same BackupObjectRestore the node path runs. One at a time, by a lock in the profile's objects/ directory. What it did lands on the inventory record, both pages read it, and the names it brought home leave the missing list at once.
• A site backed up only by its management node holds no backup storage credential, so the box names the management node and its URL: from that node's Backups tab, Bring them back (the restore_objects backup action → FleetObjectRestore::start()) runs the paged survey-and-page loop described under Restoring a managed node, in missing mode, against the newest run in backup storage that carries an offloaded-files index. The management node does not see the node's missing list; the survey job asks the node.
• A site with neither is told nothing holds them.

php public_html/utils/bring_back_objects.php --chain chain-20260912_044520 --seq 3 [--mode missing|all]

maintenance_scripts/sysadmin_tools/verify_backup.sh drives the same engine over a chain directory an operator downloaded by hand, with a key the operator recovered. This is the one verification path that exercises the recovery private key, and the runbook for proving that the key in a password manager still opens a real backup:

# 1. Download the chain's whole directory from the bucket to DIR
#    (manifest.json and every artifact it names).
# 2. Recover the chain key with the recovery private key.
php backup_envelope.php open --sidecar DIR/manifest.json \
    --private /path/to/recovery.key --key-out /tmp/chain.key
# 3. Open and read (level 2), or rehearse a restore (level 3).
./verify_backup.sh --artifacts DIR --key-file /tmp/chain.key --level 2
./verify_backup.sh --artifacts DIR --key-file /tmp/chain.key --level 3 [--seq N] [--project NAME]
# 4. Shred the key file.
shred -u /tmp/chain.key

It runs from a site's maintenance_scripts/sysadmin_tools (the engine is PHP in the public_html beside it, and a rehearsal uses restore_chain.sh and restore_database.sh from the same directory), prints the same VERIFY_* report, leaves the chain directory as it was (a rehearsal's scratch/ is removed), and exits 0 on pass or skipped, 1 on fail, 2 on a request it could not understand. --project names the directory the archive carries, for a rehearsal; left out it is read from the archive, and a wrong name is refused. This path holds a chain key, not the site key, so it proves the archives and reports the run's offloaded files unproven (VERIFY_OBJECTS=0); those are proven by the fetching path, which opens their epoch envelopes with the site key.

Disk and egress

Opened-and-read downloads the whole set each time; on Backblaze that is egress. At the monthly default on a 1 GB set it is inside the free allowance for a fleet of nodes; weekly across nine nodes would still be under 40 GB a month. A rehearsal needs scratch disk of roughly twice the site plus the database, and a PostgreSQL role that can create a database. Both are checked before anything is downloaded, and the machine says the numbers when it declines.

A backup needs no scratch disk beyond the site itself. Every archive and every dump streams from its engine into the bucket; a chain run's peak disk is the live tree plus the metadata artifact, a database-only run's is nothing, and a standalone whole-site run's is the live tree plus its compressed dump for the length of the tar. Memory is one upload part, 100 MiB. The double-space need is a restore's and a verify's, not a backup's.

The node tool

maintenance_scripts/sysadmin_tools/backup_envelope.php reads a backup's key back. open, relabel and site-key are deliberately standalone — no platform bootstrap — so they work during disaster recovery when the site will not boot, which is the moment that matters. mint runs only on a live site and does read its settings, because it must know which recovery key that site holds: nobody hands it one, and --recovery-pub is refused.

A row still marked running whose start is more than six hours old (SiteBackupNotice::STALE_RUN_HOURS) is a run that died without writing its end — the kernel killed it, or it lost its database and could not get it back. The notice names it as started at … and never finished. The runner makes the second case rare: when recording a failure throws, BackupRunner::fail() asks DbConnector::reconnect() for a fresh connection once and saves again, because the failure that ended the run (a full disk, a PostgreSQL restart) is often the one that killed the connection. reconnect() rebuilds the current mode's link from the same settings; it is for a long-running CLI process, and nothing that runs inside a web request calls it.

Because manager-profile rows land in the site's own database, the site can answer "does someone back me up?" locally: BackupHistory::manager_coverage() returns the newest manager-profile success that reached its bucket within MANAGER_COVERAGE_DAYS (7), or null. The setup wizard's Backups step reads it as a green condition, so a fleet-backed node is not asked to configure a bucket it is already archived to; coverage goes stale on its own if the management node's runs stop.

Saving a target

A target is proven before it is saved. Both forms (the site's Backups page and the fleet's Targets page) run TargetTester::test() on Save of an enabled target and refuse the save when it fails, printing why with the values kept; the separate Test button runs the same check on a saved target. A disabled target is saved untested, and enabling it is a save. The steps, in order, stopping where the rest would be meaningless:

1. Its own bucket. Not the file store's bucket (BucketCheck::collision_step, by name and endpoint host). Refused outright, before the network: one bucket for files and backups means one deletion or one public-read setting takes both.
2. Reach. The main key lists the bucket (one ListObjectsV2, one key).
3. Write. The main key stores a probe object under the target's prefix.
4. Private. An anonymous read of that probe is refused. Backups are encrypted, but a bucket anyone can read is refused all the same.
5. Prune. The main key deletes the probe; retention needs that.
6. Node key, when one is set. It stores a probe and cannot* delete it; the main key cleans up. A node key that can delete is refused, because a node handed it could erase backups.
7. Backblaze keys. Each key states what it may do (b2_authorize_account's allowed, kept by B2Client::authorize()): a key pinned to another bucket is refused; one that opens every bucket on the account passes with a warning naming the file store buckets it also opens; a missing capability (listFiles, readFiles, writeFiles, deleteFiles, plus writeKeys, listKeys, deleteKeys when minting per run is on) is refused naming it and what it would break.

A target an installer creates

A first-boot installer that was handed a bucket and its key pair (the Linode deploy form's optional backup fields) creates the first target itself with utils/install_backup_target.php: the target is saved, a Backblaze credential's region and endpoint are filled from Backblaze's authorize answer (BackupTarget::complete_credentials, the same call the Backups page makes), the target is proven (§ Saving a target), and a first target becomes the scheduled one. A target that fails is removed again so the setup wizard asks for one. The recovery key is never created here — it is shown once to a human — so nightly runs still wait on the wizard's key ceremony, exactly as they do for a target saved on the Backups page. Inputs are environment variables (JOINERY_BACKUP_BUCKET, JOINERY_BACKUP_KEY_ID, JOINERY_BACKUP_KEY, optional JOINERY_BACKUP_PROVIDER b2/s3/linode and JOINERY_BACKUP_REGION); the first output line is INSTALL_BACKUP_TARGET=ok or =error.

Artifact naming

includes/BackupNaming.php owns which files are backups, what each one is, and what restoring it would do. Every surface consults it — the management API's local listing, the node Backups tab, the job builder's globs.

Recognized suffixes are matched longest-first: .sql.gz is a suffix of .sql.gz.enc, so a shortest-first match would classify every encrypted dump as plaintext and hand the restore engine a file it will not decrypt.

Settings

The account backups run as

The scheduled task runs as the web user, which is what fix_permissions.sh makes the owner of the site tree. A backup that cannot read part of the tree fails rather than shipping a partial archive, and says which file it could not read — so a permissions problem surfaces as a failed run with a specific cause, not as a backup that turns out to be incomplete on the day it is needed.