Server Manager

The Server Manager plugin provides a web UI for managing remote Joinery production servers. Operations include status checks, backups, database copies, and applying updates -- all from the admin interface at /admin/server_manager.

The system has two components:

• PHP plugin (plugins/server_manager/) -- admin UI, job creation, command generation
• Go agent (/home/user1/joinery-agent/) -- generic step executor that polls the job queue and runs commands via SSH

Quick Start

1. Install the plugin

The plugin is already in the plugins/ directory. From the admin panel:

1. Go to /admin/admin_plugins
2. Click Actions on "Server Manager" and choose Install
3. Click Actions again and choose Activate

2. Install and start the Go agent

Release channel (how the agent normally arrives and stays current)

The agent ships inside the platform release. Publishing an upgrade bundles a signed agent artifact into plugins/server_manager/agent_dist/:

• manifest.json — agent version plus, per architecture, the artifact filename, its sha256, and an Ed25519 signature over the raw binary
• joinery-agent-linux-amd64.gz / joinery-agent-linux-arm64.gz — the binaries
• joinery-agent.service — the systemd unit

Bundling is the first thing a publish does, before the VERSION file, the archives or the release row, because its outcome decides whether the release happens at all:

• No agent source on this box — the existing artifact carries forward unchanged and the publish proceeds. Publishing never depends on a Go toolchain being present.
• Source version matches the bundle — agent_dist is left byte-identical, which keeps the server_manager plugin tree hash stable between publishes.
• Source is newer and the rebuild succeeds — the fresh artifact is bundled, and because this happens before plugin archives are built it is captured in the server_manager archive and its tree hash.
• Source is newer and the rebuild fails — the publish is refused. The build error is printed, and the VERSION file, archives and release row are all left untouched. Shipping here would mean releasing an agent the publisher already knows is out of date, and the resulting fleet has no way to tell.

First install is handled by the plugin's host_installer (provisioning/install_agent.sh), which runs at every root moment — site install, code upgrade, container start, and the node-detail Run Plugin Installers action. It installs the bundled binary, writes the env file with the right JOINERY_CONFIG, and sets up systemd or cron supervision automatically.

Every later version change is handled by the agent itself. Between jobs, the agent compares its own version with the bundled manifest. When they differ, it decompresses the artifact, checks the sha256, verifies the Ed25519 signature against the public key embedded in its binary, keeps the current binary as .bak, renames the new one into place, and exits cleanly for its supervisor to restart. The signature check is the security boundary: the site tree is writable by the web user while the agent runs as root, so the agent never installs anything the publisher did not sign. An artifact that fails verification is refused, logged under a === Self-update === header, and not retried until the manifest changes.

If the new binary fails to initialise (config, DB, or schema), it restores the .bak over itself and records the bad version in a .rejected marker — the supervisor restarts the previous working agent, and that version is never reinstalled; the next release supersedes the rejection. On the first fully healthy start after an update, the .bak and any stale marker are removed.

The dashboard's Agent Status bar surfaces all of this from the heartbeat row (ahb_bundled_version, ahb_update_state): a pending update, a refused (verification-failed) artifact, a rolled-back version, or an agent built without an update key.

Manual install (bootstrap fallback)

For a control plane that has no bundled artifact yet, build and install by hand:

cd /home/user1/joinery-agent
PUBKEY=$(cat /var/www/html/joinerytest/config/agent_signing_key.pub) make release
sudo bash joinery-agent-installer.sh --verbose [--config /path/to/Globalvars_site.php]

make release compiles the binary (passing PUBKEY bakes in the update-verification key so the manual build can still self-update later) and packages it into joinery-agent-installer.sh, a self-extracting script that handles both fresh installs and upgrades with automatic rollback if the new version fails to start.

The installer detects the host's supervision capability:

• systemd hosts: installs /etc/systemd/system/joinery-agent.service; start with systemctl start joinery-agent.
• No systemd (Docker containers, minimal hosts): installs /usr/local/bin/joinery-agent-supervise plus /etc/cron.d/joinery-agent (@reboot + a once-a-minute keepalive) and starts the agent immediately. Logs go to /var/log/joinery-agent.log.

• /usr/local/bin/joinery-agent — the binary
• /etc/joinery-agent/joinery-agent.env — configuration (from example, first install only)

Every control plane needs a live agent. All jobs (install_node, provision_ssl, backups, upgrades) sit pending until an agent polling that site's own database claims them. The Server Manager → Provisioning page shows an agent heartbeat badge as requirement #1.

Configure (usually not needed)

The agent reads database credentials directly from Globalvars_site.php — no manual configuration required on a standard Joinery install.

The default config path is /var/www/html/joinerytest/config/Globalvars_site.php. If your install is at a different path, set it in the env file:

sudo nano /etc/joinery-agent/joinery-agent.env
# Set: JOINERY_CONFIG=/var/www/html/mysite/config/Globalvars_site.php

Other optional settings in the env file:

Start

sudo systemctl start joinery-agent
sudo systemctl status joinery-agent

The dashboard at /admin/server_manager should now show Agent Status: Online.

If anything is wrong, the agent logs to the systemd journal (systemd hosts) or /var/log/joinery-agent.log (cron-supervised hosts):

journalctl -u joinery-agent -f      # systemd
tail -f /var/log/joinery-agent.log  # cron supervision

Common startup errors are self-explanatory — missing DB_NAME, wrong password, or plugin tables not installed. Each error message tells you exactly what to fix.

Upgrade

Agents upgrade themselves from the bundled artifact after each platform release lands (see Release channel above). The manual-install path also accepts upgrades: re-running the generated installer stops the service, swaps the binary, restarts, and rolls back automatically if the new version fails to start.

3. Add managed nodes

Go to /admin/server_manager/node_add (or click Add Node on the dashboard). There are two ways to add nodes:

Auto-detect (recommended)

The Auto-Detect Joinery Servers panel scans a remote host for Joinery instances automatically. Enter:

1. SSH Host -- the server IP (e.g., 23.239.11.53)
2. SSH Key Path -- path to the private key on the control plane (defaults to /home/user1/.ssh/id_ed25519_claude)
3. Click Detect

Detected instances appear as cards with Add This Node buttons. Clicking one auto-fills the entire form below -- just click Add Node to save.

Auto-detect requires the Go agent to be running (it executes the SSH commands, not PHP).

Manual

Fill in the form fields directly:

Admin Pages

All pages are at /admin/server_manager/... and require permission level 10 (superadmin).

The UI is organized around a dashboard + node detail pattern. The dashboard shows the fleet overview; clicking a node opens a tabbed detail page with all operations for that node.

Node Detail Tabs

The node detail page (/admin/server_manager/node_detail?mgn_id=N&tab=...) has six tabs:

Node console

Some fleet work needs a command no built-in job covers. The Console tab runs one, on one node, and records it.

Turning it on. Overview → Edit → Allow running commands from the Console tab (mgn_allow_console, off by default). Until then the tab shows only a link back to that setting. A newly added or provisioned node is never console-reachable by default.

Running a command. Type it, choose how long to wait (1, 2, 5 or 10 minutes), and — on a container node — choose whether it runs inside the container or on its host. A confirmation dialog shows the resolved execution context (which SSH user, which host or container, the timeout, the command) before anything is sent, so what is confirmed is what actually runs.

The gate. A command is refused unless all of these hold: the operator is a superadmin (permission 10), the node has mgn_allow_console on, and the operator has confirmed a second factor recently in this session — the standing step-up rule in Account Security. When a confirmation is owed, the passkey ceremony runs inline and the form then submits. An account holding no second factor passes that check; the gate binds a factor the account has rather than requiring an enrollment. A refusal re-renders the tab with the typed command intact.

What is not checked is the command. Nothing inspects, classifies or filters it — no shell string can be judged safe by inspection, and a filter that implied otherwise would be worse than none. The bounds are the gate above, the timeout (which kills the command and fails the job), and the audit record.

Privilege is the node's SSH identity's: mgn_ssh_user over the node's key. If that user has no sudo, neither does the console.

The record. Every run is a run_command management job — command, operator, node, timestamps, exit status and captured output — on the jobs pages, on the job detail page with its live output stream, and in the tab's own Recent commands list. node_exec.php writes the same row (marked as coming from the CLI, with no operator), so both ways onto a node produce one audit stream. Stored commands and output are redacted for credential material, including PGPASSWORD=-style env prefixes.

Dashboard Features

The dashboard shows:

• Agent Status -- online/offline indicator with version and last heartbeat time
• Managed Nodes -- cards with health-based status dots (green=healthy, yellow=warning, red=problem, gray=no data), key metrics, and action buttons
• Publish Upgrade -- build upgrade archives from control plane source code (node-independent)
• Recent Jobs -- latest 20 jobs across all nodes

• Red: Last check failed, disk > 90%, or PostgreSQL not accepting connections
• Yellow: Disk > 80% or load average > 5
• Green: All metrics healthy
• Gray: Never checked or no data

Job Types

Note on bare-metal nodes with user1 SSH: When a bare-metal install completes, install.sh disables root SSH and the node's mgn_ssh_user is automatically updated to user1. Subsequent jobs run as user1 with NOPASSWD sudo. All backup/restore commands that need root-level paths (e.g. /backups/) use sudo automatically.

One-Click Node Install

Dashboard → Install New Node opens a form that provisions a Joinery site in a single click. The Target Host dropdown offers three kinds of target:

• A known host (or Other server with manual SSH details): the form creates the ManagedNode and dispatches the install_node job immediately.
• Create a new cloud instance: no server exists yet. The form records an admin-origin CustomerCloudProvision (connected cloud account, region, instance type, plus all install parameters) and the Provision Customer Cloud task births the instance, creates the node, and dispatches the install — see Customer-Cloud Fulfillment. The instance is created in, and billed to, the selected connected account; Linode grants expire after two hours, so connect (or re-connect) shortly before submitting. Cloud targets always take a fresh source backup in From-Backup mode. In-flight provisions appear in a banner at the top of the dashboard.

Two install types:

• Fresh: empty Joinery site with default schema. Admin picks the domain. The admin login is [email protected] with a password generated for that site alone — there is no shared default. Like the generated Postgres password, it stays on the node rather than in the control plane: read it at /var/www/html/{sitename}/config/admin_credentials.txt (root only), or set a new one with maintenance_scripts/sysadmin_tools/reset_admin_password.php. usr_force_password_change=true, so the first sign-in forces a new password.
• From Backup: fresh install + restore of a source node's DB and project files, then reconciliation to the new node — its own domain (the node's recorded URL), its own deployment shape, its own paths. Use source admin credentials to log in; cut DNS over when ready and the certificate is issued on its own.

From-Backup restores files by extracting the source archive with both of its leading path components stripped, taking only the project_files/ subtree — backup_project.sh writes archives as `{backup_name}/project_files/{public_html, uploads,config,...} with the archive's own metadata (apache_config/`, backup_info.txt, the .sql dump) as siblings. The target keeps its own Globalvars_site.php (it holds this machine's database password and secret_box_key) and mints its own backup_site_key rather than inheriting the source's identity as a backup recipient. A verification step then requires every regular file the archive carries to exist at the site root and fails the job otherwise, because a clone whose files did not land still serves pages: the fresh install ran first and the database restore succeeded, so the only symptom is uploaded files missing from where the restored database says they are.

The mgn_install_state column tracks the lifecycle: installing → NULL (success) or install_failed (failure). On failure, the node detail page surfaces a Retry Install button; the target must be cleaned manually (e.g. rm -rf /var/www/html/SITENAME) before retry because install.sh refuses to overwrite an existing site. Postgres passwords are auto-generated and stored in the target's Globalvars_site.php — Server Manager does not capture or display them.

Docker notes:

• The reverse proxy step (manage_domain.sh) is skipped when the domain is a bare IP address — a routable hostname is required for Apache ServerName-based virtual hosting. With an IP domain, the site is accessible directly on its mapped port.
• backup_project.sh requires rsync. The bare-metal and Docker install scripts install rsync as part of the essential packages (install.sh line ~948). Sites installed before this was added can install it manually with apt install rsync.
• After a Docker install, mgn_container_name is automatically recorded in the control plane DB so future jobs correctly use docker exec to reach the site.

SSL Management

SSL State

Each node tracks its TLS certificate state in mgn_ssl_state:

Automatic Detection

check_status jobs include an SSH step that checks for a Let's Encrypt cert under /etc/letsencrypt/live/{domain}/. JobResultProcessor updates mgn_ssl_state and stores ssl_domain, ssl_expiry_raw, and ssl_expiry_ts in mgn_last_status_data. State transitions:

• CERT_FOUND → sets state to active (from any prior state)
• CERT_MISSING → clears state to null only if currently null or active; never overwrites pending or failed

Manual Provisioning

The Overview tab shows an SSL Setup card when mgn_ssl_state is not active, the node has a domain in its site URL, and mgn_cert_expiry_ts is empty. The last condition excludes directly-exposed, self-renewed nodes (see Certificate Expiry Monitoring) — their cert lifecycle is owned by an external renewer (e.g. Caddy), and the card's certbot-based provisioning does not apply to them. The card:

1. Resolves the domain via DNS and shows whether it points to the node's host IP
2. Enables the Provision SSL button when DNS is ready (or when the host IP is not configured)
3. On submit: creates a provision_ssl job, sets mgn_ssl_state = 'pending', redirects to job detail

Cloudflare-proxied domains skip certbot (Cloudflare terminates TLS at its edge) but are gated on a routing probe: the job writes a one-time token into the site's webroot, and the control plane fetches it through the domain. Only a match — proof that traffic for the domain actually lands on this node — patches the proxy's X-Forwarded-Proto and marks SSL active (JobResultProcessor additionally requires the CF_ROUTING_VERIFIED marker). A miss fails the job and the domain stays pending until the customer's DNS actually routes here.

Automated Provisioning (installs only)

For nodes installed via Install New Node, ProvisionPendingSsl (scheduled hourly) watches for nodes with mgn_ssl_state = 'pending', checks DNS, and kicks off provision_ssl jobs automatically. After ~16 hours of failed attempts it flips state to failed — except a Cloudflare domain still waiting on its DNS cutover (CF_ROUTING_UNVERIFIED), which keeps quietly retrying. Manual provisioning via the Setup card is the fallback.

Hosting Provisioning

Paid hosting orders on getjoinery become installed, SSL'd Joinery sites with no human touch. The Poll Hosting Orders scheduled task polls the getjoinery API each cron tick for paid orders carrying an answer to the configured domain Question, and fulfills each one in the mode the product declares in pro_fulfillment_provider:

• Shared host (the default, any other value): the pipeline picks the least-loaded provisioning-enabled ManagedHost, assigns the next Docker port, and dispatches an install_node job. The buyer's site is a container on infrastructure the operator owns.
• customer_cloud: the buyer's site runs on a server in *their own cloud account, billed to them by the provider — see below. A product opts in by picking Customer cloud server in the product-edit Purchase grants picker (CustomerCloudFulfillment, registered with the store's FulfillmentRegistry from serve.php); that stamps the provider value and contributes the domain question as a checkout requirement automatically.

The customer-cloud provisioning keypair (the public half is installed on created instances; the private half is the control plane's only access to them) is generated automatically at plugin activation (activate.php → ProvisioningSetup::ensureSshKey()), defaulting to {site root}/config/provisioning_key. The page's Generate provisioning key button runs the same idempotent action for control planes activated before the key existed; an existing key or custom path is never overwritten.

Customer-Cloud Fulfillment

The buyer connects their Linode account once at /profile/server_manager/connect_cloud (the Connect page — also the re-connect page if a grant is later revoked). The grant flows through the platform OAuth2 core (provider linode, consumer purpose customer_cloud, scope linodes:read_write only — no account or billing access). Tokens are SecretBox-encrypted on the buyer's CustomerCloudAccount row.

Each provision is a CustomerCloudProvision row that the Provision Customer Cloud scheduled task advances:

pending_connect → (grant arrives) → ready → instance created on the connected account → booting → running + IP → ManagedNode + install_node job → installing → done (or failed, which alerts the ops address).

Provisions have two origins (cvp_origin):

• order — created by a customer-cloud purchase. Starts at pending_connect, installs fresh + Docker, and sends the buyer welcome email on completion (the order-item linkage drives it).
• admin — created by the Install New Node form's cloud-instance target. Starts at ready (the admin picked an already-connected account), carries its install parameters on the row (cvp_docker_mode, cvp_install_mode, cvp_source_node_id, cvp_backup_source, cvp_port, cvp_sitename), and sends no welcome email. The row belongs to the grant owner (cvp_usr_user_id), so a stale grant is re-connectable by the person who can actually re-consent.

{"access_key": "...", "secret_key": "...", "region": "...", "endpoint": "..."}

Two columns hold two keys: bkt_credentials is the main (delete-capable) credential the control plane itself uses, and bkt_node_credentials optionally holds a write-only key handed to nodes instead (see The node may write to the shelf but never erase it). Both are SecretBox-sealed at rest.

A persisted job command never contains a credential — it carries a placeholder token that the agent resolves in memory immediately before the step runs: __SM_CREDS_<target_id>__ for the main slot, __SM_NODE_CREDS_<target_id>__ for the node slot. The builder decides at build time which token a step gets: node-side uploads carry the node token whenever the node slot is filled, while node-side downloads and cloud deletes always carry the main token, because those need the read and delete capability a write-only key deliberately lacks. The agent resolves exactly the slot the token names and never falls back to the other, so a job built against a since-emptied slot fails visibly rather than running with a more powerful key than intended.

For node-side operations (upload, delete, download), the resolved credentials are embedded into a self-contained PHP script that is piped to the node via a heredoc'd php -- invocation — never written to a file on the node and never visible in process listings as positional arguments. The S3Signer.php and node_uploader.php source is composed at job-build time by JobCommandBuilder::build_node_uploader_script().

Because the script is composed from the control plane's own copy of those two files, changes to the signer or the uploader reach every node on its next job — there is no agent release or node upgrade in the loop.

Transient Failures

A storage provider that answers a request with a 5xx does not fail the job. S3Signer::request() retries — MAX_ATTEMPTS tries, exponential backoff with jitter — for the failures that are worth another go: 5xx, 429, 408, and the transport errors that mean the connection died rather than the request being wrong. A deterministic error (403 signature, 404, 400) is returned immediately; retrying it would only burn the budget and bury the message. S3Signer::is_retryable() is the whole policy and is pure, so the classification is testable without a network.

Retrying is safe because every request the class makes is idempotent: a PUT overwrites its key (a single PUT, never multipart, so no orphaned parts survive a failure), and GET/DELETE/list have no cumulative effect.

Two bounds keep a retry from doing harm:

• Wall clock. Total time is capped at one attempt's timeout plus RETRY_WINDOW_SECONDS. An attempt that burns the entire transfer timeout leaves no room for another — the right answer, since a transfer that cannot finish in an hour will not finish on the second try. Job steps that shell out to the uploader take their own timeout from S3Signer::transfer_budget_seconds(), so the agent can never kill a transfer part-way through a retry.
• Replay. A retried upload rewinds the body stream before resending. curl consumes the stream on the first attempt while CURLOPT_INFILESIZE still claims the full length, so a retry without the rewind sends nothing and then blocks until the timeout — a hang rather than an error. A stream that cannot seek is not retried at all, rather than being sent truncated.

• The recovery keypair is generated with maintenance_scripts/sysadmin_tools/escrow_keypair.php (standalone PHP + sodium, no platform bootstrap, so it runs during disaster recovery when the platform is gone), or in the browser from the setup panel.
• The public key is stored in the core backup_recovery_public_key setting.
• Minting and sealing happen on the node, in maintenance_scripts/sysadmin_tools/backup_envelope.php. Only the recovery public key travels in the job step, so a ManagementJob row — which persists forever — carries nothing that can open anything.
• The plaintext key exists only as a 0600 file for the length of the run and is shredded by the step that seals the envelope to the finished archive.

Nothing is left on the node. The credential is substituted into the step by the agent at run time and never written to a job row or a node's database. The recovery key travels as a public key, which seals but cannot open. So a node holds no key to anyone's backups but its own, and a node that leaves this fleet takes nothing with it.

The node may write to the shelf but never erase it

A backup target holds two credential slots. The main credential (bkt_credentials) is the control plane's own — it lists, prunes and downloads. The node credential (bkt_node_credentials, on the target edit form) is an optional second key created write-only — writeFiles without deleteFiles on B2, s3:PutObject without s3:DeleteObject on S3. When it is set, that is the key nodes are handed during a run: a node can add its archives and remove nothing. When no node credential is configured, nodes receive the main key — functional, but a compromised node then briefly holds a key that could erase the shelf, so a fleet target wants the node slot filled.

FleetBackupRetention prunes from here, with the delete-capable main credential that never leaves this machine. A credential that can delete is a credential that can erase the fleet's backups, which is the first move of any ransomware worth the name and the exact thing these copies exist to survive.

Pruning is driven by a bucket listing, which is the opposite of what a site does for its own backups, and correct only here: this control plane defined the whole {prefix}/{slug}/manager/ path, knows every slug under it, and is the only party that can delete from it. It is also stricter — it keeps the newest N sets of objects that actually exist, so a run that failed part-way can never be counted as a restore point. Chains are grouped by their directory, so they are kept or deleted whole by construction.

Two provider notes:

• Linode Object Storage keys are read-only or read-write per bucket with no separate delete capability, so write-without-delete cannot be expressed there. B2 and S3 both express it cleanly.
• A chain rewrites manifest.json every run. That is a PUT over an existing key, which write-only permits, but on B2 it leaves superseded versions the node cannot remove. Give the fleet bucket a lifecycle rule keeping only the current version.

FleetBackupPolicy resolves each node's schedule: the declared fleet settings, then that node's own mgn_backup_policy overrides. The fleet default is enabled. That default is what stops a newly managed node being forgotten — there is deliberately no detector for "nobody has decided about this node", because a node nobody decided about is backed up anyway.

The node detail Backups tab edits the policy, as one of three positions:

• Fleet default stores nothing, so the node follows the fleet settings — including future changes to them.
• A schedule of its own stores the full field set (frequency, window, mode, retention, full interval), frozen against the fleet default: a value the operator saw and saved is a value they chose.
• Off stores exactly that decision, which is what lets the dashboard treat a node without fleet backups as somebody's choice rather than a gap.

{
    "steps": [
        {"type": "ssh", "label": "Check disk usage", "cmd": "df -h /"},
        {"type": "ssh", "label": "Check memory", "cmd": "free -m"},
        {"type": "ssh", "label": "Check uptime", "cmd": "uptime"},
        {"type": "ssh", "label": "Check PostgreSQL", "cmd": "pg_isready"},
        {"type": "ssh", "label": "Check Joinery version",
         "cmd": "grep VERSION /var/www/html/site/public_html/includes/version.php"},
        {"type": "ssh", "label": "Container stats",
         "cmd": "docker stats --no-stream empoweredhealthtn", "on_host": true}
    ]
}

A job's steps form two phases. Steps without the teardown flag are the main phase: they run in order, and a hard failure (no continue_on_error) stops the phase and determines the job's outcome. Steps flagged "teardown": true are the teardown phase: they run on every exit path — success, mid-job failure, or none-of-the-main-steps-ran — so the scratch files a job creates (dumps, staged archives, unpacked installers) are removed even when the job aborts on a shared production host.

Teardown semantics:

• Teardown never changes the outcome. A failed job stays failed with the original failing step in mjb_error_message; a teardown step erroring is logged under the === Teardown === output header and ignored.
• Teardown runs before the terminal status is written. The job stays running while teardown executes, so the per-node concurrency lock holds (no re-run can race the deletions) and the job detail view keeps streaming.
• Progress counts main steps only. mjb_total_steps excludes teardown steps and teardown output never advances mjb_current_step.
• Stale-job replay. Jobs force-failed at agent startup (left running by a crash or restart) get their teardown steps replayed from mjb_commands — safe because every teardown command is an idempotent rm on a per-job path.
• Placement. Builders put teardown steps at the tail of the array, after every main step, and keep continue_on_error on them. An agent that ignores the flag runs the array sequentially, so tail placement makes the steps plain trailing cleanup there — correct, just not failure-proof.

// plugins/server_manager/includes/JobCommandBuilder.php
public static function build_restart_apache($node) {
    return [
        ['type' => 'ssh', 'label' => 'Restart Apache',
         'cmd' => 'systemctl restart apache2'],
        ['type' => 'ssh', 'label' => 'Verify Apache status',
         'cmd' => 'systemctl is-active apache2'],
    ];
}

$steps = JobCommandBuilder::build_restart_apache($node);
$job = ManagementJob::createJob($node->key, 'restart_apache', $steps, null, $session->get_user_id());
header('Location: /admin/server_manager/job_detail?job_id=' . $job->key);

$job = ManagementJob::createJob(
    $node_id,               // target node ID (or null for local)
    'backup_database',      // job type label
    $steps,                 // array of step dicts from JobCommandBuilder
    ['encryption' => true], // parameters (stored for reference/re-run)
    $session->get_user_id() // who triggered it
);

Per-node IP pinning. http_status checks pin the request to the node's own mgn_host IP (CURLOPT_RESOLVE, SNI/Host preserved) when mgn_host is an IP literal and appears in the site hostname's public A records (DnsResolver::getA()) — the same directly-exposed guard check_cert_expiry() uses. A node behind a shared or round-robin hostname (e.g. two DNS servers sharing dns.scrolldaddy.app via dual A records) is therefore checked as itself, not whichever A record DNS happens to return — otherwise a single dead node hides behind its live partner. The guard matters: pinning a Cloudflare-fronted node to its origin IP would bypass the edge and hit the Apache default-vhost fallback cert, failing SNI/cert validation and reporting a false down — so nodes that aren't directly exposed are checked unpinned, through the public hostname, same as before pinning existed.

Check types (extensible via a single dispatch switch in RunNodeUptimeChecks::run_check()):

It reads the served certificate over the wire (stream_socket_client on ssl://mgn_host:443, capture_peer_cert, SNI = the site hostname), so it sees whatever the cert manager actually serves — Caddy, certbot, anything. Validity is deliberately not verified, so the notAfter of an already-expired or near-expiry cert is still readable.

It self-limits to self-renewed, directly-exposed nodes via two guards, because probing an origin behind a CDN returns a misleading cert:

1. Directly-exposed: mgn_host must appear in the public A records for the site hostname (DnsResolver::getA()). A Cloudflare-fronted hostname resolves to Cloudflare, not the origin — those are skipped, because Cloudflare renews that edge cert (not our failure surface).
2. SAN match: the served cert's CN/SANs must actually cover the hostname. A shared default-vhost fallback cert (SAN mismatch) is ignored — there is nothing dedicated to monitor.

1. Database restores replace -- A database restore leaves the target equal to the snapshot. Every restore site (restore_database, both copy jobs, the from-backup install) verifies the archive with gunzip -t before anything is destroyed, drops and recreates the public schema so target-only objects are removed too, and loads with psql -v ON_ERROR_STOP=1 so the first load error fails the job instead of completing a partial restore. Dumps are plain pg_dump snapshots -- the restore step owns the replacement guarantee, so it holds for any file it is fed. Job-internal dumps (copy jobs, install clone) add --no-owner --no-acl because they are restored as the target* site's DB user; backup files restore onto the site that made them, where the role matches.

1. Per-node concurrency lock -- The agent skips jobs if another job is already running on the same node, preventing conflicts.

1. Stale job recovery -- On agent startup, any orphaned running jobs are marked failed with a descriptive message.

1. Step timeout -- 30-minute default per step, overridable. On timeout, the SSH session is killed.

1. Single-threaded agent -- One job at a time. Queued jobs run sequentially.

1. Remote credentials at runtime -- Database credentials for backup/copy/restore are extracted from each node's Globalvars_site.php at execution time, never stored on the control plane.

API Actions

The dashboard's page JS calls these POST /api/v1/action/server_manager/{name} actions with the browser-session credential (superadmin only, floor 10) and reads the response envelope's data. The full set: probe_api, job_status, discover_nodes, backup_actions, refresh_node_status, add_discovered_nodes.

server_manager/job_status

Polled by the job detail page for live output.

Parameters:

• job_id (int) -- job to query
• output_offset (int) -- character position; only new output since this offset is returned

{
    "success": true,
    "status": "running",
    "new_output": "=== [Step 2/5] Check memory ===\n...",
    "output_offset": 1234,
    "current_step": 2,
    "total_steps": 5,
    "error_message": null
}

The UI polls every 2 seconds while a job is running and stops when status is completed or failed.

server_manager/backup_actions

Used by the backup browser on the Backups tab.

server_manager/discover_nodes

Used by the auto-detect panel on the Add Node page. Creates and polls discover_nodes jobs.

Troubleshooting

Agent shows Offline on dashboard

• Check the agent is running: sudo systemctl status joinery-agent
• Check logs: journalctl -u joinery-agent -f
• Verify DB credentials in /etc/joinery-agent/joinery-agent.env match those in Globalvars_site.php

• Agent is not running or can't connect to the database
• Another job is running on the same node (per-node lock)

• Verify SSH key path on the node record matches an actual key file
• Test manually: ssh -i /path/to/key root@host "echo ok"
• For container nodes, verify the container name is correct

• The agent crashed or was restarted mid-job. Check journalctl for the crash cause.
• The partially-completed job should be inspected manually. Use Re-run to retry.

File Reference

Plugin (plugins/server_manager/)

Go Agent (/home/user1/joinery-agent/)