Deploy and Upgrade Systems

• Independent versioning of themes and plugins
• Selective updates (update core without touching themes)
• Smaller download sizes for incremental updates
• Third-party theme/plugin distribution

Quick Reference

install.sh

Location: /var/www/html/joinerytest/maintenance_scripts/install_tools/install.sh

Universal installer for Docker and bare-metal deployments. Supports --themes flag to download published themes/plugins from the upgrade server after site creation (extensions whose manifests have included_in_publish: true).

Full documentation: Installation Guide

build_dev_from_source.sh

Location: /var/www/html/joinerytest/maintenance_scripts/install_tools/build_dev_from_source.sh

> Note: This script is functional but not recommended for production. Use upgrade.php for production deployments. build_dev_from_source.sh is suitable for development environments where git-based deployment is convenient.

# Basic deployment
./build_dev_from_source.sh joinerytest

# Verbose mode (recommended)
./build_dev_from_source.sh joinerytest --verbose

# Disable auto-rollback for debugging
./build_dev_from_source.sh joinerytest --norollback

# Manual rollback
./build_dev_from_source.sh joinerytest --rollback

Features:

• Git-based deployment from repository
• Pre-deployment validation (PHP syntax, plugin loading, bootstrap tests)
• Automatic rollback on failure (trap-based)
• Preserves extensions marked receives_upgrades: false
• Composer integration and database migrations

upgrade.php

Location: /utils/upgrade.php

Web Usage:

# Check for upgrades
https://yoursite.com/utils/upgrade?serve-upgrade=1

# Perform upgrade (verbose)
https://yoursite.com/utils/upgrade?verbose=1

CLI Usage:

# Basic upgrade
php /var/www/html/joinerytest/public_html/utils/upgrade.php

# Verbose mode
php /var/www/html/joinerytest/public_html/utils/upgrade.php --verbose

Features:

• Downloads packages from upgrade server (configured via upgrade_source setting)
• Downloads core, themes, and plugins as separate archives
• Pre-deployment validation via DeploymentHelper
• Preserves extensions marked receives_upgrades: false
• Enhanced rollback (preserves failed deployments with timestamps)
• Database migrations and composer integration
• Declared-dependency install — after the file swap, installs any PHP extension the new code declares (root composer.json ext-* + plugin requires.extensions, resolved by utils/list_dependencies.php --apt) and reloads web PHP. Needs root (Docker docker exec has it; a non-root run degrades to a warning naming the manual apt-get install). Then runs every active plugin's declared host_installer via _plugin_installers_start.sh so new host requirements land with the deploy.
• Graceful handling of missing archives — if a theme or plugin archive returns 404, the upgrade warns and skips it instead of aborting. The core upgrade and all other themes/plugins proceed normally. A summary of skipped items is shown at the end.
• Post-deploy smoke test — after the new code is in place and migrations have run, the deploy test tier runs against it. A failure restores public_html_last and preserves the broken tree for diagnosis. This is the first thing in the pipeline that reads a line of the code being installed: publish_upgrade.php builds its archive from whatever is on the publisher's disk at that moment, half-finished edits included. The tier takes a couple of seconds, is entirely reads, and asks only whether the code runs on this machine — every deployable PHP file compiles, the core classes load, the database answers, the declarative manifests parse, and the site returns a page over HTTP. See The deploy tier below for why it is not safe. The rollback returns the code but not the schema, because migrations ran first; schema changes are additive so the previous code normally runs against them, but the output says plainly that this is a recovery rather than a clean undo, and the node should be upgraded forward rather than left there.
• Site-root maintenance_scripts/ — the core archive carries install_tools/ and sysadmin_tools/ alongside public_html/, and the upgrade syncs them into the site root after the deploy swap and before fix_permissions.sh runs, so a release applies as one piece rather than with its own tooling a version behind. The sync compares by content (rsync --checksum), because the staged files carry the publishing box's timestamps while the node keeps its own and a same-size edit can otherwise look unchanged. It does not delete: a node can legitimately hold scripts the archive does not ship, so files absent from a release are left in place. *.sh are made executable afterwards. A failed sync is a warning, not an abort — public_html is already live — and it says explicitly that the node's backup, restore and permission scripts are still the previous version.

Deployment self-update (read before editing upgrade.php)

A release can change the deployment tooling itself, so before deploying anything the upgrade compares four files between the staged archive and the live site:

utils/upgrade.php
utils/update_database.php
includes/DatabaseUpdater.php
includes/DeploymentHelper.php

Any that differ are copied to live immediately and the pipeline re-executes from the start, so a release is applied by its own tooling rather than by the previous version's.

The comparison happens twice, and both passes take all four together:

1. Early — straight out of the downloaded tarball, before staging is cleared or anything is extracted. A bug anywhere in the pipeline is then one upgrade attempt away from fixing itself.
2. Post-extract — again from the fully staged tree, so a file the tarball listing missed is still caught.

The rule this imposes: during that window the new upgrade.php is running against the old core — every file outside the set of four is still the previous release. Anything those four call must therefore already exist on the oldest site expected to upgrade, or travel inside that same set.

publish_upgrade.php

Location: plugins/server_manager/includes/publish_upgrade.php Access: Requires the Server Manager plugin to be active. Superadmin only (permission level 10).

Preferred usage: Use the Publish Upgrade form on the Server Manager dashboard (/admin/server_manager). Enter release notes and submit — the plugin creates a job that builds all archives.

CLI usage:

php plugins/server_manager/includes/publish_upgrade.php "release notes here"
# Auto-detects the next version number

> Note: The legacy location utils/publish_upgrade.php still exists for backward compatibility during the Phase 1 transition. It will be removed in a future release once all remote nodes have been upgraded.

Features:

• Creates separate archives for core, themes, and plugins
• Core archive excludes theme/ and plugins/ directories
• Each theme/plugin with included_in_publish: true gets its own versioned archive
• Prevents overwriting existing versions
• Automatic cleanup on failure
• Registers upgrade in stg_upgrades table

static_files/
├── joinery-core-3.26.upg.zip        # Core application
├── theme-falcon-3.26.upg.zip        # Falcon theme
├── theme-default-3.26.upg.zip       # Default theme
├── plugin-bookings-3.26.upg.zip     # Bookings plugin
├── plugin-controld-3.26.upg.zip     # ControlD plugin
└── ...

Component version integrity:

Each publish records a per-release snapshot of every published component's (version, tree_hash) in upg_upgrades.upg_component_state (JSON, keyed by themes/plugins). The snapshot of the most recent release row that carries one is the baseline for change detection; rows without a parseable snapshot are skipped, so an aborted publish never poisons the baseline. When no prior snapshot exists, the run re-baselines — records everything, bumps nothing.

The tree_hash is a deterministic, git-independent SHA-256 of the component's working tree (.git/ and .gitignore excluded). The manifest (theme.json / plugin.json) is hashed with its version member removed, so the hash measures content-minus-version.

Per component, before archiving, the publisher applies a four-way decision against the baseline entry:

1. No baseline entry — first publish of this component: record as-is, no bump.
2. Manifest version higher — author bumped deliberately: respect and record.
3. Manifest version lower — record and archive as-is, with a warning line in the publish summary naming the component and both versions. Not aborted: a backward component version has no destructive effect at publish time, and any resulting depends violation is caught fail-closed at activation.
4. Manifest version equal — compare hashes. Equal → unchanged, carry the entry forward. Differ → auto patch-bump the manifest's version (targeted string edit, no reformatting) and archive under the new version.

Publish log:

Every publish writes the full text of its run to {site root}/logs/publish/publish-{version}-{YmdHis}.log, outside public_html so it is not web-served. The log is written by a shutdown handler rather than at the end of a successful run, so a publish that exits early — a refusal, a die on a missing prerequisite, a fatal error — is captured too. Those are the runs worth reading: a stage that reports a problem through the publish summary and continues leaves no other trace once the page or terminal is gone. The header records the version, whether the run was CLI or web, and which account it ran as; a fatal error is appended to the end. The newest 20 logs are kept and older ones removed on each publish. A successful run names its log path as the last line of the summary.

Publish refusals:

Two conditions stop a publish before anything is written — no VERSION change, no archives, no release row:

1. The requested version is lower than the one already in the VERSION file.
2. This box holds agent source newer than the bundled agent artifact and the rebuild fails (see Server Manager).

publish_theme.php

Location: plugins/server_manager/includes/publish_theme.php Access: Requires the Server Manager plugin to be active. Superadmin only (permission level 10).

# Publish a single theme
https://yoursite.com/admin/server_manager/publish_theme?type=theme&name=falcon&version=1.0.0

# Publish a single plugin
https://yoursite.com/admin/server_manager/publish_theme?type=plugin&name=bookings&version=2.1.0

# List available themes (used by marketplace and upgrade.php)
https://yoursite.com/admin/server_manager/publish_theme?list=themes

> Note: The legacy location utils/publish_theme.php still exists for backward compatibility during the Phase 1 transition.

Features:

• Publishes individual themes or plugins independently of core
• Allows different versioning for themes/plugins vs core
• Useful for third-party theme/plugin distribution
• Validates theme.json/plugin.json exists before packaging
• Serves catalog listings for the marketplace and upgrade.php

How It Works

Deployment Flow

1. Download/extract to staging directory
2. DeploymentHelper validates:
   - PHP syntax on all files
   - Plugin class loading
   - Bootstrap/core components
3. DeploymentHelper preserves extensions marked `receives_upgrades: false`
4. Backup current installation to public_html_last/
5. Deploy staged files to public_html/
6. Sync staged maintenance_scripts/ into the site root
7. Run database migrations (update_database.php)
8. Run composer_install_if_needed.php
9. Fix permissions (www-data:user1, 775)

If ANY step fails → Automatic rollback

Directory Structure

/var/www/html/{site}/
├── public_html/              # Current live installation
├── public_html_last/         # Backup (for rollback)
├── public_html_stage/        # Staging area for validation
├── public_html_failed_*/     # Preserved failed deployments (timestamped)
├── static_files/             # Published upgrade packages
│   ├── joinery-core-X.XX.upg.zip
│   ├── theme-THEMENAME-X.XX.upg.zip
│   └── plugin-PLUGINNAME-X.XX.upg.zip
└── uploads/upgrades/         # Downloaded packages (client sites)

Archive Naming Convention:

• joinery-core-X.XX.upg.zip - Core application (no themes/plugins)
• theme-{name}-X.XX.upg.zip - Individual theme archive
• plugin-{name}-X.XX.upg.zip - Individual plugin archive

Extension Distribution Flags

Themes and plugins carry two independent boolean flags in their manifests (theme.json / plugin.json) that control distribution. Both default to true when missing, and they govern different sides of the pipeline:

Example manifest (theme.json or plugin.json):

{
    "name": "controld",
    "version": "2.1.0",
    "description": "ControlD DNS management plugin",
    "receives_upgrades": true,
    "included_in_publish": true
}

update_database Behavior

Advisory Lock

update_database.php uses a PostgreSQL advisory lock (pg_try_advisory_lock(99999)) to prevent concurrent runs. If a second process tries to run while one is already in progress, it exits immediately with "already running." The lock is released automatically when the database connection closes.

Halt on Migration Failure

Migrations stop on the first failure — subsequent migrations are skipped. Fix the failing migration and re-run update_database.php to continue.

Migration test Semantics

Each migration has an optional test SQL query that returns a row with a count column. The runner interprets it as:

• count > 0 → migration is skipped (already applied)
• count = 0 → migration runs

// Insert a row — skip if it already exists
$migration['test'] = "SELECT count(1) as count FROM emt_email_templates WHERE emt_name = 'my_template'";
$migration['migration_sql'] = "INSERT INTO emt_email_templates (emt_name, emt_body) VALUES ('my_template', '...')";

> Note: do not use migrations to seed stg_settings rows. Setting names and defaults are declarative — see "Declarative Settings (no migration)" below.

The same principle applies to core admin/profile menu rows (declared in public_html/admin_menus.json) and plugin menu rows (declared in plugin.json under adminMenu / profileMenu).

Column Defaults

> An updated agent template is available. [Compare] [Switch to new version]

• Compare opens a read-only side-by-side view of the current content vs the candidate.
• Switch to new version moves target filenames from the active row to the candidate, archives the previously-active row (name prefixed Archived — , target filenames cleared), and writes the new content to disk.

DeploymentHelper API

Validation:

DeploymentHelper::validatePHPSyntax($directory, $verbose)
DeploymentHelper::testPluginLoading($stage_dir, $verbose)
DeploymentHelper::testBootstrap($stage_dir, $verbose)

Theme/Plugin Preservation:

DeploymentHelper::preserveExtensionsAcrossDeploy($stage_dir, $backup_dir, $verbose)

Rollback:

DeploymentHelper::performRollback($target_site, $preserve_failed, $verbose)

All methods return structured arrays with success status, errors, and detailed results.

Common Issues

Permission Errors:

sudo chown -R www-data:user1 /var/www/html/joinerytest/public_html
sudo chmod -R 775 /var/www/html/joinerytest/public_html

Validation Failures:

• Failed deployment preserved in public_html_failed_* directory
• Inspect for syntax errors or missing dependencies
• Fix and redeploy

• Check manifest has "receives_upgrades": false
• Restore from public_html_last/ if needed

• Check public_html_last/ exists
• Manually restore from backup
• Fix permissions after restore

The Joinery vhost bakes a RewriteCond %{HTTP:CF-Visitor} !"scheme":"https" guard into the redirect rule, so it cannot loop in any CF SSL mode (or with no CF at all). The admin Settings page also surfaces a yellow warning banner whenever the platform detects it's being served from Flexible-mode CF — so the misconfig is visible without needing to read logs.

Apache Vhost

Every site installed by install.sh has the same Apache vhost shape, regardless of whether it sits behind Cloudflare, behind another CDN, or is exposed directly to the public internet, and regardless of whether the origin has its own TLS certificate.

Shape. Defined by template files in maintenance_scripts/install_tools/ (single source of truth per deployment mode) — default_proxy_vhost.conf for Docker reverse-proxy sites, default_virtualhost.conf for bare-metal sites. install.sh write_universal_vhost substitutes the placeholders ({{DOMAIN_NAME}}, {{SITE_NAME}}, {{PORT}}, {{SERVER_IP}}) and writes the result to /etc/apache2/sites-available/${sitename}.conf.

• Port 80 — proxies/serves traffic, plus a RewriteRule that redirects to HTTPS. The redirect carries a CF-Visitor guard so it cannot loop under CF Flexible mode and is a no-op when no CF is in front.
• Port 443 — wrapped in <IfFile /etc/letsencrypt/live/${domain}/fullchain.pem>. Apache evaluates <IfFile> at config-parse time: if the cert exists, the :443 vhost activates; if not, Apache silently skips the block. Sites with no origin cert just serve port 80, and whatever's in front (Cloudflare etc.) handles TLS at the edge.

1. Domain resolves to this server → Let's Encrypt HTTP-01 challenge via certbot --apache --no-redirect.
2. Domain resolves elsewhere (Cloudflare, other CDN) and a matching DNS-API credentials file exists at /etc/letsencrypt/<provider>.ini → LE DNS-01 with the matching certbot plugin. Plugin map:

Each plugin reads its credential file in its own standard format; certbot's docs cover the schemas.

1. Neither path produces a cert → install proceeds without origin SSL. The :443 vhost stays dormant via the <IfFile> guard; CF or another front-end handles TLS.

Enabling origin SSL later (e.g. after dropping a CF API token in place to switch a CF zone to Full strict): sudo /var/www/html/<site>/maintenance_scripts/sysadmin_tools/setup_ssl.sh <domain>. The script re-enters the decision tree; the :443 vhost begins serving on the next Apache reload because the <IfFile> guard sees the new cert.

A certificate deferred at install time arrives on its own. When DNS is not pointing at the box yet, arm_ssl_retry.sh installs joinery-ssl-retry@<domain>.timer: every five minutes it checks whether the domain resolves to this server, does nothing until it does, then issues once and disables itself. A self-signed placeholder does not count as done. The DNS gate is what makes an indefinite retry safe — Let's Encrypt counts failed validations, not failed lookups. A restore that lands the site on a different domain re-arms the timer for the new name and disarms the old one.

Rebuilding a site on new hardware

A backup does not carry config/Globalvars_site.php — that file holds the machine's database password and its secret_box_key, and both belong to the machine. So a rebuild is install, then restore onto the installed site:

# 1. On the new box: server prerequisites, then the site
sudo ./install.sh -y server
sudo ./install.sh -y site --bare-metal <site> --password-file=/root/.joinery_postgres_password <domain>

# 2. Bring the backup down and restore onto it
bash maintenance_scripts/sysadmin_tools/restore_project.sh <site> <archive> --domain <domain> --force
#    or, for a chain:
bash maintenance_scripts/sysadmin_tools/restore_chain.sh <site> \
     --artifacts <chain dir> --key-file /tmp/k --domain <domain> --force

The restore reconciles the result to the new box: the domain, the deployment shape (docker vs baremetal), the paths, a regenerated virtualhost, and an armed certificate retry. It refuses if the restored database will not open with the new machine's credentials, rather than leaving that to show up as SQLSTATE[08006] on every page. What it reconciles and why is in Backups.

Cross-shape rebuilds work in both directions with no extra step: a container backup landing on a plain server, or the reverse. Neither installs the virtualhost the backup carries — see the same section.

From the dashboard, the equivalent is a Server Manager install_node job in From-Backup mode (a fresh install plus the source's data, cloned in one job), or a restore_project / restore_chain job against a node that already has a site. Both reconcile identically.

A PostgreSQL major-version jump needs nothing special, upwards. The dump-and-restore path crosses it: a PG 16 dump restores onto PG 18 as an ordinary restore.

Downwards it is refused. A dump carries the syntax of the version that wrote it, so a newer dump cannot load into an older server. The restore reads the version out of the dump header and stops before touching the target database, reporting RESTORE_SERVER_TOO_OLD. The target must run a PostgreSQL at least as new as the source — which for a container target means a base image carrying it.

Configuration

Required settings (in /config/Globalvars_site.php or stg_settings):

The distinction matters beyond the upgrade endpoint: any control that edits a file the upgrade replaces wholesale belongs only on a publishing instance, because on a consuming site the edit is discarded at the next upgrade. Joinery AI's Ship with new installs action is gated this way.

The deploy tier

php tests/run.php deploy is the set upgrade.php runs after a swap. It pulls in no other tier and no other tier pulls it in.

It exists because safe, db and test-db are development gates. They run in a checkout and are entitled to assert things about one: that the full first-party plugin set is present, that the components manifest lists what the repository holds, that maintenance_scripts has the layout the installer expects. A deployed site has none of that and never should — it carries the plugins it uses and no repository around it. Eleven of those suites fail on a production node, for reasons that say nothing whatever about the release.

So deploy asks the only question that means anything on a node: does the code that just landed run on this machine.

• No assumption of a repository. Not the plugin set, not the theme set, not a git checkout, not a sibling directory.
• Reads only. It runs on production, after a swap, with a rollback hanging on the result.
• An unreachable dependency is a SKIP, not a failure. Reverting a working release because a socket would not open is worse than the thing being guarded against.

Where a site's upgrade_source comes from

It is not a decision anyone makes twice. _site_init.sh writes it at install time from the endpoint the install actually fetched its code from — install.sh's UPGRADE_SERVER, which defaults to https://getjoinery.com and is overridden with --upgrade-server=URL. One rule covers both audiences: leave the flag off and the site tracks stable releases; pass it and the site follows wherever it was installed from.

Clones are the exception: UPGRADE_SERVER points at the clone source for the duration of a clone, and that is a peer site rather than a release endpoint, so the cloned database keeps the source's own upgrade_source.

The release channel

getjoinery.com serves stable releases. It is a site rather than a flag on a row: no schema, no promotion state to forget, and it reuses the chaining utils/latest_release already implements.

# on getjoinery, after upgrade.php brings it to 0.8.199
php plugins/server_manager/includes/publish_upgrade.php 0.8.199 "release notes"

The ordering earns something beyond housekeeping: getjoinery is running the code it serves, so a build reaching strangers has at least come up on a real site first.

Prerequisites

• upgrade_source setting must be configured (URL of the upgrade server)
• The upgrade server must have the Server Manager plugin active

Overwrite Protection

• Extensions with receives_upgrades: true (or those without a manifest) can be reinstalled/replaced from the marketplace
• Extensions with receives_upgrades: false are protected — the marketplace refuses to overwrite them

Catalog Endpoint Fields

The publish_theme.php catalog endpoints (?list=themes, ?list=plugins) include:

• name — display name (unchanged for backward compatibility)
• directory_name — filesystem directory name (used for matching and downloads)
• display_name, version, description, author, is_system, included_in_publish

Related Documentation

• CLAUDE.md - System architecture and development guidelines
• Plugin Developer Guide - Plugin development
• Server Manager - Server management, publishing, and backup targets
• Specifications: - /specs/implemented/upgrade_system.md - Feature parity analysis - /specs/implemented/fix_publish_upgrade_system.md - Publish upgrade fixes - /specs/implemented/theme_plugin_distribution_refactor.md - Separate archive distribution - /specs/implemented/server_manager_publish_upgrade.md - Moving publish/upgrade into server_manager plugin - /specs/implemented/upgrade_graceful_theme_download.md - Graceful handling of missing archives