Error Handling and Problem Reports

How an error becomes something a person can report, and how the report leaves the site without carrying anything private.

The error handler

ErrorManager (includes/ErrorHandler.php) is the uncaught-exception handler for every request. For each error it:

  1. records it through two loggers: DatabaseErrorLogger saves an err_general_errors row (GeneralError::logError()), and FileErrorLogger writes one JSON line to logs/error.log;
  2. renders the response for the kind of request, with one handler each (includes/ErrorClasses.php):
RequestHandlerResponse
Web pageWebErrorHandlerAn HTML error page
/admin pageAdminErrorHandlerThe admin variant of the page
/api/ApiErrorHandlerThe /api/v1 error envelope (docs/api.md)
AJAX (/ajax/, JSON requests)AjaxErrorHandler{success: false, error: {message, type}}
Command lineCliErrorHandlerMessage, file, line and trace on stderr
Recording comes first, so the response can refer to the row it was saved as.

What a person is told: a Displayable* exception's own message, a displayable BaseException's user message, the raw message when the show_errors setting is on, and a generic sentence otherwise. The page never claims that anyone was notified.

An exception that escapes an API action's logic file is caught by ApiLogicEndpoint, recorded the same way (ErrorReference::log()), and answered with the same user-safe message rules. An exception marked NoLog is an expected refusal and is not recorded.

Error references

A recorded error has a reference: its err_general_errors row id and a grouping hash. ErrorReference (in includes/ErrorHandler.php, beside the classes that record errors) holds the reference for the request and builds everything that points at it.

The grouping hash is the same for the same fault on any site: the error code, the message with its digits folded to N, and the file and line relative to public_html. Every part is stored on the row, so a hash computed later from the row (ErrorReference::hashForRow()) matches the one computed when the error was thrown.

Where the reference appears:

SurfaceWhat carries it
Web and admin error pagesA Report this problem button and an "Error reference: N" line
/api/v1 error envelopeserror_ref: {id, hash, report_url} (api_error(), a logic action's error, ApiErrorHandler)
AJAX error envelopesThe same error_ref
Page JavaScripterr.errorRef on a rejected joineryApi call; joineryApi.reportLink(err) makes the link
Error flash messagesA Report a problem link after every MESSAGE_ERROR alert; DisplayMessage::$error_ref names the row when there is one
The 404 pageA Report a problem button beside Contact Support
Report links are shown to signed-in members only. A guest cannot send a report, so a link would lead only to a sign-in page. An error with no recorded row (a flash message, a 404) links with the page and the message the member saw instead of a row id.

A page script's notice helper takes the rejected error as a second argument and appends joineryApi.reportLink(err) when it returns an element, keeping the notice up long enough to reach the link. drive.js, vault-manager.js, messenger.js, mailbox_reader.js and the calendar page do this.

Problem reports

/report_problem (views/report_problem.php, logic/report_problem_logic.php) is where the links lead. A guest is sent to sign in and brought back.

The page shows, top to bottom:

  1. Where the report goes: the host of the upgrade_source setting, the site this software is upgraded from. When the operator has switched sending off, it says the report stays on this site.
  2. The form: a description (required, up to 5,000 characters) and an optional screenshot (PNG, JPEG, WebP or GIF, up to 5 MB). A line above the description asks for the steps, what was expected and what happened, the page, and how often it happens.
  3. What will be sent: the bundle, built live for this member and this error, shown in full. The reporter reads what leaves before deciding to send it.
The form posts to the report_problem_submit API action (browser session only), which rebuilds the bundle from the same inputs, saves it with the description and image in prr_problem_reports (ProblemReport), and tries one send at once. A member may send at most ten reports an hour.

What a report contains

ProblemReportBundle (includes/ProblemReportBundle.php) is an explicit list of named sections with fixed shapes. There is no free-form dump.

SectionContentsIn whose report
SiteHost, platform version, schema version, theme, days since installEveryone's
RequestThe path where it happened, with query values masked except plain numbers; web, admin or API; whether in the app; browser, OS and time zoneEveryone's
Who is reportingUser id, permission level, whether an administrator is acting as this userEveryone's
ErrorThe recorded row's kind, code, file and line, message, the first five stack frames with their arguments dropped, the hash, the time, and how many errors at the same place in seven days. With no row, the message the member sawEveryone's
Recent errorsUp to 40 PHP error lines from logs/error.log, newest firstOperators' only
RuntimePHP, PostgreSQL, OS and web server versions; whether the agent is installed and connectedOperators' only
PluginsEvery installed plugin, its version, and whether it is activeOperators' only
SettingsNames of the settings changed from their defaults (never a value); how many credentials are set; how many sealed secrets exist and how many no longer openOperators' only
HealthDisk free, memory used, load, minutes since scheduled tasks last ranOperators' only
The site-wide sections go only into a report from an operator (permission 9, the level that reads the error log). Any member may report, and the reporter reads every line of the bundle, so a member's report must not show them other members' errors or the site's configuration. For the same reason a member can attach only an error recorded on their own account; a reference to anyone else's error is dropped and the report falls back to the message they saw.

Never included: member names or addresses, email bodies or subjects, file names from Drive or mail, form submissions, request bodies, session contents, IP addresses, any setting value, anything under config/, anything sealed. Building a bundle opens no sealed content: settings are compared as stored, never decrypted.

Masking. Every text value passes through LogRedactor::text() and has the site's own directory stripped from paths. LogRedactor (includes/LogRedactor.php) is the PHP copy of the agent's redact package, rule for rule: credential values by key name and shape, URL passwords, bearer tokens, the personal half of email addresses (<email>@domain), IP literals (<ip>) and opaque tokens (<token>). It masks shapes, so a name inside an exception message passes; that is why the reporter sees everything first. tests/unit/agent_redactor_parity_test.php holds its key list equal to the agent's, and tests/unit/log_redactor_test.php runs the agent's own test cases against it. SmSecretRedactor (server manager) masks with LogRedactor::secrets().

Sending

ProblemReport::send() posts the bundle, description and image as multipart form data to {upgrade_source}/api/v1/action/bug_reports/report_submit through SafeHttpClient (no redirects, 20 s, 64 KiB answer). The receiving end is the bug_reports plugin (its overview).

StatusMeaning
queuedSaved, not sent yet
sentThe upgrade source accepted it; its report id is kept
failedA try did not succeed; the reason is kept
keptSending is switched off; the report stays on this site
The hourly ProblemReportSend task retries queued and failed reports, up to five tries in all, and sends automatic reports and their counts. System › Problem Reports (/admin/admin_problem_reports, permission 9) lists every report with its status and reason (automatic ones with how often the error happened), opens one to its full bundle, and has a Send now button for one that has not gone through.

Automatic reports

With problem_reports_auto_send on (and sending on), the site reports unexpected errors itself. ErrorReference::recorded() runs after every recorded error on both paths and calls ProblemReport::noteError(), which:

  • skips errors that are people meeting a wall rather than bugs: a message marked safe to show (the Displayable* family, a displayable BaseException), a permission refusal, a sign-in requirement, a validation failure (ProblemReport::isUnexpected());
  • works out the fault's key, ProblemReportBundle::fingerprint(): the error's kind, the file it was thrown in, and the files and function names of its first five stack frames, with line numbers dropped. With no frames, the message with numbers, quoted text and long hex runs blanked stands in;
  • keeps one automatic report per fault and version, adding 1 to its count (prr_occurrences) on every recurrence, and starts at most 20 new ones a day;
  • never sends from the failing request and never throws. While the process holds sealed content it only counts, since a new report's text is a long write the sealed-content guard refuses.
The hourly ProblemReportSend task sends a new automatic report, then sends it again whenever it has counted recurrences the receiver has not heard; each send carries occurrences, the recurrences since the last one (prr_occurrences_sent records what was heard).

Nobody reads an automatic report before it goes, so it carries less than an operator's (ProblemReportBundle::automatic()): site, runtime, plugins, settings and health as usual; the request with its path masked to its shape (a segment that is not a number or a lowercase word becomes …) and no time zone; the error from its recorded row plus the exception class, with quoted text in the message masked. No reporter section and no log lines. It is built from the saved row, so an error recorded while sealed content was open carries only the row's withheld reference.

Settings

SettingDefaultMeaning
problem_reports_sendonSend reports to the upgrade source. Off keeps them on this site.
problem_reports_auto_sendoffReport unexpected errors automatically, with no one reviewing the report first. Needs sending on.
problem_reports_retention_days90Days to keep reports and their images; the daily retention sweep deletes older ones. 0 keeps them.
Both are under Settings › Problem reports. The menu entry Report a problem is in the signed-in user menu, next to Admin Help, and reaches the mobile apps through the same menu.