Running it
Queue workers, the scheduler, health checks, backups, and the failure modes that are silent by default.
The two silent failures
Everything else in this page is ordinary operations. These two are worth their own section because neither announces itself.
- The queue worker stopped
Notifications are queued, so no email is delivered at all. No verification, no password reset, no security alert, no invitation. The application looks entirely healthy. You find out when someone cannot reset their password.
- The scheduler stopped
Temporary grants never expire. Expiry reminders stop. Health probes stop, so the operations page freezes on its last known state and looks fine. This is a security failure, not just an operational one.
Each scheduled task records a heartbeat after it runs, so the operations page shows when each last completed rather than only whether the process exists. Alert on a stale heartbeat, not on the process being up.
What the scheduler runs
| Task | Frequency | Purpose |
|---|---|---|
Expire temporary grants | every 5 minutes | Ends access whose time has passed. Records a heartbeat. |
Send expiry reminders | every 15 minutes | Warns about expiring grants, tokens, and trials. Deduplicated, so nothing is sent twice. |
Reconcile stale CLI executions | every 15 minutes | Marks commands whose client stopped reporting as interrupted. |
Record health checks | every 30 minutes | Probes each component so flapping is visible, not just current state. |
Aggregate usage rollups | daily 02:15 | Analytics. |
Prune admin exports | daily 03:00 | Deletes expired data exports so no permanent copy accumulates on disk. |
Process due deletions | hourly | Executes account deletions whose grace period has elapsed. |
Prune expired tokens | daily | Sanctum housekeeping. |
Health checks
/up is a plain liveness endpoint for a load balancer. The real picture is on the control plane's operations page: database, queue depth, failed jobs, scheduler heartbeats, storage, and encryption.
Backups
Three things, and they must not all live in the same place.
- 1
The database
Holds ciphertext, audit logs, and everything else. Nightly at minimum, with point-in-time recovery if your provider offers it.
- 2
The root key
SHORTSPAN_MASTER_KEY, in your secret manager and offline. A backup and its key in the same cloud account is one compromised account away from being neither. - 3
Application configuration
The rest of
.env. Reconstructable with effort, but a bad hour to spend during a restore.
Back up the audit tables with the same care as the secrets. They are frequently the more legally significant of the two.
Key management
php artisan encryption:generate-key # mint a root key
php artisan encryption:rewrap # migrate master keys onto itrewrap rewrites one row per workspace, never touches secrets, and is idempotent and resumable. It reads each row back after writing it before reporting success.
Per-workspace master key rotation is available from the control plane. It re-encrypts that workspace's secrets under a new key as a background job, tracking versions so which key protects what stays answerable.
Operational commands
| Command | Does |
|---|---|
| Generate a root key. |
| Migrate master keys onto the root key. |
| Diff the built-in Cloudflare ranges against the live list. |
| Generate a pipeline publish token. |
| Grant control-plane access. |
| Run the reminder sweep by hand. |
| Render every transactional email to a browsable file. |
| List failed jobs. |
| Retry them. |
The control plane
At /admin, with its own login, its own session store, its own middleware group, and its own audit log. Being a workspace owner grants nothing there.
It covers customers and users, security events and incidents, billing, feature flags and experiments, CLI releases, system health, data governance, and platform settings. Sensitive operations require step-up authentication.
Every administrative action is recorded in a separate audit log from the customer one, with an integrity view for checking the trail has not been tampered with.
What to alert on
Scheduler heartbeat older than fifteen minutes. The highest-value alert here - it means grants are not expiring.
Queue depth growing, or any failed job. No mail is being delivered.
The encryption probe failing. The root key is wrong or missing.
A spike in `429`s on the reveal limiter. Either a fleet operation or a token being drained.
Repeated `403`s from one token. What a stolen credential probing for access looks like.
Failed admin logins. The control plane is the highest-value target you operate.
Certificate expiry. The API refuses plaintext, so an expired certificate is a total outage rather than a warning.
Something here wrong or missing? Tell us.