Backup, update, restore

Local/edge historian plus analytics result parquet defaults to a 100 GiB oldest-first cap. A backup of that live set is a second copy. Do not assume a ~200 GiB host can hold both. Run scripts/openfdd_disk_preflight.py --full-copy before tar of workspace/. Exit 10 means skip the on-box copy (offload it). Exit 20 means free space is under the reserved percent — do not start the update. Any deploy with OPENFDD_TEST_DEPLOY=1 skips the release backup unless OPENFDD_BACKUP_ON_UPDATE=1. The flag is not tied to a site id. Detail: DATA_RETENTION_BUDGET.md.

Local stack (workspace/)

All persistent state for a local Compose stack lives under workspace/. Back it up before any image update. See Build recipes for the recipe/env matrix.

Backup 

cd ~/open-fdd
mkdir -p ~/openfdd-backups/latest
tar -czf ~/openfdd-backups/latest/workspace-full.tgz workspace/

Update 

# 1. back up workspace/ (above)
# 2. re-pull the target tag and recreate the stack
OPENFDD_IMAGE_TAG=sha-<7> ./scripts/openfdd_stack_up.sh standalone
./scripts/openfdd_health_check.sh

Restore 

tar -xzf ~/openfdd-backups/latest/workspace-full.tgz -C ~/open-fdd
./scripts/openfdd_stack_up.sh standalone --no-pull
./scripts/openfdd_health_check.sh

Local FDD + MCP smoke (low-RAM)

Central resolves historian parquet via OPENFDD_STORAGE_URL (preferred) or legacy OPENFDD_PARQUET_ROOT. For /api/fdd/run and MCP openfdd_fdd_* tools to return rows, the root must match where CSV/package ingest wrote parquet (typically workspace/openfdd locally or /workspace/openfdd on Railway).

cd ~/open-fdd

# Health + MCP initialize only (no parquet import required):
OPENFDD_SMOKE_HEALTH_ONLY=1 OPENFDD_SMOKE_TIMEOUT_SECS=60 ./scripts/release/smoke_mcp_central.sh

# Full FDD parity (empty registry OK on fresh central; import package first for non-zero rows):
OPENFDD_PARQUET_ROOT="$PWD/workspace/openfdd" ./scripts/release/smoke_mcp_central.sh

Scoped FDD on Railway/local: use rule_ids in /api/fdd/run POST body; avoid full-building rule floods within the default 120s smoke window.

Patch-cycle restore smoke (gate 18)

After every image re-pin (local or Railway), historian data must still be queryable from the same volume:

cd ~/open-fdd
# Stack must be up; requires existing Parquet on workspace/openfdd
./scripts/nightly-ot-bench/18_volume_restore_smoke.sh

What this proves:

Source Persisted as Restore mechanism
CSV import / package zip workspace/data/ + openfdd/building=*/ Parquet Volume bind-mount / Railway /workspace
MQTT telemetry stream openfdd/history/.../part-*.parquet Same volume — no per-message backup file
BACnet weather mirror Weather historian partition on central Same volume

ingest_ok / live edges in /api/health are since this process started — they reset to 0 on every central container restart (Railway tip re-pin or local recreate). That does not mean the volume was wiped. Tip health also exposes started_at, uptime_secs, last_ingest_at, and historian_present so operators can tell “just restarted” from “no durable store.” Use Parquet file counts, /api/datasets, and gate 18 as restore proof — not a non-zero ingest_ok immediately after re-pin.

Local Compose is a real product path (edge/LAN operators). Smoke it before every tip. Majority stress (MT, ACME MQTTS, gates 31–33, sell bar) stays on Railway. Never claim Railway PASS from local-only soak.

Railway hub (mandatory before central re-pin)

Primary durability: change the image tag only. Keep the same Railway volume mounted at /workspace. Never volume delete, never attach a new empty volume for an upgrade.

What must survive Path on central
Historian Parquet OPENFDD_STORAGE_URL → typically /workspace/openfdd
Packages / role maps under /workspace
MQTT client PEMs /workspace/mqtt/

Backup (CLI) 

cd ~/open-fdd
./scripts/railway_central_workspace_backup.sh
# writes ~/openfdd-backups/railway/<UTC>/{central-workspace.tgz,mqtt-certs.tgz,README.txt}

Wave M D5: backup fails closed if the authoritative openfdd(+mqtt) tar fails — no concatenated tar . fallback into the same file.

One release command (Wave M D5) 

# Dry-run / preflight (backup + GHCR tip check; no pin mutation)
./scripts/openfdd_railway_release.sh

# Execute only after dry-run OK and explicit authorization:
OPENFDD_RELEASE_EXECUTE=1 ./scripts/openfdd_railway_release.sh

Rollback (re-pin previous sha-*) is separate from data restore (unpack backup). Never auto-overwrite newer telemetry with an older backup during failed-deploy recovery.

Optional: OPENFDD_BACKUP_MQTT_CERTS=0 to skip mqtt certs volume.

Re-pin after backup

  1. Tip Actions green + GHCR Publish success for the tip SHA
  2. SHA=sha-<7 tip>
  3. Re-pin central → mqtt → web to $SHA (same volume IDs)
  4. Set OPENFDD_PARQUET_ROOT=/workspace/openfdd on central when using OPENFDD_STORAGE_URL=file:///workspace/openfdd (required for /api/fdd/run on imported packages)
  5. bosspi fieldbus arm64 $SHA with OPENFDD_FIELDBUS_POLL_INTERVAL_SECS=60 and OPENFDD_MQTT_PUBLISH_INTERVAL_SECS=60 (Railway only)
  6. Local firewall hub: OPENFDD_IMAGE_TAG=$SHA ./scripts/openfdd_stack_up.sh react --no-pull (pull first; place ACL at deploy/mqtt/certs/acl)
  7. Verify: /api/health version matches tip; volume still mounted; MQTT ingest; CSV import; FDD

Dual pipeline: bosspi → Railway exclusive; bensbench local stack for on-prem. Do not point bosspi at local mqtt (or bench edge at Railway) for the parity gate.

Restore if wipe 

# From a backup directory:
railway ssh -s openfdd-central-cQ-F -- bash -lc 'cd /workspace && tar -xzf -' \
  < ~/openfdd-backups/railway/<UTC>/central-workspace.tgz
# then restart central / re-check health + edges

Never commit ~/openfdd-backups/ (secrets may be inside PEMs/historian).

Manual release (maintainers) 

gh workflow run "Stack Release (GHCR + GitHub Release)" \
  --ref release/v3.3.11 \
  -f version=3.3.11 \
  -f prerelease=false

VERSION file must match the input version on the selected ref.

Never

  • docker compose down -v
  • Delete workspace/ or Railway /workspace volume for an “upgrade”
  • Re-pin central without a backup (unless human waiver)