# NADILI: Operations Source: https://manaienko.com/projects/nadili/operations/ Author: Artem Manaienko # Deployment _and recovery._ I own deployment, recovery and monitoring. The tooling records release checks and open operating prerequisites. **Release:** Signed releases and rollback **Recovery:** Encrypted and signed backup sets **Monitoring:** OpenTofu-managed alerting Explore an operating mechanism 01 Release admission **Deploy a verified release with a recorded predecessor** **Rollout**02 Recovery artifacts **Authenticate a backup before restoring it** **Backups**03 Operational visibility **Keep alerting outside the application host** **Monitoring** Tooling is implemented; activation limits remain explicit. Release admission ## Deploy a verified release with a recorded predecessor The manifest deployment path binds image digests, source revision and release files to a signed release. Before changing the stack, it records the previous configuration, images and database revision. 01 / PUBLIC EDGE02 / SUPPLY CHAIN03 / PRIVATE PLANEPublic browserUser entryEdgePublic ingressPublic webWeb runtimeHardened buildCandidateSBOM + scanVerify contentsSign + attestProvenanceImmutable imagePinned digestAdmin + opsPrivate entryApplication APIService boundaryDatabase + cachePrivate state01 / SUPPLY CHAIN02 / PUBLIC EDGE03 / PRIVATE PLANEHardened buildCandidateSBOM + scanVerify contentsSign + attestProvenanceImmutable imagePinned digestPublic browserUser entryEdgePublic ingressPublic webWeb runtimeAdmin + opsPrivate entryApplication APIService boundaryDatabase + cachePrivate state The public edge, private operating plane and image supply chain have separate boundaries. Signed release admission checks the manifest-based deployment path. ### The deployment boundary **Build:** Build the candidate once, scan it and publish its unchanged digest. **Verify:** Check workflow identity, image roles and the release-file hash inventory. **Record:** Retain the previous images, configuration and Compose graph. **Recover:** Choose rollback or restore based on migration and activation state. ### Rolling back an image does not roll back the database Rollback never attempts a database downgrade. It checks compatibility and migration state. A migrated or unknown state needs an explicit recovery path and a verified backup. Inspect the remaining deployment limits The host publishes verified release files into a content-addressed tree. Existing trees are checked, not repaired in place. Rollback uses retained state and the previous graph. Operator-assembled deployment commands still exist outside signed manifest admission. The architecture records this exception, so the portfolio describes only the manifest path. Feature activation can narrow the rollback window even when a migration is backward compatible. An image from before an activation may no longer be a clean rollback target. Recovery artifacts ## Authenticate a backup before restoring it The logical backup tooling encrypts the database dump and credentials, then signs a manifest over the encrypted artifacts. Restore checks the signature, set identity, inventory and digests before decryption or database restore. ### What makes a recovery set usable **Encryption:** age-encrypted artifacts; the decryption identity stays with the operator. **Manifest:** Bind set ID, source bucket, capture time, file sizes and SHA-256 digests. **Signature:** Verify against the retained signer history at the capture timestamp. **Retention:** Storage-side locks and lifecycle rules protect retained objects. ### The capture host must not be able to erase recovery history The backup design removes host-side pruning. Storage allows write/read but refuses deletion or overwrite of retained objects. A plaintext marker cannot select an automatic restore point. Key rotation, incomplete uploads and activation status Rotation preserves old signing-key validity windows. Replacing trust history with only the newest key would orphan old backups. An incomplete upload without its manifest and signature is not restored. The runbook separates the initial whole-server backup from activation of logical backups and the first restore drill. A daily cadence implies a recovery point of at most 24 hours; point-in-time recovery is deferred. These are repository-defined recovery mechanisms and prerequisites. This portfolio does not claim a newly executed production restore or a measured recovery time. Operational visibility ## Keep alerting outside the application host The declared alerting stack uses an external monitoring service with OpenTofu-managed rules and notification routing. It receives bounded operational telemetry. ### Signals and ownership **Telemetry:** Bounded operational labels; no product records or query text in alert payloads. **Rules:** Declared thresholds, evaluation intervals and no-data/error behavior. **Delivery:** Chat notifications with severity-specific repeat intervals. **Uptime:** External synthetic checks remain gated on public cutover evidence. ### Avoid sharing the failure the alert is meant to report Alert evaluation on the application host would disappear with that host. External evaluation keeps evaluation and delivery outside that shared failure, while adding a monitoring dependency. Public health checks and the remaining failure domain The external probe uses the public /health endpoint. The token-protected /ready endpoint stays private. Alert annotations use static runbook text and bounded labels. The OpenTofu configuration is reviewable; tokens, destinations and activation evidence remain operator-owned. A monitoring outage can still blind evaluation and notifications. A second independent dead-man switch remains follow-up work. [Back to all six areas](https://manaienko.com/projects/#scope-overview-title)I design and integrate the product, with coding agents supporting implementation.