Architecture
and integration.

I design and integrate the web apps, API contracts, background processes and storage, including their recovery rules.

Runtime: API, workers, scheduler and reconciliation
Contracts: OpenAPI and generated TypeScript clients
State: PostgreSQL, pgvector and durable jobs

Explore the architecture

01System topologySeparate processes, shared durable state02Data and contractsOne database, explicit boundaries03Execution and failureA late worker cannot commit an old attempt

Implemented in NADILI. Private, pre-launch.

System topology

Separate processes, shared durable state

Two Next.js applications call FastAPI. Workers, scheduling and identity reconciliation run as separate Python processes. PostgreSQL holds durable product and job state.

Both web apps call FastAPI. API, workers, scheduler and reconciliation use PostgreSQL. Valkey holds transient state. Scheduled jobs reach workers through PostgreSQL.

Where work runs

API: Validate requests, enforce access and call application services.
Worker: Claim jobs, acquire sources and execute processing pipelines.
Scheduler: Evaluate due work and reconcile pending external AI operations.
Reconcile: Run identity reconciliation independently.

A web request does not own a background job

Jobs and progress persist in PostgreSQL. A worker can resume eligible work after another process exits. Valkey handles transient coordination; PostgreSQL owns durable jobs.

Trace a request through the system

The API validates a request and calls an application service. Repositories run in a SQLAlchemy unit of work. Background work is recorded as a job for a worker to claim.

The API, worker, scheduler and identity reconciliation run as separate services. Scheduled work reaches workers through persisted jobs.

Data and contracts

One database, explicit boundaries

PostgreSQL stores product records, jobs, evidence and vector indexes. Both web applications use generated TypeScript clients from the API contract.

What crosses each boundary

Web / API: OpenAPI produces separate public and private client outputs.
Workspace: Repositories scope owned records to the current workspace.
Knowledge: Claims retain evidence; raw acquisition payloads are temporary.
Retrieval: pgvector embeddings are keyed by content hash and model.

The public contract has its own shape

A maintained public OpenAPI fragment is merged into the exported contract. The TypeScript generator produces separate public and private clients with separate schema selections. Both applications use the shared client package.

How contract drift and stale embeddings are handled

The repository exports OpenAPI and checks it against the canonical contract. Generated-file drift has its own check. The public API shape stays explicit while private contracts evolve.

Search embeddings live in PostgreSQL beside indexed content. Content identity and model identify each derived record. HNSW cosine indexes support vector retrieval with workspace scope.

PostgreSQL is required in development, tests and CI. SQLite is unsupported because database-specific behavior matters here.

Execution and failure

A late worker cannot commit an old attempt

A job claim carries an attempt number and an expiring lease. Before completion, the repository locks the job and checks current ownership.

The job lifecycle

Claim: FOR UPDATE SKIP LOCKED lets workers claim different eligible rows.
Recover: Expired running jobs can be reclaimed within their attempt limit.
Fence: Check workspace, public ID, running status, attempt and live lease.
Wait: Persist waiting_external while an external AI operation is pending.

Cancellation must survive a late response

A provider call can finish after cancellation or lease expiry. The completion check rejects a stale attempt before it writes state. Waiting on external AI work is recorded and releases the lease.

Inspect the completion fence and scheduler failover

Completion locks the job row, then checks workspace, job identity, running status, attempt number and an unexpired lease. A reclaimed job has a different attempt number; a cancelled job is no longer running. Either change rejects the old worker.

Schedulers use a non-blocking PostgreSQL advisory lock on a dedicated AUTOCOMMIT connection. Standbys retry later; the leader probes its connection before evaluation.

A connection loss during evaluation can still allow one overlapping cycle. The design relies on job deduplication and row locks for that boundary; leadership alone does not guarantee exactly-once execution.