# NADILI: Applied AI Source: https://manaienko.com/projects/nadili/ai/ Author: Artem Manaienko # Knowledge, retrieval _and model work._ I define prompts, retrieval and model boundaries. Frozen evaluations compare behavior and cost before I accept a change. **Retrieval:** Lexical search and pgvector **Evals:** Frozen inputs and shared prompts **Cost:** Durable provider accounting Explore AI in the product 01 Search and RAG **Retrieve evidence before ranking answers** **Retrieval**02 Prompt and context engineering **Evaluate the production contract on a frozen sample** **Evals**03 Provider accounting **A failed result can still have a cost** **Cost** Frozen evaluations with explicit cost coverage. Search and RAG ## Retrieve evidence before ranking answers Public search combines lexical and vector retrieval over stored content. A model orders candidates. It cannot invent an answer or turn a semantic match into an exact match. ### The search cascade **Direct match:** Normalized equality with a published title or authored question returns first. **Candidates:** Merge lexical and eligible vector candidates, then cap the rerank input at eight. **Budget:** Embedding and reranking share a configurable 2.5-second default deadline. **Fallback:** Reranker failure returns strict lexical results only. ### A lexical hit must not block a better semantic result The earlier cascade used vectors only when lexical retrieval was empty. A weak lexical match could hide a better answer. Eligible non-direct searches now merge both pools before one reranker call. Why the deadline and failure path changed together Benchmarking showed that time to first token dominated the short reranker response. Changing model tier did not remove the timeout. Both model stages now use the time left on one monotonic deadline. If embedding fails, the lexical pool can still be reranked. If reranking fails, broad-term and vector-only candidates stay suppressed. The deadline is a budget, not a response-time guarantee. Search query text, vectors and candidate metadata remain transient. pgvector stores indexed content embeddings, while the response refers to stored published material. Prompt and context engineering ## Evaluate the production contract on a frozen sample Topic prompt evaluation runs bounded source material through diagnostic stages. Claim resolution shares the production builder, schema, parser and batching code. Results stay outside accepted knowledge. ### What stays comparable **Input:** A bounded sample and explicit Topic context. **Contract:** Shared Claim-resolution builders, structured schema and parser. **Variable:** Prompt candidate, model and reasoning effort selected for the run. **Result:** Diagnostic output, counts and usage; no automatic knowledge promotion. ### Avoid testing a second implementation of the prompt A copied prompt or loose parser can pass while the product fails. The Claim-resolution runner imports the production modules and uses the same request and validation contract. Where evaluation deliberately differs from production Evaluation selects its model and reasoning effort explicitly. Production resolves them from the Topic rules snapshot. The soft-relevance stage records its minimal semantic input; Claim resolution reuses the full shared builder, schema, parser and batching path. A prompt-only comparison can reuse the frozen sample. A source or input-contract change requires a new sample. Quality and usage are reviewed together, with evaluation spend tracked separately. Provider accounting ## A failed result can still have a cost Provider usage is recorded before output validation and product writes. PostgreSQL owns the ledger, including failed attempts and missing usage. Spend carries a completeness signal. ### One provider attempt **Reserve:** Commit a reserved attempt with immutable accounting context. **Dispatch:** Call the provider through the metered gateway. **Record:** Commit safe reported usage or mark usage unavailable. **Validate:** Only then validate the output and update product state. ### Keep accounting independent of product success A refusal, malformed response or rejected vector can still be billable. Independent accounting transactions retain that fact when product work fails. Unknown usage or pricing makes the report incomplete, never zero. Retries, duplicate finalization and accounting limits Replaying an identical reservation is idempotent. A conflicting context is rejected. Only the first reserved-to-reported transition enters the Source-history cost roll-up, so repeated reconciliation cannot count it twice. Reasoning tokens are included in completion tokens and are not charged again. Evaluation work is separate from production Source-history cost. The ledger calculates spend from reported usage and pinned pricing. Invoice adjustments and timeouts with no usage remain outside it. [Back to all six areas](https://manaienko.com/projects/#scope-overview-title)[Next: Delivery ↗](https://manaienko.com/projects/nadili/delivery/)I design and integrate the product, with coding agents supporting implementation.