Artem Manaienko

Knowledge, retrieval
and model work.

I define prompts, retrieval and model boundaries. Frozen evaluations compare behavior and cost before I accept a change.

Retrieval
Lexical search and pgvector
Evals
Frozen inputs and shared prompts
Cost
Durable provider accounting
Explore AI in the product

Frozen evaluations with explicit cost coverage.

Search and RAG

Retrieve evidence before ranking answers

Public search combines lexical and vector retrieval over stored content. A model orders candidates. It cannot invent an answer or turn a semantic match into an exact match.

The search cascade

Direct match
Normalized equality with a published title or authored question returns first.
Candidates
Merge lexical and eligible vector candidates, then cap the rerank input at eight.
Budget
Embedding and reranking share a configurable 2.5-second default deadline.
Fallback
Reranker failure returns strict lexical results only.

A lexical hit must not block a better semantic result

The earlier cascade used vectors only when lexical retrieval was empty. A weak lexical match could hide a better answer. Eligible non-direct searches now merge both pools before one reranker call.

Why the deadline and failure path changed together

Benchmarking showed that time to first token dominated the short reranker response. Changing model tier did not remove the timeout. Both model stages now use the time left on one monotonic deadline.

If embedding fails, the lexical pool can still be reranked. If reranking fails, broad-term and vector-only candidates stay suppressed. The deadline is a budget, not a response-time guarantee.

Search query text, vectors and candidate metadata remain transient. pgvector stores indexed content embeddings, while the response refers to stored published material.

Prompt and context engineering

Evaluate the production contract on a frozen sample

Topic prompt evaluation runs bounded source material through diagnostic stages. Claim resolution shares the production builder, schema, parser and batching code. Results stay outside accepted knowledge.

What stays comparable

Input
A bounded sample and explicit Topic context.
Contract
Shared Claim-resolution builders, structured schema and parser.
Variable
Prompt candidate, model and reasoning effort selected for the run.
Result
Diagnostic output, counts and usage; no automatic knowledge promotion.

Avoid testing a second implementation of the prompt

A copied prompt or loose parser can pass while the product fails. The Claim-resolution runner imports the production modules and uses the same request and validation contract.

Where evaluation deliberately differs from production

Evaluation selects its model and reasoning effort explicitly. Production resolves them from the Topic rules snapshot. The soft-relevance stage records its minimal semantic input; Claim resolution reuses the full shared builder, schema, parser and batching path.

A prompt-only comparison can reuse the frozen sample. A source or input-contract change requires a new sample. Quality and usage are reviewed together, with evaluation spend tracked separately.

Provider accounting

A failed result can still have a cost

Provider usage is recorded before output validation and product writes. PostgreSQL owns the ledger, including failed attempts and missing usage. Spend carries a completeness signal.

One provider attempt

Reserve
Commit a reserved attempt with immutable accounting context.
Dispatch
Call the provider through the metered gateway.
Record
Commit safe reported usage or mark usage unavailable.
Validate
Only then validate the output and update product state.

Keep accounting independent of product success

A refusal, malformed response or rejected vector can still be billable. Independent accounting transactions retain that fact when product work fails. Unknown usage or pricing makes the report incomplete, never zero.

Retries, duplicate finalization and accounting limits

Replaying an identical reservation is idempotent. A conflicting context is rejected. Only the first reserved-to-reported transition enters the Source-history cost roll-up, so repeated reconciliation cannot count it twice.

Reasoning tokens are included in completion tokens and are not charged again. Evaluation work is separate from production Source-history cost.

The ledger calculates spend from reported usage and pinned pricing. Invoice adjustments and timeouts with no usage remain outside it.