# How I compared two code-review models
Source: https://manaienko.com/how-i-work/code-review-model-experiment/
Author: Artem Manaienko

I wanted to choose one review model. After ten paired reviews, I kept both. The faster one cost less per review, even at a higher token price. The slower one found enough additional defects that I did not want to lose it.

I use model review before reviewing changes myself on my independent product. I compared two models on the same ten tasks because I wanted evidence for the choice. I expected to keep one. Their findings overlapped so little that I ended up keeping both.

## How I compared them

Each pair used the same working tree and rendered prompt. I alternated the order between tasks, and neither reviewer could see the other's findings. A changed working tree invalidated the pair.

The configurations were gpt-5.6-sol at xhigh reasoning effort and gpt-6-astra at medium. That matters: this compares two working configurations, and does not separate the model from its reasoning budget.

Ten tasks, twenty reviews

gpt-5.6-sol, xhigh

### Slower, with more unique findings

-   829 seconds and $2.83 per review, at the median.
-   18 defects the other reviewer missed.
-   More findings around test gates and missing coverage.

gpt-6-astra, medium

### Faster and cheaper per review

-   125 seconds and $1.83 per review, at the median.
-   7 defects the other reviewer missed.
-   Findings included runtime behaviour and deploy security.

On eight of ten tasks, the findings barely overlapped. Costs are calculated from measured tokens and the September 2026 rates used for this experiment.

## The higher token price cost less to run

Cost per completed review

-   2.5x

    higher token price

    Input/output list rates: $4/$20 per million for sol and $10/$50 for astra.

-   5.1x

    fewer tokens used

    Median tokens per review: 4.54M for sol, 0.89M for astra.

-   1.6x

    lower cost per review

    Median calculated cost: $2.83 for sol, $1.83 for astra. Astra cost less on eight of ten tasks.

Reasoning accounted for 73% of output on sol and 28% on astra. The ranges did not overlap across these ten tasks. Astra finished first on every task, with a median paired speed advantage of 6.4x. I would not have known the cost per completed review from the token price alone.

## Why I kept the slower reviewer

I read both reports for each task and matched findings to the same underlying defect.

| Finding set | Count | Examples |
| --- | --- | --- |
| Only sol | 18 | Weakened test gates, a red build, missing coverage |
| Only astra | 7 | Runtime behaviour, deploy security, a signing bypass |
| Both | 10 | Mostly in two tasks where their reports agreed |

Sol found more defects the other model missed. But on one task it approved the change, while astra found a defect that I confirmed and fixed. I could not treat the longer report as the more reliable one on every task.

The fast reviewer now runs first so I can react to blocking findings sooner. I keep the slower reviewer for the additional problems it catches. This is a decision from ten paired reviews on my own work, not a general ranking of the models.

What I would test next

I would raise the faster model's reasoning effort and repeat the comparison. That could help show how much of the difference came from the model and how much from the budget I gave it. This run does not answer that.

![Experiment results: gpt-5.6-sol at xhigh effort, 829 seconds and $2.83 per review, 18 unique defects; gpt-6-astra at medium effort, 125 seconds and $1.83 per review, 7 unique defects. Rates used were $4/$20 and $10/$50 per million input/output tokens.](https://manaienko.com/images/code-review-economics.png)

The original experiment card. Medians across ten pairs; standard-tier list rates used in September 2026, cached input at a tenth of input. Calculated costs, not invoices.