How I compared two code-review models
I wanted to choose one review model. After ten paired reviews, I kept both. The faster one cost less per review, even at a higher token price. The slower one found enough additional defects that I did not want to lose it.
I use model review before reviewing changes myself on my independent product. I compared two models on the same ten tasks because I wanted evidence for the choice. I expected to keep one. Their findings overlapped so little that I ended up keeping both.
How I compared them
Each pair used the same working tree and rendered prompt. I alternated the order between tasks, and neither reviewer could see the other's findings. A changed working tree invalidated the pair.
The configurations were gpt-5.6-sol at xhigh reasoning effort and gpt-6-astra at medium. That matters: this compares two working configurations, and does not separate the model from its reasoning budget.
Ten tasks, twenty reviews
gpt-5.6-sol, xhigh
Slower, with more unique findings
- 829 seconds and $2.83 per review, at the median.
- 18 defects the other reviewer missed.
- More findings around test gates and missing coverage.
gpt-6-astra, medium
Faster and cheaper per review
- 125 seconds and $1.83 per review, at the median.
- 7 defects the other reviewer missed.
- Findings included runtime behaviour and deploy security.
The higher token price cost less to run
Cost per completed review
2.5x
higher token price
Input/output list rates: $4/$20 per million for sol and $10/$50 for astra.
5.1x
fewer tokens used
Median tokens per review: 4.54M for sol, 0.89M for astra.
1.6x
lower cost per review
Median calculated cost: $2.83 for sol, $1.83 for astra. Astra cost less on eight of ten tasks.
Reasoning accounted for 73% of output on sol and 28% on astra. The ranges did not overlap across these ten tasks. Astra finished first on every task, with a median paired speed advantage of 6.4x. I would not have known the cost per completed review from the token price alone.
Why I kept the slower reviewer
I read both reports for each task and matched findings to the same underlying defect.
| Finding set | Count | Examples |
|---|---|---|
| Only sol | 18 | Weakened test gates, a red build, missing coverage |
| Only astra | 7 | Runtime behaviour, deploy security, a signing bypass |
| Both | 10 | Mostly in two tasks where their reports agreed |
Sol found more defects the other model missed. But on one task it approved the change, while astra found a defect that I confirmed and fixed. I could not treat the longer report as the more reliable one on every task.
The fast reviewer now runs first so I can react to blocking findings sooner. I keep the slower reviewer for the additional problems it catches. This is a decision from ten paired reviews on my own work, not a general ranking of the models.
