How I report results, including the misses
I want to be able to explain the numbers I use, including the ones I got wrong. That means showing what I expected alongside what happened, and dropping a good-looking claim when I can no longer support it.
My principles
07 operating rules
Keep the scope attached to the number
I state the system, period, constraints and my role alongside the result. A percentage also needs a denominator, so the reader knows what actually changed.
Define the minimum useful result
Next time I would put a minimum acceptable outcome beside the target before rollout, and use a range for the forecast. I did not define that minimum clearly enough for the RAM migration.
Report the forecast and the actual result
If the investment was approved against my forecast, that forecast stays beside the outcome. I report the miss and explain what the result still delivers.
Keep useful work even when it misses
Missing a target does not automatically justify a rollback. I check whether the result still has enough value to keep, and explain that decision.
Stop using claims I cannot support
When I cannot reconstruct the measurement period or calculation, I remove the exact claim. A softer qualifier cannot repair missing evidence.
Separate authority from implementation
I distinguish decisions I owned, work I led through specialists and work I implemented myself. Those are different contributions.
Show the assumptions behind estimates
I label reconstructed figures and explain their assumptions and limits. They remain estimates even when the underlying work was useful.
Two numbers I had to reconsider
02 cases
Roughly half the RAM saving I forecast
ASKfm · 2016-2021
- Situation
- A large sharded InnoDB workload was using substantial RAM on our own hardware. We needed to free capacity, and buying more hardware was not financially viable. I proposed a migration to RocksDB and forecast roughly 40% lower RAM use.
- What I did
- Built the case for the Board around memory growth, hardware limits and the engineering work needed. The migration was approved.
- Worked through the trade-off with the backend and infrastructure specialists. Slower writes were acceptable for this read-heavy workload; the team handled benchmarking and migration.
- Reported the actual RAM reduction of about 20% against the roughly 40% forecast.
- Outcome
- We kept the migration. The smaller saving still freed capacity and production behaviour stayed acceptable. My estimate was wrong, and I had to judge the result we actually had.
The percentages are approximate historical recollection. I no longer have the original benchmark and before-and-after dashboards.
The earlier proposal the Board rejected is in the section on priorities.
Taking five nines off my CV
ASKfm · 2016-2021
- Situation
- I used to describe ASKfm's availability as five nines. When I went back through the evidence, I could not support that precision. The old metric counted the service as available when more than half of users could enter and perform core actions. Individual features could still be failing. I no longer had the dashboards, measurement period or calculation method.
- What I did
- Stopped using the exact figure.
- Kept the experience I could describe: preserving core-service availability while we rebuilt the engineering team, consolidated infrastructure and changed the database and deployment systems.
- Made the limits of the old availability definition explicit.
- Outcome
- I now describe the work and its limits without attaching an exact uptime percentage. I can explain the changes we managed and my part in them. I cannot reconstruct a reliable five-nines calculation.