Can Simulated Environments Predict Real AI Safety? A New Paper Reports 3.6 Points of Error vs 10.1
TL;DR. A preprint, “Simulated Control Evals,” says simulated evaluations predict safety “with a root mean squared error of 3.6 percentage points” across five monitors at a 5% audit budget, “compared with 10.1 for BashArena.” It also says they explain 78–85% of between-environment variance.
The claim
Testing whether an AI system can be safely deployed with monitors watching it is expensive when you need a realistic environment for every setting. The paper’s answer is to use simulated environments. By the abstract’s numbers they track the real result much more closely than the BashArena comparison:
| Method | Error predicting safety (RMSE, percentage points) |
|---|---|
| Simulated control evals | 3.6 |
| BashArena (comparison in the paper) | 10.1 |
The authors are Jeffrey Olmo, David Zhang and Tyler Tracy; it was submitted 22 October 2026.
The fine print
- RMSE is an average error. It does not mean every prediction is within 3.6 points.
- The numbers apply to five monitors at one audit budget (5%), in this paper’s own setup.
- We read the abstract only, so we don’t describe the method beyond what it says.
- Preprint on arXiv. We don’t know of peer review.
Sources
What we read: The abstract only.
Scenda.AI comments on research by others. We are not the authors of the papers and reports covered here, and nothing on this page is endorsed by them.