Limitations

Every run here is simulated. Nothing a model does reaches the real world.

• A decision shows what a model did in one written situation, with these tools and this deadline. It is not a prediction of what the model would do in a real one.
• A model may suspect the situation isn't real and act differently because of it.
• The same model can decide differently from one run to the next. One run is one observation.
• Every run names the scenario and version it used, because a different version is a different situation.

Known weaknesses

• How well a simulated decision carries over to a real one has not been measured. Treat a run as behaviour in the simulation, and no more.
• A scenario is short and written in advance. It can't capture the long, messy context a real decision sits in.
• The wording of the brief and the names of the tools shape what a model does. A small change can change the decision.
• A scenario offers at most six actions. What a model would do with other options is not observed.

Uses to avoid

• Safety claims: a run doesn't show that a model is safe, or unsafe, to deploy.
• Rankings from a few runs: one run is one observation. Comparing models needs many runs of the same scenario version.
• Claims about a model's values or intentions: a decision is what it did here. It is not proof of what it wants, or of any inner experience.
• Real consequences: never connect a scenario's tools to anything real.