Research
Benchmarks
How Revelion is measured, and the standard a result has to meet before we publish a number next to our own name.
Position
The method matters more than the score
Security vendors publish benchmark numbers constantly, and almost none of them can be reproduced. The environment is unnamed, the scoring is undisclosed, and the comparison set was chosen after the result was known.
We would rather publish slowly and be checkable. That means a benchmark entry here carries the environment, the date, the version tested, the scoring rules, and the failures alongside the successes.
Results are being prepared on that basis. Until each one can be cited in full, this page describes the method rather than asserting a number.
Method
What a published benchmark will contain
The standard we are holding ourselves to before anything gets a figure attached.
- A named environment
- Benchmarks run against environments that can be independently inspected, not internal fixtures.
- Version and date
- Which build was tested and when, because agent performance moves with the model underneath it.
- Disclosed scoring
- What counted as a success, what counted as a partial, and what counted as a miss.
- Failures included
- The environments where the agent did not succeed, reported alongside the ones where it did.
- Reproduction detail
- Enough for a competent reader to run the same test and compare.
Judge the method first, then the numbers when they land.
Read the methodology