From scores to diagnoses

Leaderboards are useful. They compress a great deal of work into a ranking anyone can read, and they give a field a common point of reference. But a ranking answers only one question: who is ahead. The people building AI systems for finance need answers to two more. Why did the system do well or badly? And what should change before the next run?

Alpha Bench will publish rankings once public results begin. They will not be its main product. The main product is a diagnosis.

What a single number hides

Imagine two systems that earn the same return over the same market replay. One got there with concentrated positions and a lucky quarter. The other spread its risk, paid realistic costs and supported each decision with evidence. A single score treats them as equals. An experienced investor would not.

Financial outcomes are noisy. Over short horizons, luck can easily outweigh skill, and a strategy that suits one market regime can fail in the next. An average across twelve strategy families can also hide a system that reads filings well but handles risk carelessly, or one that is strong in equities and weak in credit. Above all, a number gives no direction. A builder who learns that a system scored below a rival still does not know what to fix.

Outcome and process

Alpha Bench scores every evaluation on two complementary pillars.

Outcome is about results. It measures risk-adjusted performance net of realistic costs, compared with simple, transparent baselines, so that added complexity has to earn its place.

Process is about how those results were reached. It asks whether claims are supported by evidence in the inputs, whether the reasoning holds together, whether risk was sized and managed with discipline, whether constraints and mandates were respected, whether costs were counted, and whether stated confidence matched what actually happened.

Neither pillar is enough on its own. A good outcome reached through poor process is luck that will not last. A sound process with a poor outcome may simply have met bad luck. Read together, they tell skill from luck with more confidence than either can alone.

A score should be the start of a conversation about what to fix, not the end of one.

Capabilities and failure modes

Diagnostics, the third pillar, break results down along two axes. The first is capability: the specific skills a strategy family demands, such as reading a filing accurately, sizing positions to volatility, estimating the cost of a trade, or explaining a risk to a client in plain language. The second is failure modes: recurring mistakes that the tasks are built to surface.

Some failure modes appear across many families:

• Look-ahead leaks: using information that was not available at the time of the decision.
• Cost blindness: proposing trades whose edge disappears once spreads, fees and slippage are counted.
• Overconfidence: claiming more certainty than the evidence supports, or making forecasts that are poorly calibrated.
• Constraint violations: ignoring position limits, a mandate, or a client’s stated circumstances.
• Unsupported claims: citing figures or facts that do not appear in the inputs.

Others belong to a single family, such as mishandling futures rolls in trend following or misreading the terms of a convertible bond. Naming failure modes precisely turns a vague sense that a system is weak into something a team can test and fix.

What a diagnostic report contains

Every evaluation is designed to return a report that a builder can act on. It includes:

• A scorecard by track and strategy family, with uncertainty ranges rather than bare point estimates.
• Outcome and process scores side by side, so it is clear where results came from.
• The capabilities where the system was strongest and weakest, set against reference baselines.
• The most frequent failure modes, illustrated with examples from published tasks.
• The exact versions of the tasks, data and harness used, so that any run can be reproduced.

Checking that improvements generalize

Any benchmark that reports in this much detail invites a risk: teams may learn to fix the benchmark rather than the system. Alpha Bench is designed so that this does not pay. After a builder makes a change, the system is evaluated again on tasks it has never seen, drawn from the rolling refresh and from held-out sets that are never published. The Live track adds questions whose answers do not yet exist.

An improvement counts only if it holds out of sample. If a fix for cost blindness also reduces wasted costs on new tasks, in other strategy families and in live decisions, it is real. If it helps only on yesterday’s questions, the next report will show that too.

That is the loop Alpha Bench is built around: evaluate, diagnose, improve, re-evaluate. Rankings will tell the field who is ahead. Diagnoses are meant to help every system get better.