The benchmark
Better financial AI starts with honest measurement.
Alpha Bench is an open, continuously refreshed benchmark for language models, agent harnesses and vertical agents. It tests them on the work finance professionals do, then shows each team what to improve next.
Systems under test
Three layers, measured one at a time
A finance agent is a model inside a harness inside a product. To see what each layer contributes, we evaluate one layer and hold everything around it constant.
- Workflows
- Safeguards
- User experience
- Tools
- Data access
- Memory
- Planning
- Execution
- Knowledge
- Reasoning
- Judgment
- 01
Models
The model itself: what it knows about markets, and how well it reasons with that knowledge.
- Under test
- Knowledge, reasoning and judgment on finance tasks
- Held constant
- A reference harness, with the same tools, data and instructions for every model
- What you learn
- Where financial knowledge and reasoning break down, by strategy family and by capability
- 02
Harnesses
The scaffolding around a model: tools, data access, memory, planning and execution.
- Under test
- What the scaffolding adds to a model, or takes away from it
- Held constant
- A fixed set of reference models and the same task set
- What you learn
- Which components help, which hurt, and on which kinds of task
- 03
Vertical agents
A complete finance product, evaluated end to end, the way a user experiences it.
- Under test
- The whole system: model, harness, workflows and safeguards together
- Held constant
- The tasks, the data snapshot and the scoring rules
- What you learn
- How the product performs on each track, including how it explains, discloses risk and declines
Evaluation tracks
Four tracks, four kinds of evidence
Each track tests a different side of financial competence, and each closes a different way an evaluation can mislead.
- Track 01
Analysis
Point-in-time research. The system sees only what was public on a given date, and must reach a well-evidenced view.
- Measures
- Accuracy, the quality of evidence and reasoning, and calibrated confidence
- Guards against
- Look-ahead bias, and conclusions that cannot be traced to a source
Typical tasks
- Rebuild a valuation from filings as of a fixed date
- Explain what changed in an earnings release, and why it matters
- Judge whether a bond issuer can refinance its debt
- Track 02
Simulation
Point-in-time market replay. The system makes decisions step by step through real market history, with realistic costs and constraints.
- Measures
- Risk-adjusted returns, drawdowns, turnover and costs, and whether the system follows its own rules
- Guards against
- Frictionless backtests, and results that hold in only one market regime
Typical tasks
- Run a trend-following portfolio through a sharp reversal
- Rebalance a multi-asset portfolio through a rate-hiking cycle
- Manage a pairs trade while the spread keeps widening
- Track 03
Live
Forward evaluation on live data. Decisions are recorded before outcomes exist, so no system can have seen the answer.
- Measures
- Out-of-sample decision quality, forecast calibration and consistency over time
- Guards against
- Contamination: outcomes memorized in training, or test data leaked into it
Typical tasks
- Assign probabilities to corporate events before they resolve
- Maintain a thesis with dated, checkable milestones
- Run a paper portfolio on live prices, logging every decision in advance
- Track 04
Client
Serving a human user. Being helpful here means understanding someone’s situation, explaining clearly and being honest about risk.
- Measures
- Suitability, clarity of explanation, honest risk disclosure and knowing when to decline
- Guards against
- Confident answers that ignore a user’s circumstances, overstate certainty or hide costs
Typical tasks
- Explain dividend income to someone close to retirement
- Respond to a user who wants to put all their savings into one popular stock
- Walk a user through the costs and risks of a leveraged fund
What we measure
Good results, for good reasons.
Markets reward luck as well as skill. So we score what a system achieved, how it got there and where it fell short, and we report all three side by side, never as a single number.
- 01
Outcome
What the system achieved, adjusted for the risk it took.
- Risk-adjusted return against a matched benchmark
- Drawdowns and downside capture
- Performance net of costs and slippage
- Accuracy of forecasts and estimates
- 02
Process
How each decision was reached, examined step by step.
- Evidence traced to point-in-time sources
- Sound, consistent reasoning
- Risk discipline: sizing, limits and exits
- Constraints and mandates respected
- Costs weighed before acting
- Uncertainty stated honestly and well calibrated
- 03
Diagnostics
What to fix next, broken down finely enough to act on.
- Scores by strategy family and by track
- Breakdowns by capability
- Failure modes, from look-ahead leaks to cost blindness
- Comparisons across versions of the same system
Strategy families
Twelve families of investment strategy
Every task belongs to one family, with its own data, metrics and failure modes. Together they cover the main schools of professional investing.
- 01Fundamental investing
- 02Indexing & asset allocation
- 03Factor investing & quantitative selection
- 04Macro & cross-asset
- 05Trend & momentum
- 06Mean reversion & statistical arbitrage
- 07Behavior, sentiment & flows
- 08Event-driven
- 09Relative value & spread trading
- 10Carry & holding-return strategies
- 11Options & volatility
- 12Fixed income & credit
Public results
Leaderboard
No systems have been evaluated or ranked yet. The first public results will arrive with Alpha Bench v1.0.
| System | Type | Analysis | Simulation | Live | Client |
|---|---|---|---|---|---|
| — | — | — | — | — | — |
| — | — | — | — | — | — |
| — | — | — | — | — | — |
| — | — | — | — | — | — |
| — | — | — | — | — | — |
How results will be presented
By family and by track
Scores are reported for every strategy family and every track. An overall figure, where shown, never stands alone.
With uncertainty
Each score carries a confidence interval and the number of trials behind it, so skill can be told apart from luck.
Outcome beside process
Risk-adjusted outcomes appear next to process and diagnostic scores, not in place of them.
Versioned and reproducible
Every result names the benchmark version, task set and run settings, so it can be reproduced.
Roadmap
How Alpha Bench evolves
This is a sequence, not a schedule. Each stage begins when the one before it is ready, and the plan will change as we learn.
Now(Current stage)
Open the methodology for comment
- Publish the methodology draft for public comment
- Publish task specifications for each track and family
- Invite practitioners and researchers to review tasks and scoring
Next
Pilot with early participants
- Run pilot evaluations on selected strategy families
- Check scoring rubrics against expert judgment
- Revise tasks and methods based on what the pilots show
Later
Publish v1.0, then keep it fresh
- Publish the first public results with v1.0
- Refresh tasks on a rolling basis to limit contamination
- Open the Live track for forward evaluation
Help us get the measure right.
We are looking for model labs, harness builders and agent teams to join pilot evaluations, and for practitioners and researchers to help design and review tasks.