Introducing Alpha Bench

People already turn to AI systems with questions about money. They ask what a company’s latest annual report really says, whether a portfolio leans too heavily on one sector, how a bond fund might behave if interest rates rise, or whether a strategy they read about is worth trying. Model labs, harness builders and finance teams are working hard to answer these questions well. What the field still lacks is a shared, honest way to tell whether the answers are getting better.

Today we are introducing Alpha Bench, an open benchmark for AI in finance, maintained by the Alpha Bench Foundation. Its purpose is to help every large language model, agent harness and vertical finance agent become better at finance, and know how to become better, so that the financial help people receive keeps improving.

Alpha Bench is still being built. There are no results yet, no leaderboard entries and no partners to announce. This post explains what we are building, why, and how you can help shape it.

Why finance needs a living benchmark

Most benchmarks are snapshots. A fixed set of questions is published, systems are scored against it, and over time the questions find their way into training data. Scores keep rising while their meaning drains away. In finance, a snapshot has a second weakness: markets change. A system that reasons well in one regime of interest rates, inflation or volatility may reason poorly in the next.

Finance is also unforgiving of shortcuts that other domains tolerate. An answer can read fluently and still rest on a number nobody could have known at the time, ignore the cost of trading, or skip the one risk a client most needed to hear about. Financial decisions compound, so quiet errors like these matter.

That is why Alpha Bench is designed to be living rather than fixed. Tasks are refreshed on a rolling basis. A held-out portion is never published. A live track scores decisions before their outcomes exist. And every run is versioned, so results can be reproduced and compared over time.

What Alpha Bench measures

Alpha Bench evaluates three kinds of systems, each under conditions that keep its results comparable.

• Models are evaluated inside a fixed reference harness, so differences in results reflect the model rather than the tooling around it.
• Harnesses are evaluated with the underlying models held fixed, which isolates what the scaffolding adds: tools, data access, memory, planning and execution.
• Vertical agents are evaluated end to end, as complete systems, the way a user would experience them.

The tasks cover twelve families of investment strategy, from fundamental investing and asset allocation to event-driven strategies, options and volatility, and fixed income and credit. Each family has its own tasks, metrics and failure modes, because sound judgment in a merger looks different from sound judgment in a rebalancing decision.

Tasks are organized into four tracks:

• Analysis: point-in-time research tasks, such as reading filings, building a valuation or interpreting a macroeconomic release using only what was known on the day.
• Simulation: historical market replays with realistic frictions, in which a system makes decisions step by step without seeing what comes next.
• Live: forward evaluation on live data. Outcomes do not exist when decisions are made, so no system can have seen the answers in its training data.
• Client: conversations with a person, scored on suitability, clarity of explanation and honest disclosure of risk.

Built to show how to improve

A single number can tell you who is ahead. It rarely tells you why, or what to change. Alpha Bench reports every evaluation on three pillars. Outcome measures risk-adjusted results. Process measures how those results were reached: the quality of evidence and reasoning, discipline about risk, respect for constraints and costs, and whether stated confidence matches what actually happened. Diagnostics break both down by capability and failure mode.

We want every result to point to something a builder can fix, and every fix to be tested again on questions the system has never seen.

That is the loop at the center of the benchmark: evaluate, diagnose, improve, re-evaluate. Fresh tasks and the live track check whether an improvement holds out of sample, not just on yesterday’s questions.

What happens next

We will build Alpha Bench in stages and publish as we go. The first stage is the methodology: how tasks are constructed, how data is kept point-in-time, how frictions are modeled and how scores are calculated. Task sets for each strategy family come next, together with the reference harness used to evaluate models. Public results will begin with version 1.0, once every run can be reproduced end to end.

We are not attaching dates to these stages yet. We would rather publish a schedule we can keep than one we cannot. Throughout, integrity comes before speed: point-in-time data, contamination control, realistic costs, testing across different market regimes, and enough trials to tell skill from luck.

An invitation

A benchmark for finance should not be designed by one group alone. We are looking for model labs, harness and agent builders who want to test their systems, practitioners who know where financial reasoning goes wrong, and researchers who care about careful evaluation. If that describes you, we would like to hear from you.

Alpha Bench evaluates AI systems. It does not give investment advice, and nothing it publishes is a recommendation to buy or sell anything. Its aim is narrower and, we think, more useful: to make the financial help people receive from AI measurably better over time.