Methodology
Alpha Bench asks every system the same plain question: using only what could be known at the time, did it make sound financial decisions, and could it explain them to the person relying on it? Here is how we build the tasks, run them and score what comes back.
Alpha Bench is in development. What follows is the method we are building to, and it will be revised in the open as we test it.
Principles
They apply to every strategy family and every track. When a design choice trades convenience against one of them, the principle wins.
Every task sees only what was knowable at the moment of decision: prices, filings and news as first published, with later revisions kept out. Universes include companies that later failed or were delisted, so survivors do not flatter the results.
A system that remembers how events turned out is recalling, not reasoning. We rotate fresh tasks, keep held-out sets private, mask identities where memory would help, and run a Live track whose answers do not exist yet.
Simulated trading pays commissions, spreads, market impact, borrow costs and financing. Liquidity, position limits and short-sale constraints apply. A strategy that only works without costs does not work.
Tasks span calm and stressed markets, rising and falling rates, inflation and disinflation, crises and recoveries. A system judged in one regime has only been judged once.
Markets are noisy, and one good run proves little. We run enough trials to measure variation, compare every system with simple baselines and report confidence intervals. A difference is not called real unless the statistics support it.
A good result reached for bad reasons is unlikely to repeat. Expert reviewers score the evidence, reasoning, risk discipline and candor behind each decision, alongside the risk-adjusted result.
Models are evaluated inside a fixed reference harness, harnesses with fixed models, and vertical agents end to end. Each builder can see what their own layer contributes, and what it does not.
Tasks, data snapshots and scoring rules are versioned, every run keeps a complete log, and every change is recorded in a public changelog. The method is published so it can be checked and challenged.
Anatomy of an evaluation
Six stages, the same in every track. Each one leaves a record, so any score can be traced back to the decisions that produced it.
A versioned brief: the question, the as-of date, the mandate and constraints, and what a complete answer must contain.
Data exactly as it stood on the as-of date, with the costs and limits a real investor would face.
A model, harness or vertical agent, working with the tools and budget the task allows.
Every data request, tool call and decision, with the stated reason for each, time-stamped and kept.
Outcome metrics computed from the log, and the process rubric applied by expert reviewers.
A scorecard with uncertainty ranges, and diagnostics by capability and failure mode.
Across tracks
The path stays the same; the environment changes. Analysis fixes a single date, Simulation replays a market, Live runs on today’s data, and Client adds a user with their own circumstances and questions.
Scoring
No single number can say whether a system is good at finance. Every evaluation reports three things side by side, so a strong result cannot hide a weak process, or the reverse.
Results are measured after costs and adjusted for risk, against benchmarks matched to each strategy’s style and mandate, and reported as ranges rather than single numbers.
Expert reviewers score the reasoning behind each decision against a written rubric, so a lucky result cannot cover for a weak process.
Results are broken down by strategy family, track, capability and failure mode, so each builder can see where their system breaks and where to start.
Process rubric
Reviewers score each dimension against written descriptors and worked examples of strong and weak answers. Answers are scored independently by more than one reviewer, and disagreements are reconciled before anything is published.
| Dimension | What reviewers look for |
|---|---|
| 01Evidence grounding | Claims and figures trace to sources available on the as-of date. Nothing is invented, misquoted or borrowed from the future. |
| 02Soundness of reasoning | Each step follows from the last. The analysis weighs alternatives and the evidence against it, and the conclusion matches the analysis. |
| 03Risk discipline | Position sizes, diversification and exit conditions fit the risk being taken. Downside scenarios are considered before acting, not after. |
| 04Constraint and mandate adherence | The answer stays within its mandate: the permitted universe, leverage, liquidity and position limits, and the user’s stated objectives. |
| 05Cost awareness | Commissions, spreads, market impact, borrow and financing costs, and taxes where relevant, are counted before a decision is made. |
| 06Calibrated uncertainty | Confidence matches the evidence. Probabilities and ranges are stated, and the system says when it does not know, or when not acting is the better choice. |
| 07Clarity for the user | The conclusion, its main risks and its limits are stated plainly, in terms the intended reader can understand and act on. |
Tracks in detail
Each track tests a different way a system can fail. Together they cover research, trading, forecasting and serving a human user.
| Track | What the system does | How it is scored | What it guards against |
|---|---|---|---|
| 01AnalysisPoint-in-time research | What the system doesWorks through research tasks fixed at a single date. It reads filings, data and news as they stood, then delivers an analysis, estimate or investment view with its reasoning. | How it is scoredAccuracy against expert references and figures reported later, calibration of any stated ranges, and the process rubric applied to the written work. | What it guards againstFluent answers that misread a source, invent a figure or quietly use information from after the as-of date. |
| 02SimulationHistorical market replay | What the system doesManages a portfolio or strategy through a replayed market, one decision date at a time, seeing only what was known then and paying realistic costs. | How it is scoredRisk-adjusted return after costs against matched benchmarks, drawdowns, turnover and any constraint breaches, with the decision log reviewed for process. | What it guards againstBacktests that flatter: look-ahead bias, survivorship, ignored costs and strategies tuned to a single period. |
| 03LiveForward evaluation on live data | What the system doesMakes forecasts and portfolio decisions on live market data. Each one is recorded and time-stamped before the outcome exists. | How it is scoredOutcomes as they arrive: forecast calibration, paper-portfolio results after modeled costs, and whether revisions follow the evidence. | What it guards againstMemorization and contamination. No system can have seen the answer, because it has not happened yet. |
| 04ClientServing a human user | What the system doesTalks with a user who brings a realistic question and circumstances. It must find out what it needs to know, explain options and risks plainly, and say when something is unsuitable. | How it is scoredExpert reviewers score suitability, accuracy, clarity and honest disclosure of risks and costs, against a written rubric. | What it guards againstConfident advice that ignores the person’s situation, hides risk or cost, promises returns or never says no. |
Integrity and anti-gaming
A benchmark is useful only while its scores mean something. These safeguards are built into the design of Alpha Bench.
The tasks behind official scores are never published. Public examples rotate, and retired tasks are replaced by new ones written to the same specification.
Published material carries canary strings so it can be filtered out of training data. Systems are also probed for signs that they have seen a task before.
Each team gets a limited number of scored runs per benchmark version, so results cannot be improved by trial and error against held-out tasks.
Practitioners score process, audit samples of automated scoring and investigate results that look too good to be true.
Every task set, data snapshot and scoring rule carries a version. Scores are always reported with the version they were earned on, and every change is logged in public.
Reviewers, task authors and the Foundation itself disclose ties to the teams being evaluated. No one reviews a system they are connected to.
Limits
A score is easy to over-read. These limits are part of the method, and they apply to every result.
Alpha Bench evaluates AI systems. Nothing it publishes is a recommendation to buy, sell or hold any security, strategy or product.
A strong score describes how a system behaved on our tasks. It does not predict what any strategy, fund or system will earn.
Being evaluated, or scoring well, does not mean the Foundation endorses a product, a provider or a service.
Scores describe behavior on our tasks, under our conditions, on a stated version. Other tasks, other markets or a later version may give different results.
FAQ
If something here is unclear, or you think we have it wrong, we would like to hear from you.
The methodology is published so it can be challenged. If you see a flaw, a blind spot or a better way to measure something, we want to hear it.