Why every task is point-in-time

Every backtest is a story about the past, told by someone who already knows how it ended. The hard part of financial evaluation is keeping that knowledge out of the story. When information from the future leaks into a decision, even slightly, the results look better than anything a real investor could have achieved, and the evaluation stops measuring skill.

For an AI benchmark the stakes are higher still, because a language model may have read a great deal about what happened after the date a task is set in. So Alpha Bench treats point-in-time as a rule rather than a preference: every task sees only what was knowable at the moment of decision.

Two ways to see the future by accident

Look-ahead bias is the use of information that was not available when a decision was made. It is rarely deliberate. It usually enters through data that was cleaned, corrected or reorganized after the fact.

Survivorship bias is its close relative. It arises when the set of things being studied has already been filtered by what happened later. If you look only at companies that still exist, you are studying the winners and calling them a sample.

Both biases push results in the same direction. They make strategies look more profitable and less risky than they were, and they reward systems that pattern-match on hindsight over systems that reason from evidence.

Where the future leaks in

The leaks are usually mundane. These are among the most common.

Restated financials

Companies sometimes revise figures they have already reported, and many databases keep only the latest version. A test that uses today’s numbers for an earlier quarter sees the corrected figures, not the ones investors saw at the time. Quarterly results are also published weeks after the quarter ends, so using them on the last day of the quarter quietly borrows from the future.

Index constituents

Testing a strategy over the past decade on today’s members of an index means testing only on companies that became large or successful enough to be included. Many were added because they had already done well. Success is built into the sample before the test begins.

Delisted companies

Companies that went bankrupt, were acquired or were delisted often vanish from datasets. Leaving them out removes many of the worst outcomes, and some takeover premiums too, so both returns and risk are misstated. A value strategy that buys cheap stocks looks far safer when the cheap stocks that failed are missing.

News timestamps and prices

A headline stamped with a calendar date may have been released after the market closed. Pairing it with that day’s closing price assumes a trade was placed before the news was public. Time zones, trading hours and release schedules all matter, and a gap of a few hours can turn an ordinary signal into an impossible one.

Training cutoffs and contamination

This leak is particular to AI. A model asked to decide as of a past date may already know how the following months unfolded: which companies thrived, which crises came, how a merger was resolved. Hiding company names helps less than one might hope, because a model can often recognize a business from its numbers or its history. A historical task can never fully prove that a model has not seen the answer.

How Alpha Bench designs it out

No amount of care after the fact can catch every leak, so we try to make leaks impossible when tasks are built:

1. Every input records when it became known, not just the period it describes, and a task sees only inputs known at or before its decision time.
2. Universes are rebuilt for each date, with the index membership of that date, original rather than restated figures, and companies that were later delisted.
3. Where release times are uncertain, conservative publication lags apply, so information never reaches a system earlier than an investor could have acted on it.
4. Orders fill no earlier than the next available price after a decision, never at the price that produced the signal.
5. Historical tasks are refreshed on a rolling basis, and a held-out set is never published.

The strongest protection is to ask questions whose answers do not exist yet. In the Live track, systems submit decisions and forecasts on current data, and those decisions are scored later as outcomes unfold. No model can have read about this future in its training data, because it has not happened.

What it costs, and why it is worth it

Point-in-time evaluation is slower and more expensive to build. It needs data that keeps a record of its own history, and many small decisions about lags and timestamps that are tedious to get right. It also produces results that look less impressive than a naive backtest. The Live track adds a need for patience: forward results accumulate only as fast as markets move, and it takes many decisions across different conditions before skill can be told apart from luck.

We think the price is worth paying. An evaluation that lets the future leak in measures memory and hindsight. One that keeps it out measures what people actually need from an AI system in finance: judgment under uncertainty, using only the information available at the time.