Strategy family 07 / 12
Behavior, sentiment & flows
Prices move on what investors believe, feel and are obliged to do, not only on what assets are worth. This family covers sentiment and the narratives that draw attention, information-driven trading on news and disclosures, the footprints left by order flow and fund flows, and the fragility that builds up in crowded trades. The common thread is reading human and institutional behavior from signals that are noisy, fast and easy to misread.
Covers
- Sentiment
- Narratives
- Information-driven trading
- Order flow
- Fund flows
- Crowded trades
Why it is a demanding test
AI systems read text at scale, which makes this family both a natural fit and a trap: a fluent summary of the mood is not a tradable signal. It tests timestamp discipline, the ability to tell information already in the price from information that is new, and reasoning about who is forced to buy or sell, and when. Many behavioral signals decay quickly and are often small relative to trading costs, so speed, honesty about noise and cost awareness matter as much as insight.
What we test
What a system needs to do well in this family. Tasks are built so that each ability can be observed and scored on its own.
Scoring tone against a baseline
Measuring sentiment in news, transcripts and social posts relative to what is normal for that source, sector and speaker, rather than in absolute terms. Guarded language from a habitually upbeat management team can carry more information than the words suggest.
Tracing a narrative's life cycle
Judging whether a story is emerging, consensus or fading, and how much of it is already reflected in prices, positioning and analyst expectations.
Isolating what is genuinely new
Separating new information from restatements of known facts, and checking each item's timestamp against the moment the market could first act on it.
Reading order flow in context
Inferring buying or selling pressure from signed volume, order-book imbalance and large prints, while judging how much of it reflects hedging, rebalancing or liquidity provision rather than informed trading.
Mapping forced flows and crowding
Anticipating trades that happen regardless of anyone's view, such as those driven by mutual fund subscriptions and redemptions, ETF creations and redemptions, or scheduled rebalancing, and estimating how crowded holders would behave if a shared position had to be unwound.
Example task types
Illustrative designs that show the shape of the work. They are not items from the benchmark itself.
- Track: Analysis
Separate news from repetition in a feed
The system receives a timestamped stream of headlines, filings and transcript excerpts about one company across a trading day. It must flag which items carry new information, estimate the direction and rough size of each surprise, and state what was knowable at each point. A good answer discounts reposts and restatements and uses nothing published after the decision time.
- Track: Simulation
Manage a crowded book through deleveraging
In a historical replay that includes heavy redemptions and rapid deleveraging among funds with overlapping holdings, the system manages a long-short equity book. Using only point-in-time holdings, short-interest and flow data, it decides when and how far to cut exposure, paying realistic market impact for trading the same names as other funds that are unwinding. A good answer recognizes the crowding before the drawdown rather than after, and sizes its exit to the liquidity actually available.
- Track: Live
Forecast next-day reactions to fresh headlines
As headlines arrive during live trading, the system classifies each as new or already known and forecasts the probability that the stock outperforms its sector over the next trading day. Forecasts are locked at submission and scored over rolling windows. A good answer is well calibrated, avoids confident forecasts on ambiguous headlines, and does not mistake heavy coverage for new information.
- Track: Client
Talk through buying a trending stock
A user asks whether to buy a stock because it is trending on social media and everyone seems to be talking about it. The system must explain what sentiment and crowding indicators can and cannot tell them, describe the risks of buying into an attention-driven move, and relate the decision to the user's goals, horizon and capacity for loss. A good answer neither encourages nor dismisses the idea, discloses risk plainly, and is open about what it does not know.
How it is scored
Each run is scored on its outcome and on the process behind it, so a lucky result is not mistaken for a sound one.
- 01
Information coefficient by horizon
We measure the rank correlation between forecasts and subsequent returns at several horizons. The decay across horizons shows how quickly a signal loses value and whether the system's holding period matches it.
- 02
Net-of-cost return and turnover
Many behavioral signals require frequent trading, so returns are measured after commissions, spreads and estimated market impact, and reported alongside turnover. A signal that looks useful only before costs is labeled as such.
- 03
Drawdown during crowding episodes
We measure losses specifically in historical periods of rapid deleveraging, when crowded positions were unwound together. This shows whether risk controls hold when they are most needed.
- 04
Novelty and timestamp checks
We check that every piece of evidence was available at the decision time and that new items were correctly told apart from repeated ones. These errors are recorded even when a trade happened to work.
- 05
Calibration of short-horizon forecasts
On the Live track, stated probabilities are compared with observed frequencies using reliability curves across confidence levels. Overconfidence in noisy sentiment signals is a common and costly failure.
Diagnostics
Failure modes we diagnose
Recurring mistakes the tasks are built to surface. Each one is reported separately, so builders can see which to fix first.
- 01
Coverage mistaken for new information
Treating heavy coverage or social chatter as new information when it repeats what the market already knows.
- 02
Look-ahead through backfilled data
Using articles, transcripts or sentiment scores that were edited, backfilled or published after the decision time. Vendor sentiment histories can be revised, which makes this easy to miss.
- 03
Confusing reaction with cause
Treating sentiment that follows a price move as the cause of the next one. Much positive tone is a reaction to returns already realized.
- 04
Reading crowding as confirmation
Taking strong inflows and positive sentiment as validation when they signal crowding. A position can look safest just before many holders try to leave at once.
- 05
Overreading a single large print
Inferring informed buying from one block trade or a brief imbalance that may simply reflect hedging, index rebalancing or liquidity provision.
Environment and data
Tasks use news, filings, transcripts and social text as first published, with each item's release time preserved, together with holdings disclosures, fund flow reports, short interest and intraday trade and quote data. Holdings and flow data are released with lags, and the environment respects those lags exactly. Because horizons are often short, frictions weigh heavily: simulations charge bid-ask spreads, market impact that grows with trade size relative to available volume, and borrow costs on short positions.