Selection bias in trading research
Selection bias is why most crypto and TradFi trading research does not survive a fresh dataset: cherry-picked examples, missing denominators, and outcomes used as filters. Here is how it happens, and how EdgeDepth makes the common forms impossible to express.
Most trading research is selection bias wearing a chart. Not because researchers are dishonest, but because the easiest question to ask is the one that bakes the answer in: “show me the times this setup worked.” The moment you filter your examples by what happened next, you have stopped measuring the market and started measuring your own hindsight.
This page is about that failure in its several forms, and about a specific design decision: EdgeDepth Research makes the most common kinds of selection bias impossible to express, rather than warning you about them after the fact.
What selection bias actually is
Selection bias is any process that lets the outcome you care about influence which data you look at. In market study it shows up in a few recognizable shapes, and most real analyses commit more than one at once.
Cherry-picking, or outcome conditioning. You collect “bullish engulfing candles that pumped 5 percent,” admire how reliable they look, and never notice that you defined the sample by the pump. The engulfings that did nothing, or bled out, were filtered away before you counted. The hit rate you compute is not the hit rate of the setup; it is the hit rate of a sample you built to have a high hit rate.
Survivorship. The examples that reach you are the ones that survived a filter you did not choose. Delisted tokens vanish from the dataset, blown-up strategies stop posting, and the screenshots that circulate are winners because losers are not worth screenshotting. What remains looks far better than the population it came from.
Look-ahead. You use information that was not available at the moment you claim to have acted. Filtering on a future return is the purest example: “times VPIN spiked and price then dropped 2 percent” quietly reaches into the future to pick the past. Every backtesting guide warns about look-ahead bias; almost every informal backtest commits it anyway.
The missing denominator. A count with no denominator is not a measurement. “This happened 40 times” means nothing until you know 40 times out of how many eligible chances. Drop the denominator and a common event and a rare one look identical.
Multiple comparisons. Test enough conditions and some will look significant by chance alone. Point at a handful of moments that all moved, and with enough features to describe them, they will always share something. That shared thing is not a discovery; it is arithmetic.
Why it is so easy to do by accident
The honest version of a question is almost always more work than the biased version, and it usually returns a less exciting number. “Setups that worked” is one filter. “Every occurrence of the setup, and the full distribution of what followed” is a scan, a denominator, and a result that is often disappointing precisely because it is true. Human attention does the filtering for free: the moments you remember are the ones that moved, so any pattern you extract from memory is already conditioned on the outcome before you write a single line of code.
The result is a research culture, formal and informal, where most claims cannot be checked and the few that can rarely reproduce. A number gets posted, the data moves on, and even the author cannot recompute it.
How EdgeDepth removes the failure modes
EdgeDepth Research is built so that the common forms of selection bias are not discouraged, they are unrepresentable. The honesty is enforced by the query grammar, not by a disclaimer.
Outcomes can never be a filter. Outcome fields are computed forward from
each anchor and displayed, but they cannot appear in a predicate. Ask the
engine to filter on what happened next and it returns a named error,
OUTCOME_IN_PREDICATE, and refuses. To study “setup, then move,” you state the
move as the next step of a sequence, which makes the completed pattern the
thing you count, and the outcome distribution is then reported over every
occurrence of that pattern, winners and losers alike. You cannot select on the
future because the grammar has no place to put it.
Every count carries its denominator. A result is N occurrences across M eligible symbol-buckets scanned, never a bare number and never a win rate. The denominator is part of the answer, so a condition that holds almost everywhere prints as common and is visibly uninformative.
Absence is reported, never coerced to zero. When a feature is not yet warm at a given minute, that bucket leaves the denominator and the exclusion is stated. Missing data shrinks the sample honestly instead of being quietly filled with a convenient zero.
Multiple comparisons are surfaced, not hidden. When you point at several moments and ask what they share, the tool tells you on the surface that any intersection is hypothesis generation conditioned on the moments you chose, then runs the corrective scan: every occurrence of the shared setup, with the outcome distribution over all of them. Expect the pattern to deflate. That deflation is the product working, not failing.
Anyone can check you. Every result carries a five-part reproducibility key, so the same question returns byte-identical bytes on any account, any day. A published report pins its result to that key forever. This is the part selection bias cannot survive: a stranger can rerun your claim and diff the bytes. You cannot quietly re-pick your sample when the sample is a canonical hash.
The same restrictions hold whether a person uses the browser, an application uses the public API, or an AI agent calls the read-only MCP tools. Changing the client does not open a second, looser grammar. See Research from the browser, API and MCP.
What honest research looks like instead
State the setup as of the anchor, with no reference to what came after. Count every occurrence across the record, with the denominator attached. Read the outcome distribution over all of them, not the tail you would have remembered. If the setup is real, it holds up over the full population; if it only looked special on the examples you started with, the full distribution says so, plainly, and that is a finding too. A verified zero is publishable. So is a verified “this happens constantly and leads nowhere.”
None of this makes the market predictable. It makes your claims about the market checkable, which is a rarer and more useful thing.
Occurrences describe only the recorded markets, dates and eligible buckets reported by the scan. They are observed examples, not a guarantee. Nothing here is trading advice.