How to Draw the Right Conclusions from Your Own Trading Data

Small samples mislead — and we don't expect them to

Tversky and Kahneman (1971) described the "belief in the law of small numbers": people expect small samples to look like the population they come from. They don't. A short run of results can look like skill or failure by chance alone.

The chart shows the idea with simple arithmetic: for a strategy that truly wins half of its trades, ten trades can produce anything from roughly a 30% to a 70% win rate by chance. Only with hundreds of trades does the observed rate settle near the truth. Real strategies also vary with market conditions, which needs even more data.

Outcome bias: judging the decision by the result

A good decision can lose and a bad one can win. If you rate decisions by their outcome you reinforce luck and punish sound process. The remedy is to grade the decision separately: did I follow my plan, size within my limits, and use my stop? Track that alongside profit and loss.

A practical way to test a strategy honestly

• Write the rules before testing: entry, exit, size, stop, and the market conditions you trade.
• Decide the sample size in advance, and don't stop when the result looks good.
• Test across different conditions (trending, choppy, volatile), not one lucky period. Scenario testing in a tool like Strategy Lab is one way to do this cheaply.
• Include costs: fees and slippage change the picture, especially at high frequency.
• Change one variable at a time and keep a record of versions.

The same care applies to behavior. "I overtrade on Fridays" needs many Fridays before you can believe it. Profiles that show a pattern together with how much evidence stands behind it are more honest than ones that show a single number.

Signs you are over-reading the data

• You changed the strategy after one or two losses.
• You found a "pattern" after looking at many slices of the data.
• You cannot state in advance what result would change your mind.

Educational content, not financial advice. Past results, simulated or real, do not guarantee future results.