Why Backtesting Can Deceive You?

The Accuracy of the Backtest

Most quant strategy testers provide multiple different levels of modeling accuracy. An MT4 example below

It supports 3 types

• Control Points: A faster, less detailed method that approximates tick data using control points within a bar. It uses the high, low, and close prices to estimate intermediate movements.
• Open Prices Only: This method only considers the open price of each bar to evaluate trades, ignoring intraday price movements.
• Every Tick: This is the most precise and detailed backtesting option. It uses every tick data, simulating each individual price movement within a bar.

Based on your trading strategy trigger, if your strategy is sensitive to the price change, you should use tick-level data, while for less sensitive strategies,  control point can be used.

Different accuracy levels for the same strategy can sometimes produce vastly different results. Unless the strategy is intentionally designed to work with non-tick level data, backtesting should always use tick data if your strategy is built for real trading.

Modeling Quality

To begin with the importance of modeling quality, let’s look at an example backtest result below

Isn’t this backtest result perfect? The balance curve keeps growing over the whole period with almost no drawdowns. Now check another one,

These two reports look totally different, don’t they? What if I tell you these two are using the same strategy with the exact same parameters? Can you figure out what’s gone wrong?

The key difference is at the top right corner  Modelling quality, the first graph has a quality  n/a  , whereas the second has a  99.9%  quality.

Most backtesting platforms offer minute-level data as their highest granularity. For any missing data, simulation algorithms are typically used to interpolate price movements. The modeling quality largely reflects how much of the backtest relies on real minute-level data. The higher the modeling quality, the more reliable the testing results, and the closer they reflect actual performance in a live trading environment.

Rule of thumb: A good backtesting result should have a modeling quality of over 90%. Achieving higher levels of accuracy typically requires premium data, as superior data quality comes at an additional cost.

Cost, Cost, and Cost

Cost is very very important in the backtest. If trading were free, even the simplest strategies could perform well in real-world conditions. However, for active trading strategies, the difference between profitability before and after accounting for costs can be significant, often determining whether a strategy is viable in practice.

The general types of costs that must be considered are

1. Commission: The charge on open and close a trade. It generally relates to your position size.
2. Swap: The cost of holding a position overnight. While this cost can sometimes be positive, for most trend-following strategies, this cost can contribute significantly to profitability.
3. Execution cost: Put in other words, slippage cost. For market orders, when you place a trade at $71, you might end up buying/selling the position at $71.1 or $70.9, which means some profit can be taken by the slippage.
4. Spread cost: Some brokers have implicit charges on the spread. You need to relate to the fee compositions to understand how is it charged.
5. Ticket fee: Some brokers also charge a fixed fee when opening a ticket.

To illustrate the importance of cost, if you buy one lot size of USDJPY, which is $100,000, a slippage of 0.001 would result in an extra cost of 0.001 x 100,000 = $100, and the swap can range from $1 ~ 10 based on your holding period, the total cost is not-negligible compare to your profit.

Unfortunately, most backtesting tools do not factor in the real trading cost. The way we follow is to use backtesting to generate trades and built our own simulation system to incorporate real trading costs into the system.

Online Performance

Lastly, once you have everything in place, it’s essential to have a reliable system for deploying and monitoring your trades. It’s important to remember that backtesting does not fully capture future performance. Only in a real trading environment can you truly validate the profitability of your strategy.

On MQL, even for signals or EA with trading performance monitoring, there’s still multiple pitfalls you need to watch out:

Martingale or Grid Strategies

• How it Works: Martingale and grid strategies are common in automated trading and are designed to increase position sizes in response to losses, aiming to recoup losses on the next winning trade.
• Why It Shows High Win Rates: These strategies can often maintain a very high win rate and smooth profit curve in the short term because they’re structured to avoid realizing losses until necessary. However, they tend to risk significant drawdowns when market trends don’t reverse, potentially wiping out entire accounts in extreme cases.
• Signs to Look For: Check if the strategy uses position doubling or adds positions incrementally without clear stop losses. Look at the historical drawdown percentage on the Signals page—it can reveal if the strategy’s true risk tolerance is high despite a high win rate.

Short-Term Signal and High-Frequency Trading (HFT)

• How it Works: Some strategies generate high profitability through frequent small trades, relying on short-term price movements. High-frequency trading often produces small but consistent profits and can be programmed to close trades quickly to avoid long exposure to risk.
• Why It Shows High Performance: HFT strategies can produce high win rates and low drawdowns in certain market conditions because they capitalize on minute price changes. However, they require excellent execution speeds and may perform poorly in high-slippage or low-liquidity environments.
• Signs to Look For: Check the average trade duration and the frequency of trades. If the trades close in seconds or minutes, the strategy is likely high-frequency, which may perform inconsistently if market conditions change.

Unregulated Brokers with Manipulated Spreads and Slippage

• How It Works: Shady brokers can artificially adjust spreads, slippage, or execution prices to make it appear as though a strategy performs better than it actually does. They may offer abnormally low spreads or zero slippage in demo or test environments while charging higher spreads in live trading.
• Why It Shows High Performance: By reducing trading costs in backtests or demo accounts, these brokers make high-frequency or scalping strategies look more profitable than they would be in a live environment. Traders who go live might face much worse execution, which eats into profits and increases losses.
• Signs to Look for:
  • Check Broker Regulation: Use strategies or signals from brokers regulated by reputable authorities like the FCA (UK), ASIC (Australia), CySEC (Cyprus), or other well-regarded regulators. Regulated brokers are more likely to follow fair trading practices.
  • Compare Signal Performance Across Brokers: Look for the same strategy or signal across different brokers to see if performance is consistent. If a strategy only performs well on one unregulated broker, it’s a red flag.

Rule of thumb: There are no perfect trading strategies with both high win rates and low risk (low variance and low drawdown). If a strategy has a >50% win rate, it’s likely a negative-skew strategy that should have a profit factor < 1.0. Conversely, a strategy with a < 50% win rate should have a higher reward-to-risk ratio and a profit factor > 1.0.

How We Do Backtesting at @Lookatus

At @Lookatus, backtesting is just the starting point of our strategy development process. While we use MetaTrader’s strategy tester to confirm directional correctness, we recognize its limitations. To ensure the robustness of our strategies, we conduct a series of rigorous evaluations in an offline simulation system. Here’s how we refine our process:

1. Risk Evaluation: We thoroughly assess the strategy’s risk profile, including deviations and drawdowns, to understand its stability under various market conditions.

2. Trading Speed and Frequency Analysis: We evaluate whether the additional trades generated by the strategy justify the associated costs and contribute meaningfully to profitability.

3. Instrument Suitability and Correlation Analysis: By analyzing which instruments best align with the strategy, we ensure diversification and aim to minimize overall portfolio risk.

4. Trading Cost Assessment: We account for all potential trading costs, including commissions, swaps, spread costs, and execution costs, to determine the net profitability of the strategy.

5. Live Testing in Real Trading Environments: Finally, we deploy the strategy in a real trading environment to evaluate system reliability and measure actual trading performance under live market conditions.

Summary

In summary, backtesting can provide insights into a strategy’s potential but is often fraught with pitfalls that can mislead traders. For realistic evaluations:

1. Use the most accurate data available, like tick-level modeling for sensitive strategies.
2. Prioritize high modeling quality (>90%) to ensure the reliability of results.
3. Account for real-world trading costs, such as commissions, swaps, slippage, and spreads.
4. Be cautious with strategies relying on Martingale, grid, or high-frequency methods, as they often mask risks with high win rates.
5. Validate strategies through regulated brokers and compare performances in live environments to mitigate the effects of manipulated backtests.

Real-world validation is the ultimate test of a strategy’s viability, as even the most promising backtests may falter in live trading due to unaccounted factors.

About Us

We are @lookatus, a dedicated team of traders and engineers committed to creating REAL profitable, systematic trading solutions. With a strong foundation in quantitative analysis and cutting-edge technology, our mission is to deliver reliable, data-driven trading systems that capitalize on market opportunities with precision and consistency. Beyond building advanced tools, we are passionate about empowering traders through practical education, equipping them with real, actionable insights to navigate markets intelligently and successfully.

Contact us at:  haylookatus@gmail.com