Methodology
From an idea to a trading system you can trust
Writing a trading system is quick. Building one that does under real conditions exactly what its test promised is a different matter.
Between the two lie hundreds of checks, thousands of test runs, and a discipline that allows no shortcuts. This page does not show what our systems trade. It shows how we work: which checks a system must pass before it even runs on a demo account, and why almost every idea is rejected along the way.
Principle
A result only counts when it survives every cross-check
A beautiful backtest curve proves nothing. It can appear because the system accidentally looks into the future, because costs are set too low, because one lucky trade carries everything, or because the test ran on the wrong data. We have met each of these traps in practice. Each now has its own fixed check.
- Measure, never assume. Every result states its basis: period, data source, cost model, version.
- Define first, then test. What counts as success is fixed before the test runs. Polishing results afterwards is ruled out.
- Every measurement needs a benchmark. A result only matters if it beats what chance would have produced under the same conditions (null model).
- Never overwrite, always add a version. Every version, setting and report is kept and traceable.
- A found error is never just noted. It gets a root cause, a fix and its own test, so it can never return.
Chain of checks
The path of an idea
- 01
Specification before code
Before a single line of code is written, the rule is written down and approved. What is not described unambiguously cannot be tested unambiguously.
What this means for you: You know in advance exactly what a system should do, and can measure it against that later.
- 02
Two independent implementations
Every strategy exists twice: as a research model in a separate analysis environment and as an Expert Advisor in the trading platform, built independently. Only when both produce the same result trade by trade is the implementation accepted. If they differ, one of them is wrong, and we keep digging until we know which.
This parity check has caught, among other things: signals evaluated at the open instead of the close of a candle, dropped weekend candles, rounding that differed from the broker's, an event calendar that ended earlier in the test than in live trading, and settings that had no effect at all.
What this means for you: A mistake in our work shows up, because two independent calculations must produce the same result.
- 03
Compile without a single warning
The code must compile with zero errors and zero warnings. This is not assumed; it is executed and read on every build.
What this means for you: No program enters testing if it already raises warnings when it is compiled.
- 04
The look-ahead test
The most dangerous backtest error is invisible: the system uses information it could not have had at that moment in real trading. We therefore delay all data by one extra candle. If the result collapses, the system was looking into the future, and the idea is finished.
What this means for you: A result that only looks good thanks to impossible foresight never reaches you.
- 05
Development data and verification data never mix
This step is one of the most important and is widely underestimated. If you develop a system on the whole history and judge it on the same history, you can make almost any system fit: parameters are turned until the past looks good. In a report this looks convincing; in live trading it often collapses. This practice is called overfitting. Some developers use it knowingly, many unknowingly.
We therefore separate two periods. The first (in-sample) is where we develop and select. The second (out-of-sample) stays locked until the end. Only when rules and selection are frozen is it opened, exactly once. There is no second attempt and no re-tuning. An idea that fails here is not repaired; it is discarded.
The advantage: to the system, the verification period behaves like a real future. It has never seen it, and it is played through tick by tick on real market data (data quality 99.9 %), with spread and costs. This lets us map our development into a realistic future that is 99.9 % tick-based, before any capital is risked in live trading. This is not optimising for the past; it is the counter-proof to it.
This evidence is not a promise of returns either. It shows whether a system holds outside its development data, and so whether it is more than a well-fitted pattern.
What this means for you: What you see was confirmed on data that was not available during development.
- 06
Monte Carlo: how much was luck?
History is only one of many possible sequences. From the same trades we generate thousands of alternative histories: shuffled trade by trade, resampled with replacement, and reshuffled in blocks so that connected losing phases stay together. This shows which drawdown is realistic and how likely a loss would have been. The block-wise variant in particular shows how much risk is underestimated when trades are only shuffled one by one.
What this means for you: You see not only the one path history took, but also how badly it could have gone in a different order.
- 07
Cost stress
Costs (spread, commission, swap) are the real bottleneck of every trading system. Every result is recalculated with costs raised step by step. A system that only works with ideal costs is rejected.
What this means for you: A system must stay viable under worse trading conditions than today, otherwise it is not released.
- 08
Robust, not optimal
We do not look for the single best setting but for stable regions: a result must hold for neighbouring settings, across different market phases, and must not depend on a few years or a single trade. Every equity curve and yearly distribution is also checked by eye, because some weaknesses show up in no statistic.
What this means for you: No result hangs on a lucky hit or on a single setting.
- 09
Forward test under real conditions
Only after all checks does a system run on demo accounts with real brokers. The start follows a checklist with every item evidenced individually, and live entries are continuously compared with the test.
What this means for you: Before real capital is involved, the system must show under market conditions that it behaves as it did in testing.
Updates
No update without proof of identity
Even a pure speed-up or code clean-up may not change behaviour. For every new version, the old and the new version run side by side over all productive configurations, and every column of every single trade must be identical. As a negative control, the checking tool must flag two deliberately different versions as different.
Afterwards there is exactly one active version. The version number is built into the program and shown in its panel, so it is never unclear what is running.
One example: reducing the work done on every price tick made test runs more than twice as fast. It was adopted only after every trade was proven identical.
Observers
Observe instead of believe
We do not rely on a system's own reports to know what it really does. Dedicated observer programs run alongside on the test accounts and independently record every event: new orders, modifications, stop and target moves, deals and account snapshots. This lets us compare test and reality event by event, and analyse the behaviour of third-party systems whose code we cannot see.
The observers are checked too. Errors in their recordings were found and fixed in a new version, while the old version was kept unchanged.
Infrastructure
The test infrastructure
Many checks mean many test runs. They run in parallel on several isolated platform instances:
- Job queue. It distributes the planned runs.
- Scheduler. It limits parallel tests during the day and releases more capacity at night.
- Hard resource cap. All tests run on an invisible desktop under a cap on CPU and memory, so the workstation is never noticeably slowed down.
- Automatic report checks. Does the number of deals match the summary, is the report complete? A test that runs empty is detected and repeated, never silently accepted.
- Append-only logs. Handoff log, findings log and status matrix: each work session continues exactly where the last one stopped.
- Knowledge graph. Completed findings are also stored as a graph, so connections between insights stay visible.
Dashboard
Only verified reports enter the dashboard
For live operation we built our own equity dashboard. It is generated only from complete, verified test reports. Before a report may enter the dashboard, it passes its own test bench: period, modelling, spread, settings against the expected configuration, trade list, yearly distribution, and a cross-check against the parity result.
The dashboard exists in several editions: interactive in the browser, a static version without scripts (works offline and in messenger previews), and a shareable online view.
Try an example backtest yourself →News reader
A protection layer, not a signal source
Markets behave differently around major economic releases. Our news reader is therefore not a signal source but purely a protection layer: it pauses new entries in a window around scheduled events.
It works with two independent sources: a fixed event table that is identical in backtest and live trading, and a live economic-calendar feed. The live feed is throttled, has a fallback and shows its state in the panel. For each market, event times are derived from official sources, never assumed.
The news reader is tested candle by candle: hits and misses, time base, window width, coverage, equality between test and live, and every display state including table expired.
Error catalogue
Errors our checks have caught
Each of these would have produced convincing but false results:
- Future data: a filter used a candle that was not yet closed at decision time.
- Time zones: a wrong daylight-saving rule shifted trading hours.
- Silent fallback: an invalid setting was ignored without any message, so an entire test series ran with different settings than intended.
- Wrong data source: tests ran on the wrong price data set and were later declared invalid.
- Empty reports: the test platform started but produced no result.
- Encoding: a file format garbled a path.
- Resource clash: parallel tests competed for the same network port.
- Formula drift: two versions of the same indicator calculated slightly differently.
- Damaged data: a price file held only a fraction of the expected ticks and was restored from backup.
In short
Why we trust the few ideas that pass
A trading system that runs without errors is neither luck nor a single good idea. It is the result of a chain of checks, each of which rules out one specific kind of self-deception. Most ideas do not survive this chain. That is exactly why we can trust the few that do.
What happens next?
Two ways forward, depending on what you need.
Discuss our checks
Want to understand how we check a system? We are happy to walk you through the chain using an example.
To the contact form →Request development
Need an Expert Advisor, a copy system or an integration? Tell us about your project.
To the contact form →