WFO Results and NOT Consistant between tests of SAME Parameters

[rayko]

For the last couple days I`ve noticed my strategies have been producing WFO results that Do Not match the results for the same parameters and date ranges I tested and used last month.
So I`ve reproduced the problem using NT SampleMACrossOver and have attached screen prints of the results hoping you guys at support can explain what is going on.


File “01 Batch WFO.gif”
Shows the results of a 10period over 10Test day WFO of SamplaMACrossOver, for a typical batch of CME & CBOT futures while connect to Kinetic… Combined NET Results = $23,197.62


File “02 Batch WFO.gif”
Shows the results of the same test parameters as “01 Batch WFO.gif”, executed immediately after the previous test…. Combined NET Results = $11,061.06.


File “03 Batch WFO.gif”
Same as the previous example. This time the Combined Net Results = $20,072.91.


My observation is, between each test the perpetual contract data for each instrument is static (due to the specified date range of 1/1/2015 – 12/12/2015) and strategy SamplaMACrossOver is static, so I suspected the reason for the differences in the results between the tests could be the NT CACHE at C:\.. \NinjaTrader 7\db\cache.


So I shutdown NT and deleted the contents of C:\.. \NinjaTrader 7\db\cache, restarted NT and rerun the test again. File “04 Batch WFO.gif” shows the Combined Net Results = $21,560.32


Next, I repaired the DB using Tools>Options>Data>Repair DB.


I ran another test using the same parameters again, but NOT connected to Kinetic, File “05 Batch WFO.gif”… Combined NET Results = $21,830.88”


I ran another test using the same parameters again, but NOW connected to Kinetic, Combined NET Results = $19,798.76”


I am using NT 64bit 7.0.1000.30 running on a Windows 10 desktop


From my perspective, it looks to me that somehow the CACHE and/or DB is being corrupted and causes NT to produce random invalid WFO results.


Can you please provide me with assistance? I can easily reproduce these anomalies. Please let me know what I can do next.


Thank you

[rayko]

For some reason I am able to upload the screen captures, so I`ve upload a ZIP file in 2 parts

Here is Part 01

[rayko]

Here is Part 02

[rayko]

Hello NT Support,

Should I expect to hear back from you ?

The NT7 and NT8 documentation is really lite with no examples or tutorial on how to use the Ninjatrader WFO and more importantly on how interpret its results.

I spent an hour putting this forum thread together for you with screen prints to demonstrate these WFO anomalies and I`m hopping I didn`t waste my time doing it.

[NinjaTrader_CodyB]

Hello,
My apologies on the delay I am currently attempting to reproduce this on my end and have not been able to. I recommend to create a new database file and then try to reproduce this scenario.
You can create a new database by following the steps below.

Shutdown NinjaTrader and Go to the Start Menu
Select My Documents--> NinjaTrader 7--> DB--> NinjaTrader.SDF.
Right click on NinjaTrader.SDF and select "Rename." *Name it "OLDNinjaTrader.SDF."
Then restart the software and NinjaTrader will create a fresh database file to use
Unfortunately the following items stored in the old database will be lost - ATM Strategy templates, Session templates, Instrument Lists / Custom Instruments, historical trade execution data, saved Strategy Analyzer backtest results
To prevent running into situations where you can potentially lose important data, we advise to run regular backups via our inbuilt backup feature in NinjaTrader.

http://www.ninjatrader.com/support/helpGuides/nt7/creating_a_backup_archive.htm

While we can not make a recommendation on how for you to interpret the results of any of the backtests, optimizations or walk forward optimizations, as this would be considered trading advice, we are more than happy to answer any question that you may have on the strategy analyzer. I do apologize that you find the help guide lite on this subject. We are constantly working to improve our educational resources to assist further. If you have any recommendations on how we could improve or clarify the video and examples, found at the following links: NinjaTrader 7 Walk Forward Optimize A Strategy & NinjaTrader 8 Walk Forward Optimize A Strategy, please let us know and I can direct this towards the correct team that manages the help guide. The help guide is one of the major improvements we are striving to make in NinjaTrader 8 so any feedback on this is incredibly useful to our teams. Thank you.

[rayko]

Hello

Thanks for updating this thread. I did as you asked and unfortunately I am still able to produce different WFO results for the same parameters.

I think I have found the root of the bug in WFO.

I have attached a couple of screen scrapes.

"WFO Accumulate PnL.JPG" shows a WFO of SampleMACrossOver using YM ##-## for 1-1-2015 to 3-31-2015. I have used yellow highlighter to indicate the trade summary row showing accumulated total of $491.72 and below trade details row highlighting Cum. profit of $1,139.80.

The summary row of $491.72 is WRONG and should be $1,139.80. NT has added the detailed trades incorrectly.

"WFO Accumulate PnL 02.JPG" shows the same WFO with wrong summary row total of $2,789.60 and the below trade details row Cum. profit of $2,638.64.

You will also notice that the Cumulative profit for the WFO shown on the "Instrument Summary.JPG" is $7,012.60. This is also wrong, since this total includes incorrect subtotals of $491.72 & $2,789.60 when it should really include $1,139.80 & $2,638.64.

The WFO grand total is off by -$497.12 in as little as 3 months of trading.
(491.72 + 2789.6) - (1139.8 + 2638.64) = -497.12

I have only uploaded this one sample, however this Accumulation Error is done by the WFO all the time.

Can you pls investigate this asap.

This is a big problem for me and potentially many other WFO users.

I can`t trust the results of WFO if it can`t always add trades correctly.

Thank you

[NinjaTrader_CodyB]

Hello,
I do agree your results are incorrect. I have not been able to reproduce this though. So that we can investigate this further can you provide the data that you are using for the YM##-## from 1/1/2015 to 3/31/16. You can export this by using the export utility in the Historical Data Manager. Please see the following link on exporting historical data: http://ninjatrader.com/support/helpG.../exporting.htm

Also, do you experience this with the default criterion or is this only occurring with the Au_SQN criteria? Please send the custom Au_SQN criteria as well.

In addition what session template do you have set for the YM to be its default? This can be found by going to Tools> Instrument Manager> Select YM > Press Edit> Session Teamplate.

I look forward to your reply.

[rayko]

Hello,

I`ve uploaded Au_SQN. Au_SQN calculates Van Tharp's System Quality Number (SQN) optimizer type for NinjaTrader. However I can reproduce the error using Profit Factor criteria as well.

The forum won`t allow me to upload YM ##-## 1-1-2015 to 3-31-2015. The zip file for it is too big. Its over 2meg. However I did downloaded the perpetual contract from Kinetick. I can send it to you if you give me an address.

I used the default "CME US Index Futures ETH" session for YM in the WFO test.

Please let me know.

[NinjaTrader_CodyB]

Hello,
Please send the Data to platformsupport[AT]ninjatrader[DOT]com with this thread in the body of the email. Thank you.

[rayko]

Hello,

I`ve sent the email with a zip file of the data attached.

Please let me know.

[rayko]

Hello,

I ran another WFO test using 6A ##-## as input, strategy SampleMACrossOver with date range from 1-1-2015 to 3-31-2015. Same as my previous example using YM ##-##.

However this time I used max. Profit Factor as the optimization criteria.

This small test has 4 separate trade accumulation errors in the WFO trade summaries. I`ve uploaded screen scrapes of all 4 errors including the Instrument Summary. The errors are highlighted in YELLOW.

File:6A WFO WalkForward Line 2 Accumulation Error.JPG
Summary shows incorrect total of $185.76, while detail below shows total of -$980.20

File:6A WFO WalkForward Line 4 Accumulation Error.JPG
Summary shows incorrect total of -$376.36, while detail below shows total of -$346.36

File:6A WFO WalkForward Line 6 Accumulation Error.JPG
Summary shows incorrect total of $61.52, while detail below shows total of -$491.72

File:6A WFO WalkForward Line 7 Accumulation Error.JPG
Summary shows incorrect total of $2,143.64, while detail below shows total of -$1,851.72

File: 6A WFO Instrument Summary.JPG
Shows an incorrect grand total profit of $1,295.48, since its accumulated using the incorrect totals of lines 2,4,6 & 7. Its should be -$359.96.

Calculation: 1295.49 - sum(185.76, -376.36, 61.52, 2143.64) - sum(-980.20,-346.36, -491.72, -1872.72) = -$359.96

As you can see, the accumulation errors occur many times in a single WFO test. The errors are not limited to any one future contract and are not the direct result of the optimization criteria.

Please let me know if you want me to send you more examples using the basket of Futures I trade. I have them all listed in the screen scrapes under the Default Instrument list.

Please let me know.

[NinjaTrader_CodyB]

Hello,
I have attempted to reproduce the behavior you are experiencing by using your optimizer criterion and data, and have been unable to reproduce. To troubleshoot further I'd like to request that you create a secondary installation of NinjaTrader. This installation will be a brand new installation and will not have the same settings as your previous installation. With the instructions provided you will still have a copy of the installation with all of your settings. Please follow the instructions below to create a secondary installation to test with.

1. Copy your license key from Control Center -> Help -> License Key...
2. Close all running applications.
3. Then from the Start menu select --> Control Panel --> Add and Remove Programs.
4. From Add and Remove Programs select NinjaTrader 7 and click Remove.
5. Rename the NinjaTrader 7 folder (something like NinjaTrader 7 Back Up)
6. Once these steps are completed, download NT7 from the link below:

Download
Once you've reinstalled NinjaTrader, you will now have a new NinjaTrader 7 folder.

Please try to reproduce the behavior with this new installation. If you are able to reproduce please detail the steps you used to reproduce.

You can change back your previous installation by following the isntructions below:

1. Close all running applications
2. Then from the Start menu select --> Control Panel --> Add and Remove Programs.
3. From Add and Remove Programs select NinjaTrader 7 and click Remove.
4. Rename the NinjaTrader 7 folder to NinjaTrader 7 secondary
5. Rename the NinjaTrader 7 Back Up to NinjaTrader 7
6. Once these steps are completed, download NinjaTrader 7 from the link below:

Download

[rayko]

Hello,

I followed your steps and I now have a new installation of NT7 with No custom strategies and No custom optimization criteria installed.

I did not copy any of my ninjascript to the new c:\...\Documents\NinjaTrade 7 directory. I wanted to be sure there was no custom script that could cause this WFO accumulation error.

I setup my Interactive Brokers Account and Kinetick Account connections.

I setup my default instruments to what I usually use (see attached screen scrapes)

I setup my commissions for Futures with Interactive Brokers, Kinetick, Simulator & Replay

I connected to Kinetic and used the History Mgr to batch download ##-## data for all my default instruments from 12-1-2014 to present.

I ran the same WFO test as before using 6A ##-## and SampleMACrossOver, from 1-1-2015 to 3-31-2015 using "profit factor" as the optimization parameter (see attached screen scrapes)

WFO gave back 3 trade summaries with Accumulation ERRORS.

Could this problem stem from from License? I have a Lifetime Multi-Broker license.

Please let me know what to do next

[NinjaTrader_CodyB]

Hello,
Thank you for providing the results of that test. I am checking with our development team on this inquiry. Thank you for your patience on this.

[rayko]

Hello CodyB,

Can you please contact me directly so that I can show you how I reproduce the WFO accumulation error.


I have emailed [email protected] my contact information.


Thank you