Backtesting Discrepancies - Obvious Example

[markkm]

Hello,


DailyLossLimitExample_NT8.zip - created by ChelseaB does what it's expected to do in the backtest, but not in live / Market Replay (Playback).

https://ninjatrader.com/support/forum/showthread.php?p=490325

DailyLossLimitExample_NT8.zip is supposed to stop trading after $500 of daily loss. It stops trading in the backtest. Not in the live trading / Market Replay. All I did was download from the link above, compile (I did not change anything in the script), then backtest, then test in the Market Replay. The results are different.
In the attached screen shot of the Market Replay trade history, we can see that it's clearly not stopping from trading after $500 of daily loss.

Can someone please shed some lights here ?



Thank you,

[markkm]

Can someone from Ninjatrader support please answer?

I spent so much time backtesting, creating strategies, and now, all my work seem to be a waste of time.

I went through almost all the threads regarding backtest vs live discrepancies on this site, Elite trader, futures.io, and I just don't understand how the simple DailyLossLimitExample_NT8.zip that is supposed to always open a Long position with a 10 tick stop loss (stop order) if daily limit of $500 is not hit yet --- would fail on a live account.

This makes me wonder about all the strategies I've created.

[NinjaTrader_ChelseaB]

Hello markkm,

I am finding this is working with Market Replay and is preventing new orders after the LossLimit amount is reached.

https://drive.google.com/file/d/1o_RIqY-28e8GIJIUgmlQPTQ2jl97EhXx/view?usp=drivesdk

(I notice that I should have made the print in OnPositionUpdate print when currentPnL is equal to less than the LossLimit, not just less than it.. But this doesn't change the behavior of the entries.)

[markkm]

Thank you for getting back to me.
I noticed that you ran it for only 2 days. Can you please run it for a longer time?
The behavior of the entries is not consistent. It stops trading after $500 limit is hit one day, maybe 5 days, maybe longer, but not all the time.
I first implemented DailyLossLimitExample_NT8.zip in my strategy, and I noticed that it didn't stop trading after daily loss $500.
Then I just tested DailyLossLimitExample_NT8.zip , and I noticed the same behavior --- not consistently stopping after daily max hit ($500).
Can you please test it for a longer time (1 month) and just for the sake of consistency, can you please test it on NQ ?

[markkm]

Another example just FYI.
The attached screen was my strategy running in the simulator. I had implemented DailyLossLimitExample_NT8.zip into it. My daily loss limit is $250. It didn't stop trading.

[NinjaTrader_ChelseaB]

Hello markkm,

Ok, so you are confirming that you have experienced the script work correctly.

However, you feel that on later sessions, new entries are being allowed even though currentPnL is less than the LossLimit variable, is this correct?

I would recommend you print the currentPnL value to see what it is and print the value of the LossLimit. This will help you to understand the behavior.

Below is a public link to forum post that demonstrates using prints to understand behavior.

http://ninjatrader.com/support/forum/showthread.php?p=510979#post510979

This would be faster, than me not being able to reproduce the behavior on my end over a longer time period.

(I would also need to add these prints to look at the PnL level for later sessions as the currentPnL is reset on the first bar of a session. I don't even see what you are comparing in the screenshot to show its not working. What orders are you adding up for a session?)

Anytime you are experiencing behavior that you do not expect, always use prints to debug the script and understand the behavior.

[markkm]

However, you feel that on later sessions, new entries are being allowed even though currentPnL is less than the LossLimit variable, is this correct?

That is correct.

I would also need to add these prints to look at the PnL level for later sessions as the currentPnL is reset on the first bar of a session

I don't think the problem is in the reset of the variable. I changed

if (Bars.IsFirstBarOfSession)

by

(ToTime(Time[0]) >= ToTime(15,45,00) && ToTime(Time[0]) < ToTime(17,30,00)

. And still same problem.

I am in the process of making a video that shows from NT8 instal to the discrepancy.
I'll post it when I am done.

After I post the video, I will currentPnL prints OnPositionUpdate.

[NinjaTrader_ChelseaB]

Hello markkm,

I was not suggesting that you change when the currentPnL is reset.

I am suggesting you use prints to see why the entry condition is still evaluating as true.

Would you like assisting creating a print?

Instead of creating a video, provide the output from the prints.

I created a video for you just to prove that the condition works.
However, if I wanted assistance debugging, I would not create a video but would provide prints and output instead.

[markkm]

Sure, where would you like me to add the prints in DailyLossLimitExample_NT8.zip?

Below is the code source.

#region Using declarations
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.ComponentModel.DataAnnotations;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Input;
using System.Windows.Media;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Gui;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Gui.SuperDom;
using NinjaTrader.Gui.Tools;
using NinjaTrader.Data;
using NinjaTrader.NinjaScript;
using NinjaTrader.Core.FloatingPoint;
using NinjaTrader.NinjaScript.Indicators;
using NinjaTrader.NinjaScript.DrawingTools;
#endregion

//This namespace holds Strategies in this folder and is required. Do not change it.
namespace NinjaTrader.NinjaScript.Strategies
{
public class DailyLossLimitExample : Strategy
{
private double currentPnL;

protected override void OnStateChange()
{
if (State == State.SetDefaults)
{
Description = @"Enter the description for your new custom Strategy here.";
Name = "DailyLossLimitExample";
Calculate = Calculate.OnBarClose;
EntriesPerDirection = 1;
EntryHandling = EntryHandling.AllEntries;
IsExitOnSessionCloseStrategy = true;
ExitOnSessionCloseSeconds = 30;
IsFillLimitOnTouch = false;
MaximumBarsLookBack = MaximumBarsLookBack.TwoHundredFiftySix;
OrderFillResolution = OrderFillResolution.Standard;
Slippage = 0;
StartBehavior = StartBehavior.WaitUntilFlat;
TimeInForce = TimeInForce.Gtc;
TraceOrders = false;
RealtimeErrorHandling = RealtimeErrorHandling.StopCancelClose;
StopTargetHandling = StopTargetHandling.PerEntryExecution;
BarsRequiredToTrade = 1;
// Disable this property for performance gains in Strategy Analyzer optimizations
// See the Help Guide for additional information
IsInstantiatedOnEachOptimizationIteration = true;

LossLimit = 500;
}
else if (State == State.DataLoaded)
{
ClearOutputWindow();
SetStopLoss("long1", CalculationMode.Ticks, 10, false);
}
}

protected override void OnBarUpdate()
{
if (Bars.IsFirstBarOfSession)
currentPnL = 0;

// if flat and below the loss limit of the day enter long
if (Position.MarketPosition == MarketPosition.Flat && currentPnL > -LossLimit)
{
EnterLong(DefaultQuantity, "long1");
}

// if in a position and the realized day's PnL plus the position PnL is greater than the loss limit then exit the order
if (Position.MarketPosition == MarketPosition.Long
&& (currentPnL + Position.GetUnrealizedProfitLoss(PerformanceUnit.C urrency, Close[0])) <= -LossLimit)
{
//Print((currentPnL+Position.GetProfitLoss(Close[0], PerformanceUnit.Currency)) + " - " + -LossLimit);
// print to the output window if the daily limit is hit in the middle of a trade
Print("daily limit hit, exiting order " + Time[0].ToString());
ExitLong("Daily Limit Exit", "long1");
}
}

protected override void OnPositionUpdate(Position position, double averagePrice, int quantity, MarketPosition marketPosition)
{
if (Position.MarketPosition == MarketPosition.Flat && SystemPerformance.AllTrades.Count > 0)
{
currentPnL += SystemPerformance.AllTrades[SystemPerformance.AllTrades.Count - 1].ProfitCurrency;

// print to output window if the daily limit is hit
if (currentPnL < -LossLimit)
{
Print("daily limit hit, no new orders" + Time[0].ToString());
}
}
}

#region Properties
[NinjaScriptProperty]
[Range(0, double.MaxValue)]
[Display(ResourceType = typeof(Custom.Resource), Name="LossLimit", Description="Amount of dollars of acceptable loss", Order=1, GroupName="NinjaScriptStrategyParameters")]
public double LossLimit
{ get; set; }
#endregion

}
}

[NinjaTrader_ChelseaB]

Hello markkm,

It is much cleaner to provide an export instead of posting the full strategy code in text on the forum.

To export a NinjaTrader 8 NinjaScript do the following:

1. Click Tools -> Export -> NinjaScript...
2. Click the 'add' link -> check the box(es) for the script(s) you want to include
3. Click the 'Export' button
4. Enter a unique name for the file in the value for 'File name:'
5. Choose a save location -> click Save
6. Click OK to clear the export location message

By default your exported file will be in the following location:

• (My) Documents/NinjaTrader 8/bin/Custom/ExportNinjaScript/<export_file_name.zip>

Below is a link to the help guide on Exporting NinjaScripts.

http://ninjatrader.com/support/helpGuides/nt8/en-us/export.htm

However, the code for this script is already exported and available on the forums.

http://ninjatrader.com/support/forum/showthread.php?p=490325#post490325

When adding prints, add the print outside of any conditions above the condition.
This is demonstrated in the videos I have provided.

[markkm]

Ok, prints are already added to DailyLossLimitExample_NT8.zip as shown on the video you provided in this post.

I really just want to point out this discrepancy, so I think a video showing prints would tell more.
I will send you a link to the video when I'll have it done.

Let me know if I need to stop and add more prints (other than the 2x prints provided in the script), so we all understand why new entries were being allowed even though currentPnL is less than the LossLimit variable.

[NinjaTrader_ChelseaB]

Hello markkm,

We only need the Time[0], currentPnL, and LossLimit values for the specific bar where the unexpected order was submitted. (If Calculate is 'On bar close', this would be the bar directly previous to the order)

[markkm]

Ok, sounds good, will do. I'll let you know.
Thank you for your assistance.

[markkm]

Ok, I think I found the problem.

First of all, I just want to point out that DailyLossLimitExample_NT8.zip has been changed. I still have the version that was online yesterday. Today's version is different.

1. Stop loss was 10 tick, today's uploaded version is 5 tick (not a big deal though)
2. OnStateChange() has been all changed.

Is there a reason you made changes and uploaded a new version?

Now, let's talk about the problem:
1. Starting a strategy from a chart yields totally different results than starting it from the strategy tab.
2. Starting a strategy from a chart yields correct behavior (no more entries allowed if daily loss is reached), close enough to the backtest (without considering granular tick, fancy type of charts i.g Renko, Kagi, timeframe etc... It's been widely discussed on this forum)
3. I was starting strategies from the tab strategy, mostly because I have multimeframe / instrument strategies.

Please correct me if I am wrong, but why does it make a difference to start a strategy from a chart than starting it from the tab window ?

[NinjaTrader_ChelseaB]

Hello markkm,

The properties were all set to defaults. I just removed these to give better focus on the demonstration.

The stop loss was moved so that I can get quicker results in real-time.

The main change was in OnPositionUpdate

if (currentPnL < -LossLimit)

To

if (currentPnL <[B]=[/B] -LossLimit)

I tested adding to the Strategies tab of the Control Center and I am finding that 'SystemPerformance.AllTrades[SystemPerformance.AllTrades.Count - 1].ProfitCurrency' is always 0.

I'm researching this further.