No totals in Strategy Analyzer? All passes not run when I click 'Run'?

[ntdev]

Q1: Is there a trick to getting the strategy analyzer when using Backtest Optimization?

It shows that it runs the 9 tests for my parameters but nothing shows in the totals.



NOTE: I do see the totals when I run an individual Backtest but not when running an optimization.

Q2: It also doesn't seem to run each test until I click on the individual row. It might run the first one but then it actually runs that pass only after I click on the row in the first table which then changes the Summary, etc. results to show what happened during that pass. Is there something that I need to set in my code for it to run all tests when I click "Run" and not wait?

NOTE #2: Nothing strange in the log except that there was an open position at the end of the day which was closed by the strategy...

≋≋ OnOrderUpdate(【0】, Exit on session close) | order | orderId='NT-00829-263' account='Backtest' name='Exit on session close' orderState=Submitted instrument='CL 05-21' orderAction=BuyToCover orderType='Market' limitPrice=0 stopPrice=0 quantity=6 tif=Gtc oco='' filled=0 averageFillPrice=0 onBehalfOf='' id=-1 time='2021-03-30 23:59:34' gtd='2099-12-01' statementDate='2021-04-11'

NOTE #3: The same thing happens for both optimizers too.

Please let me know if there's something else I can check.

Version 8.0.24.2 64-bit and I have 48GB of RAM on my 12-core Intel processor

[NinjaTrader_ChelseaB]

Hello ntdev,

May I confirm this also occurs when testing the SampleMACrossover included with NinjaTrader?

If you comment all custom logic and enter long when flat and exit long when long, are there still no results?

[ntdev]

Thanks Chelsea for your quick response and on a Sunday evening...

May I confirm this also occurs when testing the SampleMACrossover included with NinjaTrader?

Yes, SampleMACrossOver works as expected -- all profit numbers are shown and every pass runs with a single click of the "Run" button.

If you comment all custom logic and enter long when flat and exit long when long, are there still no results?

That's not possible as my code is literally 10k lines at this point with very complex logic, all running on tick data.

- - - - - -

OK. You've got me thinking about what's different with my code.

#1. I exclusively use SubmitOrderUnmanaged() -- no Atms or EnterLong() / EnterShort()

Maybe I'm using too low level of an api to make my trades and it's not supported or compatible with the optimize function of the strategy analyzer? Or a bug?

#2. Thread safe & statics -- I don't see any static variables, locks, Mutexes or anything that would stop multiple threads from running at the same time.

And I don't see any errors from that in the Visual Studio output log when running under the debugger.

Are there other things to look for?

#3. I do have 20+ parameters and they are of all different types, including some enums, but it runs fine and displays fine when I run on test at a time without the optimizer. NOTE 3a: I added my enum type to SampleMACrossover and there's no problem.

#4. I use different group names for some of my parameters than the sample.

Sample:

GroupName = "NinjaScriptStrategyParameters"

Me:

GroupName = "NinjaScriptStrategyOrderProperties"
GroupName="STRATEGY TESTING"
GroupName="Trading"

NOTE 4a: I added different GroupNames to SampleMACrossover and there's no problem.

Unrelated Question: Is there a list of all the predefined strings that are supported in NT?

• NinjaScriptStrategyParameters
• NinjaScriptStrategyOrderProperties
• NinjaScriptGeneral
• NinjaScriptLines
• NinjaScriptSetup
• NinjaScriptStrategyGeneratorProperties
• NinjaScriptTimeFrame

#5. Is it because I run with Tick data? NOTE 5a: I ran SampleMACrossover with tick data and there's no problem.

NOTE 5b. My strategy also creates a second tick data series to use when the main series is not Tick/1 but I usually run with Tick/1 because I don't understand the conflict with running orderfill resolution set to 'high' and having the primary series as something other than tick/1. I could ask that question separately.

[NinjaTrader_ChelseaB]

Hello ntdev,

Commenting out all logic and running a simple test with entry and exit will tell you if the issue is with your custom logic or an issue with the platform.

SubmitOrderUnmanaged() is part of NinjaScript (compiled internally in NinjaTrader), and is not part of the API (Application Programming Interface) that gives access to external applications through the API. Below is a link to a forum post on the API.
https://ninjatrader.com/support/foru...626#post793626

Creating new threads is not directly supported by NinjaTrader, but if you are creating new threads, there would be extra considerations for that.
Everything in the help guide under Indicators and Strategies is thread safe. Most information under Common is thread safe (but not everything). Information under Accounts or other sections of the help guide may not be thread safe.
As an example, a lock is used when accessing the Accounts collection.
https://ninjatrader.com/support/help...rderupdate.htm

If there is an issue with the inputs, create a new test script and add the inputs and only logic to enter and exit. Can the behavior be reproduced?
If so, please provide the test script with the inputs and a simple entry and exit without any custom logic.

There is not a list of the language resource strings used internally by NinjaTrader.

Strategies can be run on tick data. If the strategy is not running on tick data specifically, this would be because the script was implicitly written to not run on specific bar types, or the logic is not evaluating in a way that places orders.

My recommendation is that you debug the code and find the line that is causing the issue. This would be faster than attempting to guess the issue.
https://ninjatrader.com/support/foru...121#post791121

[ntdev]

OK. I'm stuck. I spent the whole night looking at this and see nothing more to try or check.

SubmitOrderUnmanaged() is part of NinjaScript (compiled internally in NinjaTrader), and is not part of the API (Application Programming Interface) that gives access to external applications through the API. Below is a link to a forum post on the API.

No external applications involved here. I have a class derived from injaTrader.NinjaScript.Strategies.Strategy, like the sample.

I must be missing the point of why you included this. Does this mean that the SubmitOrderUnmanaged() function is in a different (NTDirect.dll) DLL?

I'm confused. I can use SubmitOrderUnmanaged() in a strategy, right? For my logic and to be able to control the trade like I want, I can't do it any other way.

Creating new threads is not directly supported by NinjaTrader

No threads are part of this process.

I only mentioned "threads" because I figured that the optimizer would spin up multiple instances of my Strategy derived class and run them in parallel.

There is nothing in my code that should have threading issues -- I commented out anything in question and, anyway, the other behavior of the optimizer is weird.

One weird thing I see when tracing (I added a GUID to my class so that I can track instances) is that it creates one instance, Instance A, with one value then terminates it and creates another instance, Instance B, with a different parameter.

1. ≋≋ Strategy1.OnStateChange(Configure) ≋≋ ➽➽❰ES 06-21❱ ⊢f0c84706-dfcb-4a06-8e05-456c50130704⊣ Instance A
2. _{ProfitTargetTicks=40} Instance A
3. ≋≋ Strategy1.OnStateChange(Terminated) ≋≋ ➽➽❰ES 06-21❱ ⊢42234260-73c6-4e1a-9cb8-00c9946653da⊣ Instance A
4. ≋≋ Strategy1.OnStateChange(Configure) ≋≋ ➽➽❰ES 06-21❱ ⊢53e1d3b0-deac-4f74-b8ed-a6ce82a74db1⊣ Instance B
5. _{ProfitTargetTicks=60}Instance B
6. ≋≋ Strategy1.OnStateChange(DataLoaded) ≋≋ ➽➽❰ES 06-21❱ ⊢53e1d3b0-deac-4f74-b8ed-a6ce82a74db1⊣ Instance B
7. ≋≋ Strategy1.OnStateChange(Historical) ≋≋ ➽➽❰ES 06-21❱ ⊢53e1d3b0-deac-4f74-b8ed-a6ce82a74db1⊣ Instance B
8. ★☆★ 0. OnBarUpdate(Default::Historical) [03/14 18:00, vol=60, *3937.5*] ➽➽❰ES 06-21❱ Instance B
9. ...
10. ≋≋ OnOrderUpdate(【0】, Exit on session close) | order | orderId='NT-00687-7789' account='Backtest' name='Exit on session close' orderState=Filled instrument='ES 06-21' orderAction=Sell orderType='Market' limitPrice=0 stopPrice=0 quantity=4 tif=Gtc oco='' filled=4 averageFillPrice=3966.25 onBehalfOf='' id=-1 time='2021-03-31 16:59:59' gtd='2099-12-01' statementDate='2021-04-12' ➽➽❰ES 06-21❱ ⊢53e1d3b0-deac-4f74-b8ed-a6ce82a74db1⊣ Instance B
11. ≋≋ Strategy1.OnStateChange(Terminated) ≋≋ ➽➽❰ES 06-21❱ ⊢53e1d3b0-deac-4f74-b8ed-a6ce82a74db1⊣ Instance B

Instance A, or a strategy with this test is not run until I click on the line in the strategy analyzer. I could but debugging into the sample to see how it breaks things down.

If there is an issue with the inputs, create a new test script and add the inputs and only logic to enter and exit. Can the behavior be reproduced?

I already did that and there were no problems.

Strategies can be run on tick data.

Again, not a problem. My code looks at the Bar and creates another series if necessary but this code doesn't come into play now because there is only one series and it's a Tick series.

My recommendation is that you debug the code and find the line that is causing the issue. This would be faster than attempting to guess the issue.

I'm out of things to try.

The only thing I have left is to make the sample run with SubmitOrderUnmanaged but I doubt it's reproducable.

It could be something else -- I'll turn on order tracing right now and run again, looking at that. That's one thing I didn't do.

TraceOrders = true;

For example, I had previously seen examples where SubmitOrderUnmanaged returned null because a buy stop price was already inside the bid/ask but in this case it just went on.

EDIT: nothing to report from TraceOrders = true. It all looks good but only runs 1 of 4 tests when I hit "Run." And it only sets up (State.Configure) with 2 of the 4.

--------

EDIT #2: I read stuff about 32-bit installs and how it's not optimized and it feels a bit like I'm experiencing this.

https://ninjatrader.com/support/help...ionPerformance

"NinjaTrader will still take advantage of all available threads for it's processing, however those resources available would be determined at the start-up of the NinjaTrader platform. So if your virtual resources would have changed while you were in a working session, then please restart fresh to ensure performance will be optimal."

Maybe I have so much memory that NinjaTrader has a bug which causes it not to see that correctly?

Maybe there's some serialization problem with the attributes of my strategy and the way the optimizers creates the next instance?

It would be helpful if I understood the implementation on your end, especially if I'm now trying to debug an NT bug.

Is there an API, internal or not, to report on how NT sees the system resources?

[ntdev]

My settings in State.SetDefaults:

Calculate                                   = Calculate.OnEachTick;
EntriesPerDirection                         = 1;
EntryHandling                               = EntryHandling.AllEntries;
IsExitOnSessionCloseStrategy                = false;
ExitOnSessionCloseSeconds                   = 30;
IsFillLimitOnTouch                          = false;
MaximumBarsLookBack                         = MaximumBarsLookBack.TwoHundredFiftySix;
OrderFillResolution                         = OrderFillResolution.High;
Slippage                                    = 0;
StartBehavior                               = StartBehavior.ImmediatelySubmit; //.WaitUntilFlat;
RealtimeErrorHandling                       = RealtimeErrorHandling.StopCancelClose;
StopTargetHandling                          = StopTargetHandling.PerEntryExecution;
BarsRequiredToTrade                         = 2;
IsInstantiatedOnEachOptimizationIteration   = true ;
IncludeTradeHistoryInBacktest               = true ;

And, yes, some of them are overridden in the Strategy Analyzer.

[NinjaTrader_ChelseaB]

Hello ntdev,

Are you having an issue with no results when a backtest completes?
(This would not be related to threading, performance, or running 32-bit instead of using the 64-bit icon. This would be due to something in the code of the script.)

Or are you having performance issues and the test is taking a long time, but does have the proper results when the test is complete?
(This would be related to threading, performance, 32-bit vs 64-bit, memory, CPU, etc)


You are not out of things to try. I am suggesting that you comment out of all of the logic and put in an simple entry and exit. This would be the start of debugging your code.
May I confirm that you have done this and the simple test also does not produce results?
These are the steps all developers take to address a logic error in code.


The API is for external applications. Its application interface (API is Application Programming Interface) This gives an external application an interface to communicate with NinjaTrader through programming..
There are no API commands for viewing system resources.

NinjaScript are internally compiled into NinjaTrader and do not use the API. These are not separate applications that are trying to interface with NinjaTrader. These are already in NinjaTrader.
There are no NinjaScript methods for viewing system resources.

NinjaScript resource utilization can be viewed in the New -> NinjaScript Output -> NinjaScript Utilization Monitor.

https://ninjatrader.com/support/helpGuides/nt8/index.html?output.htm#NinjaScriptUtilizationMonitor

[ntdev]

Are you having an issue with no results when a backtest completes?

Works perfectly when I do a single backtest.

Or are you having performance issues and the test is taking a long time, but does have the proper results when the test is complete?

No, no performance issues. The problem is that NinjaTrader just runs a single test, like is as documented for the 32-bit environment.

I see the trace output which you can get via the Print() statement or put breakpoints in Visual Studio. I can verify that I only get to OnStateChange(DataLoaded), OnBarUpdate(), etc. for 1 run with 1 set of parameters -- the other runs do not happen until I click on them each individually.

This is why I think that the bug is on the Ninja Trader side -- maybe it's not recognizing my hardware correctly?

64-bit icon

Yes, I am for sure running the 64-bit version as verified Help / About and posted in my first message.

You are not out of things to try. I am suggesting that you comment out of all of the logic and put in an simple entry and exit. This would be the start of debugging your code.
May I confirm that you have done this and the simple test also does not produce results?
These are the steps all developers take to address a logic error in code.

I've already worked on this for 14 hours today -- I'll continue to pull my code out but I've tested the main things.

The longer I do this the more likely the bug is on your side. I must be setting up my situation in a unique way that causes it not to try the 2nd test.

Again, I wish that I could get insight from the dev team as to what they do when preparing or what checks they have to abort the running of multiple instances.

NinjaScript resource utilization can be viewed in the New -> NinjaScript Output -> NinjaScript Utilization Monitor.
https://ninjatrader.com/support/help...izationMonitor

Thanks, I'll check that out and report back tomorrow.

[NinjaTrader_ChelseaB]

Hello ntdev,

We have previously determined that when optimizing the SampleMACrossover included with NinjaTrader this behavior cannot be reproduced, is this correct?
This would indicate the 64-bit version is being used and that the optimizer is performing as expected.

What line of code is causing a bug?

Do you have specific code that would interfere with multiple instances?
Does the script have static properties, invalid code in OnStateChange(), improper inheritance, improper initialization (outside of State.DataLoaded)?

If the issue is performance or resources, the optimization would not be able to finish due to resources.

Below is a link to a forum post on the instances.

https://ninjatrader.com/support/forum/forum/historical-beta-archive/version-8-beta/85086-why-is-state-terminated-run-when-another-different-indicator-is-loaded?p=726288#post726288

NinjaTrader creates instances and runs backtests on them, the same as with the SampleMACrossover included with NinjaTrader. Do you have code that specifically cannot be initialized?

[ntdev]

We have previously determined that when optimizing the SampleMACrossover included with NinjaTrader this behavior cannot be reproduced, is this correct?

What line of code is causing a bug?

Do you have specific code that would interfere with multiple instances?
Does the script have static properties, invalid code in OnStateChange(), improper inheritance, improper initialization (outside of State.DataLoaded)?

Yes, there is no problem with SampleMACrossover but it's such a simple example so that's Apples and Oranges.

If I had known which line of code was causing the bug then I'd fix it. LOL

NinjaTrader creates instances and runs backtests on them, the same as with the SampleMACrossover included with NinjaTrader. Do you have code that specifically cannot be initialized?

This was getting closer.

So, after more debugging and looking at the states of NT objects with Visual Studio I've worked around the Ninja Trader bugs.

Here are some of the bugs and/or issues and/or observations that I noticed.

BUG #1. Instrument is NULL during OnStateChange(State.Configure) for all but the last call.

#1b. Since there is only 1 call during a normal Backtest, this wasn't an issue (because this 1 call was both the first and last call) but I was using the Instrument and other things to create and initialize my own volume profile code among other things like creating a tick series if it was not there.

HACK#1 to work around this bug I to added the same initialization code during both State.Configure AND State.Historical which always has Instrument set.

BUG #2. Strategy.IsEnabled is false during Optimization -- I relied on this.

BUG #3. Strategy.IsInStrategyAnalyzer is false during Optimization -- I also relied on this.

HACKS#2&3 to work around this bug is to add the following code to my initialization code which is sometimes called during State.Configure and sometimes called during State.Historical.

if (!IsInStrategyAnalyzer && Account != null && Account.Name == "Backtest")
     IsInStrategyAnalyzer = true;

if (IsInStrategyAnalyzer)
     IsEnabled = true;

There are a few other things but I've only got one eye open at this point.

[NinjaTrader_ChelseaB]

Hello ntdev,

Specifically, we use SampleMACrossover as a baseline test to know if we need to look for a specific line of code, or look for something within the platform.

SampleMACrossover is a strategy being optimized. Any complex strategy is a strategy being optimized. We are comparing oranges to oranges. We are not comparing indicators to strategies or other addon types.

We are trying to identify the issue. We know that its something in your logic if you can enter and exit without the behavior being reproduced so we know how to move forward. Either looking for a line of code, or reporting with a simple script a bug to our development.

At this point, I do not have any specific lines of code I can report.
How would you like to proceed.?

Instrument being null or not available is not a bug. We warn about this in the help guide.
"Warning: The properties in this class should NOT be accessed within the OnStateChange() method before the State has reached State.DataLoaded"
https://ninjatrader.com/support/help...instrument.htm

You cannot use any information from data series until State.DataLoaded. This has warnings in many places all over the help guide.

IsEnabled is not supported to be set from a NinjaScript.

IsInStrategyAnalyzer I am finding is working.