Strategy analyzer inconsistent: summary v trades data

[roman_ch]

Dear NT team,

Please could you look into a likely bug with NT8 RC2 strategy analyzer. As per attached screenshots, data for total no.of trades and cum P&L in the analyzer columns does not match the actual trades data from backtest.

It appears that the trades' data is correct: on a chart, the strategy's historic performance matches analyzer's trades data, but not analyzer's aggregate columns.

Please could you confirm this is a bug or let me know how to resolve the issue. Thank you.

Regards,
Roman

[NinjaTrader_Jesse]

Hello,

Thank you for the post.

I am not seeing this occur in my specific test, I wanted to check, are you able to see this using the SampleMACrossOver strategy? If so could you provide a screenshot of the settings used for testing?

If this only happens with a custom script, I would like to see the syntax being used in the case that may be the cause. Could you please test the Sample strategy and let me know if this occurs still?

I look forward to being of further assistance.

[roman_ch]

Hi Jesse,

Thanks. This problem does not occur with a "factory" strategy.

You probably appreciate that this behaviour likely is caused by problems with IEnumerator implementations and/or memory management.

I have done some limited re-testing and I believe that this is related to the garbage collector working with encapsulated references to, say, indicators (either factory NT indicators or custom-coded).

Ordinarily, objects with such "inside" references should be GC-collected some time after the "outside" object itself becomes orphaned. I cannot confirm this with certainty, but from memory stats it looks like in optimisations with many iterations at some stage GC stops processing inside objects and a memory leak occurs. If inside references are nulled out by user-end code as if in a destructor, the problem goes away.

There is a related thread re NT8 memory leaks, I am not sure if you would like to cross share this with the development team.

Regards,
Roman

[NinjaTrader_Jesse]

Hello,

Thank you for the reply.

Would you be able to provide a test that shows your findings?

If you can confirm this does not happen with the standard items, but is reproduced using the syntax you have created, I would like to see that specific syntax.

Unfortunately without being able to see the same result on my end and not knowing what syntax was used specifically, I would not really have enough information to make a report that something may be not working correctly.

I look forwards to being of further assistance.

[roman_ch]

Hello Jesse,

Thanks. The type of code I am using is along the lines of:

public class UserTests { public EMA MyEMA; ... }

public class MyStrategy : Strategy
{
...
UserTests MyUserTests; ...

protected override void OnStateChange() { ...
else if (State == State.Configure)
{
...

MyUserTests.MyEMA = EMA(iLength);

...}

}
In theory upon a new optimization iteration, the "old" EMA becomes orphaned and therefore available for GC collection. Could you please let me know if that is / is not the intended NT behaviour? My actual observation is that memory is not freed unless in State.OnTermination MyEMA = null; is included. (Perhaps, it has something to do with how internal indicator value caching works in NT.)

Referring back to my first post, without the =null line I saw a memory leak which I also might have caused IEnumerator problems (ie the screenshot where the trades collection has 60+ elements but somehow the total count is only 8).

Regards,
Roman

[NinjaTrader_Jesse]

Hello,

I wanted to check, what setting are you using for IsInstantiatedOnEachOptimizationIteration, http://ninjatrader.com/support/helpG...htsub=isinstan

Based on your newest description, it sounds like you are having un expected values and you are also using a class level variable which is the specific scenario in the help guide. Can you confirm if this is the case?

Please let me know if I may be of further assistance.

[roman_ch]

Hello Jesse,

Thanks. I am afraid I do not quite follow: you have correctly described the code I've provided but that does not explain the memory leak or the inconsistency in the collection. Seems that you were referring to the re-setting of the class variables, but as per my sample, the class variables are in fact re-set during State.Configure, so that should not cause any problems under the help guide you mentioned.

Could you please explain where do you see the problem in the sample I posted before?

Just in case and for the sake of completeness I would rather describe the issue as inconsistency in collection data rather than incorrect values. As far as I can see in managed code one cannot possibly have a result where a collection has 68 entries and Count of 8: something must be wrong here with the internal implementation of collections interfaces. But I am happy to hear otherwise.

Regards,
Roman

[NinjaTrader_Jesse]

Hello,

Unfortunately I cannot really answer these questions as this is not a complete sample we are referring to, what setting are you using for IsInstantiatedOnEachOptimizationIteration specifically?

I previously asked this because you had explained "My actual observation is that memory is not freed". This property would directly effect user variables and if they can or can not be reused between iterations. This would also depend on how the file is structured. If you are using false, then where you are resetting the variable in what you provided is not in the correct location and should instead be in DataLoaded. Unfortunately without a more specific example, this would be the observation I could make.

Additionally garbage collection is handled by the .net framework and not the platform in the case you are expecting items to be collected immediately. You may see items persist for awhile until garbage collection is run once again, it will only spend CPU to do that when it absolutely needs to.

At this point it would be best if you could export a working sample that describes your question clearly and we could look further into that specific syntax. Without seeing the settings you are using for the script, and logic being executed as well, I could not really confirm anything.



Please let me know if I may be of further assistance.

[roman_ch]

Hello Jesse,

Thank you.

Unfortunately I cannot really answer these questions as this is not a complete sample we are referring to

I fully agree we should deal with THE question I specifically asked on the sample code.

what setting are you using for IsInstantiatedOnEachOptimizationIteration specifically?

Sorry if it was not clear. False.

I previously asked this because you had explained "My actual observation is that memory is not freed". This property would directly effect user variables and if they can or can not be reused between iterations.

I do understand the difference between instantiating a class and re-setting its members through an event call. This must have been clear from my posts.

... If you are using false, then where you are resetting the variable in what you provided is not in the correct location and should instead be in DataLoaded.

I will test this, but in the meantime, I cannot see how this can make a difference here. I am confident there should be none. I assume that State.Configure must be triggered by the optimizer at the start of each iteration because strategy parameters are obviously changed for every new run and strategy must be re-configured. In fact, NT's "SampleMACrossOver" re-sets indicators in State.Configure rather than in State.DataLoaded.

(There are "species" of user variables which must be re-set in State.DataLoaded (those which need access to, say, TickSize, etc) but this is not the case in the example.)

At this point it would be best if you could export a working sample that describes your question clearly

No, it is not the best or even feasible approach. There are many pages of code and it would be impossible even for a committed and knowledgeable customer support to analyze it. Instead please let us deal the sample and the specific question that I have provided.

I will test when State.Configure is invoked. I will send a further post then.

Additionally garbage collection is handled by the .net framework and not the platform in the case you are expecting items to be collected immediately. You may see items persist for awhile until garbage collection is run once again, it will only spend CPU to do that when it absolutely needs to.

Of course. (As a diagression, as I do not think this is the point here, one could try to call GC.Collect().) Is there a way to get a meaningful memory dump during NT optimizations? This might actually be very helpful.

Thanks,
Roman

[roman_ch]

Hi,

Here is an update on my earlier post: I mentioned I will test how various OSC event are invoked. There is some long text below, but I must say here that I should be grateful for any comments on my post (would be great to have proper technical feedback).

I could not debug properly while optimizer is running. It always seems to run multi-threaded, irrespective of whether assembly compile was for debug. Using breakpoints to test was not a meaningful option.

I printed the number of hits for various OSC flags into the output window instead (ie, int iCountStateConfigureHits++, etc). As a reminder, the optimizer was ran w/out instantiating the strategy on each iteration. Output printouts are at the end of the post, but what they suggest is that the optimizer always splits its work between threads and creates several instances of the strategy. This would explain counts out of sequence in the prints for the same State event.

Part of the previous post related to State.Configure and when it was hit and whether it could be used to re-set user objects (it was suggested that it should not be used so). Prints suggest it is hit approx the same no. of times as DataLoaded. It seems it can be used to re-set user coded objects. NT 8 help guide also mentions that re-setting should be done State >= State.Configure.

State.Terminated, however, was only hit at the end of the whole optimisation (not iteration) so having there code along the lines of "myTestClass = null", as I did previously, is pretty much pointless. Even more obvious to me now is that it is pointless to null out references to factory NT indicators: I assume they must be actually created in the NT core through a class factory. In the context where indicator parameters do not change from iteration to iteration, re-setting them can only interfere with the factory cache and reduce performance.

I am not sure what actually happens when parameters are changed from iteration to iteration. Is it NT's core logic that, for example, if an indicator "creation" method was not called from SetDefaults or Configure, it is no longer to be cached? (since Terminated is not invoked until all optimizations are finished).

From the above, it seems there is almost no point to try to manage active references from the user code, so whatever memory leaks I have observed, there were not related.

On a side note, I have to say that it took awhile to carry out all the tests and perhaps things could be a bit more efficient on the technical side of support.

Output pints below:

SetDefaults was hit 0 times
Configure was hit 0 times
SetDefaults was hit 0 times
SetDefaults was hit 0 times
SetDefaults was hit 0 times
SetDefaults was hit 0 times
SetDefaults was hit 0 times
SetDefaults was hit 0 times
Configure was hit 0 times
Configure was hit 0 times
Configure was hit 0 times
Configure was hit 0 times
DataLoaded was hit 0 times
DataLoaded was hit 0 times
DataLoaded was hit 0 times
DataLoaded was hit 0 times
SetDefaults was hit 0 times
...
...
DataLoaded was hit 7 times
DataLoaded was hit 10 times
Configure was hit 6 times
Configure was hit 4 times
DataLoaded was hit 6 times
DataLoaded was hit 4 times
Configure was hit 8 times
Configure was hit 11 times
DataLoaded was hit 8 times
Configure was hit 7 times
Configure was hit 7 times
Configure was hit 5 times
...

Regards,
Roman

[roman_ch]

more follow up

Jesse, NT team,

I have spent some more time with the strategy analyzer over the few days. I can confirm my earlier observations: there must be something going on with the implementation of the optimizer.

Problems occur when testing CPU-heavier strategies with, say, > 500K iterations running over 4-5 hours. The same strategy can be optimized without issue for a smaller no. of properties (reducing iterations to, eg, 20K).

To recap, here are the inconsistencies:

• results from the optimizer do not match the results obtained outside of the optimizer (eg applying strategy to a chart);
• numbers reported by the optimizer in the summary tab vs other views are inconsistent (eg net profit from summary does not match the 3d graph plot, the no. of trades does not match no. of executions)

For illustration, I have attached two screenshots: one shows net profit graph reading around +250 max (I am not particularly concerned about the discrepancy in currencies here) but the summary reports max net profit around 1,500, another screenshot shows difference in results between optimizer and historic perfomance for the same strategy from a chart. I have previously also posted a screenshot showing difference in the no. of executions and trades as reported in the summary.

I believe another thread mentioned that the analyzer window becomes irresponsive after optimizations have finished. I have also observed this and it is often at this stage an unusually large commit charge can occur (I saw up to 20GB). Generally, memory consumption grows as the optimization progresses (from around 2GB in idle to 5GB on completion). Again, if there is a way to get a meaningful memory dump during optimizations, please let me know, that would be helpful.

Having in mind some of the previous questions, I will mention again that my thinking is that given the nature of c#, user code cannot cause, for example, the inconsistency in the representation of the results by the optimizer. To me, this looks like a internal optimizer collection overflowing and starting to overwrite itself rather than throw an exception.

Regards,
Roman