NT causing an indicator problem

[ETFVoyageur]

The fundamental bug is that NT is playing fast and loose with indicator objects, and the results do not always match a reasonable model for the program. What I mean by a reasonable model is that indicator code needs to be able to act as if there is one ongoing object. If anyone is shifting it to a new instance, that must be done such that it is transparent to the indicator.

Unfortunately, NT does shift indicators to new instances, and it fails to do so transparently. As far as I can tell, what is happening is that NT is making new instances by creating new objects and assigning the values it thinks it should to properties. THAT IS TOTALLY BOGUS, because there is no way it can be transparent. One of the problems is that "set" methods can, and do, have side effects.

NT absolutely must make a new object as a complete copy of the existing one! That is the only way to be transparent. Currently I am seeing non-transparency problems such as:

• Property method call patterns are affected
• Non-property storage is not preserved

I have been bitten by both of those issues in the last few days. The symptom I just had was that I set a value in the indicators dialog and OK'd it. When I reopened the dialog the value had been reset. I traced the problem to non-transparency.

My assumption is that there is no way to get this kind of bug fixed in Version 7. What I do expect is for the folks at NT to understand the problem and ensure that Version 8 does not have the same problem. It really is a serious problem, at least for me.

--EV

[sledge]

Can you post some sample code that displays the issue So we can look at it?

[NinjaTrader_JC]

Hello EV,

I would like to send you suggestion to our Development team for consideration on the next major release of NinjaTrader but I do not follow exactly what you are trying to say.

It may be easier to have code example showing the behavior you see and tell me what the behavior that you are expecting.

Happy to be of further assistance.

[ETFVoyageur]

I appreciate the requests for code, but that is not practical. I could generate some traces, but I have posted such already. Have you looked at those? The best way might be for someone from support to talk to me on the phone -- I am willing to do that. (I'd need to talk to someone who understands how NT handles this, in case I have any misconceptions.)

The basic problem, as I see it, is that: my indicator must be able to run as if all that matters is its own code, what the user does with the configurable properties, and any documented system behavior.

NT breaks that. A good example is when accepting a configuration dialog. As my posted traces show, NT ends up constructing new instances of my indicator object and transferring control to them. That is a huge problem unless it is done transparently (and it is not).

Setting properties as a part of initializing the new objects is inherently non-transparent, because it causes calls to the property set methods that would not otherwise be made. That is big trouble for any set methods that have side effects. (a) it applies the side effect more times than expected, (b) it applies the side effect at unexpected times, and (c) it gives an unpredictable result, because just what happens depends on whether or not NT has already initialized any other properties the side effect depends on.

As an example, one of the the NT support guys suggested (in an earlier thread) that I implement line hiding by setting the line transparent, and then bring the line back by restoring the color. That cannot work, because of this problem. In certain use cases where the user has a line with a non-default color, restoring the hidden line sets it back to the default color.

I also discovered in the line color case that the location where I was saving the color to restore was getting reset out from underneath me because the new objects are not copied. Setting all of the properties is sufficient initially, but only a copy constructor is acceptable for an ongoing indicator.

One obvious question is why the property set methods have side effects to begin with. The answer is that the NT architecture provides no other place to do those actions. Besides, it ought to be OK to do that.

--EV

[sledge]

You have 579 posts.. where do I begin to see your 'traces' for this unknown code that you are thinking?

Are you trying to only use 1 instance of an indicator that was created manually on multiple charts?

Is this a serialization issue?

Are you using static variables?

I don't know. I don't see any sample code that shows this issue or steps involved on this code.

[ETFVoyageur]

You have 579 posts.. where do I begin to see your 'traces' for this unknown code that you are thinking?

Are you trying to only use 1 instance of an indicator that was created manually on multiple charts?

Is this a serialization issue?

Are you using static variables?

I don't know. I don't see any sample code that shows this issue or steps involved on this code.

Here is the tracing you requested. It is for one indicator on one chart.

I thank you for your somewhat sarcastic post. It did motivate me to add enabling / disabling printing trace information for property methods to my debug gui that I mentioned earlier. The tracing was always there, but now I can turn it on and off with a mouse click or two, without needing to recompile. Much more convenient.

I do not understand what the number of my posts has to do with anything. I do have a few previous posts on this topic, but nowhere near 579 of them.

You are not going to see the code, because that is not practical in this case. Furthermore, it is not germane -- all that matters is the creation of new objects and the calling of properties when doing so. Any such calling is inherently non-transparent and a bad idea -- that is NOT the correct way to initialize an object.

My indicators generate unique instance IDs -- that is what the first trace column is showing. This trace shows 5 instances of the indicator RwbMomentum (#20, #29, #60, #69, and #78.

Here is what I did that produced this trace:

First, I hit F5, refreshing the chart. That caused #20 and #29.

I do not know why #20, but note that it did enter one set method while the object was constructed, but Initialize() had not yet been called. That seems like a bug already -- calling methods before the object has had Initialize() called.

#29 was called after Initialize() has completed, but before OnStartUp() has been called. (That's why the state is only "INITIALIZED".) I'm not very happy about that timing either, since some object initialization cannot be done until OnStartUp(), and other initialization should not be done until then. Nevertheless, the important point is that many property set methods got called. That is a problem, because calling property set methods is inherently not transparent, and is not the right way to initialize an object.

Next, I brought up the configuration dialog, and clicked "OK" on it. That produced the remaining tracing -- creating and calling a bunch of set methods on instances #60, #69, and #78. That is bad.

By the way: I'd like to know who is printing the line "Initialize() called". AFAIK it is not any of my code.

Trace will be posted in the next message -- it makes this one too long, exceeding system limits.

--EV

[ETFVoyageur]

Here is the trace for the previous message. I set it as "CODE", rather than "QUOTE" to disable folding the lines -- doing that makes it unreadable.

(#20) CONSTRUCTED   21:17:05 17:05:353  CurrentBar=n/a  UNKNOWN  INSTRUMENT  RwbMomentum  Entered DebugLinReg.set()  old value=False  new  value=False  
(#29) CONSTRUCTED   21:17:05 17:05:442  CurrentBar=n/a  UNKNOWN  INSTRUMENT  RwbMomentum  Entered DebugLinReg.set()  old value=False  new  value=False  
(#29) INITIALIZED   01/01/00 17:05:446  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered BearColor.set()  old value=Color [Red]  new  value=Color [Red]  
(#29) INITIALIZED   01/01/00 17:05:446  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered BandColor.set()  old value=Color [DarkGray]  new  value=Color [DarkGray]  
(#29) INITIALIZED   01/01/00 17:05:446  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered BullColor.set()  old value=Color [Lime]  new  value=Color [Lime]  
(#29) INITIALIZED   01/01/00 17:05:446  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered DebugLinReg.set()  old value=False  new value=False   
(#29) INITIALIZED   01/01/00 17:05:446  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered WhichTrendType.set()  old value=SignalLine  new  value=SignalLine  
(#29) INITIALIZED   01/01/00 17:05:446  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered DebugLevel.set()  old value=None  new value=None  
(#29) INITIALIZED   01/01/00 17:05:446  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered DebugProperty.set()  old value=Set  new value=Set  
(#29) INITIALIZED   01/01/00 17:05:446  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered EnableDebug.set()  old value=True  new value=True  
(#29) INITIALIZED   01/01/00 17:05:447  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered BandTransparency.set()  old value=3  new value=3  
(#29) INITIALIZED   01/01/00 17:05:447  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered SmoothLength.set()  old value=1  new value=1  
(#29) INITIALIZED   01/01/00 17:05:447  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered DebugSymbol.set()  old value=  new value=  
(#29) INITIALIZED   01/01/00 17:05:447  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered SignalLength.set()  old value=10  new value=10  
(#29) INITIALIZED   01/01/00 17:05:447  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered EnableCenterBand.set()  old value=True  new  value=True  


Initialize() called
(#60) CONSTRUCTED   21:17:22 17:22:732  CurrentBar=n/a  UNKNOWN  INSTRUMENT  RwbMomentum  Entered DebugLinReg.set()  old value=False  new  value=False  
(#60) INITIALIZED   01/01/00 17:22:733  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered BearColor.set()  old value=Color [Red]  new  value=Color [Red]  
(#60) INITIALIZED   01/01/00 17:22:733  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered BandColor.set()  old value=Color [DarkGray]  new  value=Color [DarkGray]  
(#60) INITIALIZED   01/01/00 17:22:733  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered BullColor.set()  old value=Color [Lime]  new  value=Color [Lime]  
(#60) INITIALIZED   01/01/00 17:22:733  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered DebugLinReg.set()  old value=False  new value=False   
(#60) INITIALIZED   01/01/00 17:22:733  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered WhichTrendType.set()  old value=SignalLine  new  value=SignalLine  
(#60) INITIALIZED   01/01/00 17:22:733  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered DebugLevel.set()  old value=None  new value=None  
(#60) INITIALIZED   01/01/00 17:22:733  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered DebugProperty.set()  old value=Set  new value=Set  
(#60) INITIALIZED   01/01/00 17:22:733  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered EnableDebug.set()  old value=True  new value=True  
(#60) INITIALIZED   01/01/00 17:22:733  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered BandTransparency.set()  old value=3  new value=3  
(#60) INITIALIZED   01/01/00 17:22:733  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered SmoothLength.set()  old value=1  new value=1  
(#60) INITIALIZED   01/01/00 17:22:733  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered DebugSymbol.set()  old value=  new value=  
(#60) INITIALIZED   01/01/00 17:22:733  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered SignalLength.set()  old value=10  new value=10  
(#60) INITIALIZED   01/01/00 17:22:733  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered EnableCenterBand.set()  old value=True  new  value=True  
(#69) CONSTRUCTED   21:17:59 17:59:027  CurrentBar=n/a  UNKNOWN  INSTRUMENT  RwbMomentum  Entered DebugLinReg.set()  old value=False  new  value=False  
(#69) INITIALIZED   01/01/00 17:59:028  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered BearColor.set()  old value=Color [Red]  new  value=Color [Red]  
(#69) INITIALIZED   01/01/00 17:59:028  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered BandColor.set()  old value=Color [DarkGray]  new  value=Color [DarkGray]  
(#69) INITIALIZED   01/01/00 17:59:028  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered BullColor.set()  old value=Color [Lime]  new  value=Color [Lime]  
(#69) INITIALIZED   01/01/00 17:59:028  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered DebugLinReg.set()  old value=False  new value=False   
(#69) INITIALIZED   01/01/00 17:59:028  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered WhichTrendType.set()  old value=SignalLine  new  value=SignalLine  
(#69) INITIALIZED   01/01/00 17:59:028  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered DebugLevel.set()  old value=None  new value=None  
(#69) INITIALIZED   01/01/00 17:59:028  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered DebugProperty.set()  old value=Set  new value=Set  
(#69) INITIALIZED   01/01/00 17:59:028  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered EnableDebug.set()  old value=True  new value=True  
(#69) INITIALIZED   01/01/00 17:59:028  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered BandTransparency.set()  old value=3  new value=3  
(#69) INITIALIZED   01/01/00 17:59:028  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered SmoothLength.set()  old value=1  new value=1  
(#69) INITIALIZED   01/01/00 17:59:028  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered DebugSymbol.set()  old value=  new value=  
(#69) INITIALIZED   01/01/00 17:59:028  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered SignalLength.set()  old value=10  new value=10  
(#69) INITIALIZED   01/01/00 17:59:028  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered EnableCenterBand.set()  old value=True  new  value=True  
(#78) CONSTRUCTED   21:17:59 17:59:077  CurrentBar=n/a  UNKNOWN  INSTRUMENT  RwbMomentum  Entered DebugLinReg.set()  old value=False  new  value=False  
(#78) INITIALIZED   01/01/00 17:59:078  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered BearColor.set()  old value=Color [Red]  new  value=Color [Red]  
(#78) INITIALIZED   01/01/00 17:59:078  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered BandColor.set()  old value=Color [DarkGray]  new  value=Color [DarkGray]  
(#78) INITIALIZED   01/01/00 17:59:078  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered BullColor.set()  old value=Color [Lime]  new  value=Color [Lime]  
(#78) INITIALIZED   01/01/00 17:59:078  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered DebugLinReg.set()  old value=False  new value=False   
(#78) INITIALIZED   01/01/00 17:59:078  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered WhichTrendType.set()  old value=SignalLine  new  value=SignalLine  
(#78) INITIALIZED   01/01/00 17:59:078  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered DebugLevel.set()  old value=None  new value=None  
(#78) INITIALIZED   01/01/00 17:59:078  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered DebugProperty.set()  old value=Set  new value=Set  
(#78) INITIALIZED   01/01/00 17:59:078  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered EnableDebug.set()  old value=True  new value=True  
(#78) INITIALIZED   01/01/00 17:59:078  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered BandTransparency.set()  old value=3  new value=3  
(#78) INITIALIZED   01/01/00 17:59:078  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered SmoothLength.set()  old value=1  new value=1  
(#78) INITIALIZED   01/01/00 17:59:078  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered DebugSymbol.set()  old value=  new value=  
(#78) INITIALIZED   01/01/00 17:59:078  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered SignalLength.set()  old value=10  new value=10  
(#78) INITIALIZED   01/01/00 17:59:078  CurrentBar=-1    TSLA (Daily)   RwbMomentum  Entered EnableCenterBand.set()  old value=True  new  value=True

[sledge]

Good luck Bro

[ETFVoyageur]

Good luck Bro

Thanks. I hope everyone understands

• Why I am insisting that transparency is important
• That V7 does not provide that transparency
• The problem is not only calling property methods -- it is also preserving the value of member variables that are not public properties. Calling a constructor (other than a copy constructor), followed by setting properties does not preserve member variables. Unfortunately, that is just what my trace shows happening.

V7 is, and will be, as it is. I do not expect any change in V7 in this regard -- that would be a huge and risky thing to do. I would not do it if I were running NT software development.

OTOH, I do think it is reasonable to point out the problem and expect that V8 will not have the same problem.

Bottom line is that some things just will not work in V7. One example is the one I gave. I do not believe it is possible to implement a robust hiding and restoring of lines in response to property changes, in V7.

--EV

[sledge]

Have you watched the NT 8 videos on YouTube from big mikes?
The 12/2013 was most telling.

Lots of code to rewrite for paradigm shift

[ETFVoyageur]

Have you watched the NT 8 videos on YouTube from big mikes?
The 12/2013 was most telling.

Lots of code to rewrite for paradigm shift

I have watched one fairly long one by Ray explaining V8. I did not explore for others. Are there others I ought to go look at?

----- -----

Thanks for getting me thinking about a solution to the problem.

As to it being a lot of work -- maybe ... depends on the design. While I would like the transparency I ask for, that may be a lot of work and there may be alternatives that would work. I can live with a non-transparent environment as long as there is supported behavior and there are conventions that allow getting the practical problems solved.

I have not thought about the best solution, but your remark prompted me to think a little. It seems to me that a practical solution:

• Needs to be an acceptable amount of work for NT to implement
• Needs to cause minimal ripples for indicator developers
• Can be ignored by indicators for which it is not important (most indicators)
• Is easy to use for indicators that need it

The thing that is causing all of the problems is:

• Set methods that have side effects -- they either depend on the value of other properties, or they set the value of other properties
• The inability of set methods to know why they are being invoked -- whether that is due to normal program invocation, or whether NT is just making a copy for some reason, and setting properties as part of that effort
• The "copied" objects are not fully copied -- member variables other than public properties are not preserved

You cannot just outlaw such side effects, because they are the only way to do some things. What you can do is allow the set methods to know why they are getting called. All that would really require would be a new base class property "InitializeInProgress". The rule would be that set methods should avoid any side effects if called with (InitializeInProgress==true)

You could address the need to preserve member variables (such as the cached color that the hidden user line needs to be restored to) by making the rule be that all such values need to be kept in public serializable non-browsable properties.

It should be obvious that I have not spent time thinking about the technical solution, but shooting from the hip I think the above suggestions might do it. If so, there is one very compelling point to this solution -- implementation would be trivial for NT -- all they would need to do would be to add that one property. The rest is a couple of easy actions for the affected indicator developers (i.e.very few of them). The solution would be easy enough to even implement in V7

I still need to think about how this preserves settings the user has changed, but that seems to work today. If it is a problem, then part of making the new object could be to get values from the old object and set them in the new one.

--EV

[koganam]

Thanks. I hope everyone understands

• Why I am insisting that transparency is important
• That V7 does not provide that transparency
• The problem is not only calling property methods -- it is also preserving the value of member variables that are not public properties. Calling a constructor (other than a copy constructor), followed by setting properties does not preserve member variables. Unfortunately, that is just what my trace shows happening.

V7 is, and will be, as it is. I do not expect any change in V7 in this regard -- that would be a huge and risky thing to do. I would not do it if I were running NT software development.

OTOH, I do think it is reasonable to point out the problem and expect that V8 will not have the same problem.

Bottom line is that some things just will not work in V7. One example is the one I gave. I do not believe it is possible to implement a robust hiding and restoring of lines in response to property changes, in V7.

--EV

It sounds to me like your code is changing either a Property or its corresponding backing store somewhere. I follow what you are describing, but unfortunately, these really are your conclusions of what is happening, rather than issues with how NT may actually be handling the code, in this particular instance.

The code is doing it, of course: I mean that the results are not matching your expectations, which may not really be because of how the code is handling what you wrote, so much as what you wrote is what the code is doing.

I can understand why you may not want to disclose your code, as it may have some proprietary coding techniques that you do not want to give away. As such, I would suggest that you send in some code that demonstrates the problem to NT. They are unlikely to give you direct access to anyone on the development team, for various legal reasons that I can think of, but I am sure that they will forward any code that you send in, and that way, your code is exposed to only a minor set of folks who are obliged to keep it undisclosed the the general public.

As a matter of fact, I once wasted their time by sending in some code that had me tearing out my hair, simply because I was making a check for a value different from what I had set. I was so embarrassed that I not yet bothered them ever since, but it does demonstrate that they are willing to engage the development team when necessary.

[ETFVoyageur]

I'll get back to that matter. I've done all I can for now. It is clear to me from my traces, including the one I recently posted for Sledge, that multiple objects are getting created. For each one the default constructor is getting called, and properties are getting set. I'll get around to a small demo program at some point, but what I just described is already a problem.

Do you disagree with what I allege is happening -- the multiple objects and how they are getting initialized?

BTW: I trust you noted my later message than the one you are quoting, in which I said that I think (not positive -- have not thought it through thoroughly) that all that is really required from NT is a property that tells when it is setting properties as part of its initializing a new object. The property in the indicator really should be able to tell calls to set that happen as part of normal operation from those that happen as part on NT's behind-the-scenes initializing new instances.

--EV

[koganam]

I'll get back to that matter. I've done all I can for now. It is clear to me from my traces, including the one I recently posted for Sledge, that multiple objects are getting created. For each one the default constructor is getting called, and properties are getting set. I'll get around to a small demo program at some point, but what I just described is already a problem.

Do you disagree with what I allege is happening -- the multiple objects and how they are getting initialized?

BTW: I trust you noted my later message than the one you are quoting, in which I said that I think (not positive -- have not thought it through thoroughly) that all that is really required from NT is a property that tells when it is setting properties as part of its initializing a new object. The property in the indicator really should be able to tell calls to set that happen as part of normal operation from those that happen as part on NT's behind-the-scenes initializing new instances.

--EV

I follow your arguments. The issue is in determining what is really happening. Yes, NT runs the Initialize code at multiple different times. However, I have never seen spurious changes to my private variables, and I surmise that that is because, NT deliberately does not provide default constructors for the user-created (or shipping) NT Indicator or Strategy class objects, thus forcing all calls to the object to specify all public properties, after which it touches nothing else. Yes, I do have setters that set the value of other parameters, (which technically are side-effects), but even those behave as I expect, only changing other parameters when I use the PropertyGrid to change the initiating variable.

Now, if as some programmers sometimes do, and provide a default (i.e., parameterless) constructor, then I would not be surprised if in copying or initializing said indicator, NT might call the default constructor, in which case it will use default values for any private variables, especially if they were not initialized when declared. It would then seem that values have been changed or defaulted, whereas, in fact, the code is doing what c# code does when an object is copied or initialized with a parameterless constructor.

[NinjaTrader_Brett]

ETFVoyageur,

Thanks for the post detailing the issue your running into. Unfortunately I'm not understanding your core problem scenario which is where I needed to start else we get lost and wont get down to the core issue.

Is this problem reproducible on your side with a single indicator? Basically: What is scope we talking here: issue with single indicator usage and reusing an indicator or issue with multiple indicators on a chart?

Furthermore can you summarize a simple scenario for me of what you get vs what you expect: for example:

1) I start indicatorA that uses indicator B and get value 100 on bar 1.
2) I start indicatorB with same settings and get value 50 on bar 1. -> Why do I get 50?