Propagating Property Values

[ETFVoyageur]

When I "OK" the Indicators dialog NinjaTrader does some indicator object creation and destruction. After it creates the new object, but before it calls Initialize() on the new object, NT copies some indicator property values. It looks like it is copying the ones that appear in the Indicators dialog, but not other properties. That means Indicator dialog properties persist across the behind-the-scenes hocus-pocus that should be invisible, but plain non-parameter properties are not persisted.

QUESTION: what has to happen so that a non-parameter property will also be persisted?

TRACE info to illustrate:

(#40, INITIALIZED)getting EnableUpperLine: value=True
(#40, INITIALIZED)    01/01/00 10:57:793  CurrentBar=-1    ^NDX (Weekly)  Entered VsaTiming.ToString(): isInitialized=True
(#40, INITIALIZED)    01/01/00 10:57:793  CurrentBar=-1    ^NDX (Weekly)  Leaving VsaTiming.ToString()

<Note: above here was when the Indicator dialog was up; below here after clicking "OK">

(#40, INITIALIZED)getting EnableUpperLine: value=True
(#40, INITIALIZED)getting EnableUpperLine: value=True

<Note: below here is constructing the new object>
(#44, constructing)Entered VsaTiming.VsaTiming()
(#44, constructing)Leaving VsaTiming.VsaTiming()

<Note: Below here is copying EnableUpperLine from the old to the new.>
<Note: Notice the copy overrides the default value (false), which is correct behavior>
(#40, INITIALIZED)getting EnableUpperLine: value=True
(#44, CONSTRUCTED)Setting EnableUpperLine: from=False to=True

<Note: now that parameter properties have been copied over Initialize() is called for the new object>
(#44, initializing)   20:11:16 11:16:008  CurrentBar=n/a  ^NDX (Weekly)  Entered VsaTiming.Initialize(): (VsaIndicator:Initialize)
(#44, initializing)   20:11:16 11:16:008  CurrentBar=n/a  ^NDX (Weekly)  Leaving VsaTiming.Initialize(): (VsaIndicator:Initialize)
(#44, initializing)   20:11:16 11:16:008  CurrentBar=n/a  ^NDX (Weekly)  Entered VsaTiming.Initialize()
            User colors: upper=Color [DodgerBlue]  center=Color [DarkGray]  lower=Color [DodgerBlue]  enableUpperLine=True

VsaTiming is the indicator
The #'s are object IDs I keep just to allow clarity in understanding cases like this.
#40 was the original indicator object.
#44 is the new object NT is making
"EnableUpperLine" is a parameter that *does* get propagated

Non-parameter properties do not participate. How can I set one up so that it will participate, but not show up in the Indicators dialog?

Thanks,
EV

[ETFVoyageur]

Note: I can get the desired property copied over, but at an unacceptable cost. If I comment out "[Browsable(false)]" it gets copied -- but it also shows up in the Indicator dialog.

So the question remains -- how do I get the copying behavior without having it appear in the Indicator dialog? Is there some other attribute I can set that will do that? The idea that the only things that get persisted are properties the user is allowed to see is unacceptable.

The fundamental question is how to persist state when NT does its behind-the-scenes shenanigans. Any ideas?

[Philosophical note: NT has no business doing non-transparent things behind the scenes. I have complained about this for years, and wasted a lot of time trying to deal with it. I sure hope that NT8 no longer does such things.]

[NinjaTrader_PatrickH]

Hello ETFVoyageur,

Thank you for your note.

The Initialize method is called when the parameters window (Indicator window) is open, this calls any user defined parameters. As you are setting a property to not be a user defined property it would not go through the same process.

What is the behavior you expect your parameter to go through if it is not being changed in the UI?

[ETFVoyageur]

Patrick,

I'm not following you. There are two kinds of properties -- ones that show up in the Indicators dialog and ones that do not. The former are persisted when NinjaTrader does its unfortunate non-transparent meddling. The latter are not persisted, resulting in said meddling NOT being transparent.

The first kind preserves internal state across NinjaTrader meddling -- they are effectively transparent. What I want is some way to persist additional state without requiring it to show up in the Indicators dialog. How can I do that?

The base problem is architectural -- I should be able to write an indicator class and not have to worry about NinjaTrader pulling the rug out from under me, such as re-initializing my non-user-visible properties on the fly. Since that's never going to get fixed in NT7 (I hope the problem has gone away in NT8) I am looking for how best to live with it. How can I keep track of internal state without NinjaTrader gratuitously re-initializing it out from under me?

Thanks,
EV

[ETFVoyageur]

Patrick,

I apologize if my post sounded harsh. I'm not worrying about getting NT7 fixed -- just how best to live with it and pray that NT8 is better designed. The fundamental issue is that a programmer has a right to assume that a class will remain intact for the duration. Specifically, if a variable or a property gets set, then that item will remain at the value that was set for as long as the indicator is on the chart. I would expect this to be true for the class behind an indicator on a chart -- but, unfortunately, that does not hold so I need to learn how best to live with reality.

NinjaTrader 7 violates this big time. It creates new instances as it pleases, it has some bizarre execution paths for some of those instances it creates (if you do not believe me, just do some thorough tracing using something like I used to produce the code segment in my first message), and it freely replaces the class behind an indicator on the chart (for instance this happens every time you click OK on the Indicators dialog). That replacement can pull the rug out from under you -- it destroys all information other than that visible in the Indicators dialog. Scarcely transparent.

I can live with that, but I really need to know how to persist a value across NinjaTrader's machinations (without needing to make the value visible in the Indicators dialog). Note that NinjaTrader 7's behavior is a problem for me because I am trying to persist some internal state for the duration of the indicator on the chart. That's not a real common need, and the behavior I am concerned with is not a problem unless you need persistent internal state.

I believe I know the party line on what gets called when. I have read the documentation and it is a mixed bag. There is a lot of very helpful information there, but it is scarcely comprehensive. I also understand about "supported" -- that's why I am asking you how to persist my values. If you want an example of something not documented, consider ModifyProperties(). That is called frequently, and it is called between Initialize() and OnStartUp(). It is a crucial class for certain things.

[koganam]

Patrick,

I apologize if my post sounded harsh. I'm not worrying about getting NT7 fixed -- just how best to live with it and pray that NT8 is better designed. The fundamental issue is that a programmer has a right to assume that a class will remain intact for the duration. Specifically, if a variable or a property gets set, then that item will remain at the value that was set for as long as the indicator is on the chart. I would expect this to be true for the class behind an indicator on a chart -- but, unfortunately, that does not hold so I need to learn how best to live with reality.

NinjaTrader 7 violates this big time. It creates new instances as it pleases, it has some bizarre execution paths for some of those instances it creates (if you do not believe me, just do some thorough tracing using something like I used to produce the code segment in my first message), and it freely replaces the class behind an indicator on the chart (for instance this happens every time you click OK on the Indicators dialog). That replacement can pull the rug out from under you -- it destroys all information other than that visible in the Indicators dialog. Scarcely transparent.

I can live with that, but I really need to know how to persist a value across NinjaTrader's machinations (without needing to make the value visible in the Indicators dialog). Note that NinjaTrader 7's behavior is a problem for me because I am trying to persist some internal state for the duration of the indicator on the chart. That's not a real common need, and the behavior I am concerned with is not a problem unless you need persistent internal state.

I believe I know the party line on what gets called when. I have read the documentation and it is a mixed bag. There is a lot of very helpful information there, but it is scarcely comprehensive. I also understand about "supported" -- that's why I am asking you how to persist my values. If you want an example of something not documented, consider ModifyProperties(). That is called frequently, and it is called between Initialize() and OnStartUp(). It is a crucial class for certain things.

I recall that we went through a similar question a few years back, and eventually came up with code to manipulate the Browsable attribute of a property by using code. When I hide a property in this manner, any changes that I make on the indicator on a chart get persisted: even to an F5 press.

If I recollect, you posted your code, which was pretty much identical to mine, but I never posted my code.

[ETFVoyageur]

Thanks, koganam,

You are close. I thought about it overnight after my last post and realized I already have dynamic properties and should be able to do a modified version of that. That worked out. It turns out that the answer is:

• Set the [Browsable(true)] attribute for the property you want to persist-- just like a normal user parameter property that will show up in the Indicators dialog. Just set it, no manipulating it. You do not need any of the other attributes normally set for a user parameter property.
• Implement the ICustomTypeDescriptor interface as we have discussed before
• in ModifyProperties() always delete the persistent property from the collection. That's the only difference from a dynamic property. Dynamic properties are conditionally removed, while persistent properties are unconditionally removed (so they will not be visible in the dialog).

That's all there is to it. It has taken me a little time to get back because I took the time to print trace information and understand about what is really going on with object creation and destruction -- unlike the mental model of a single indicator object, by the time you bring up the dialog and OK it you have gone through several instances, which is why the ability to have persistent data is important. I highly recommend that exercise -- I now understand things that used to be mysterious, and some things I used to think were NinjaTrader bugs now make sense.

I don't suppose that this method is "supported", but it is simple to do and the best way I know to get the job done.

--EV

[ETFVoyageur]

FWIW: I just came on another case where I need a persistent property.

In my indicator, ToString() returns a value that includes a property (the index in use, such as NASDAQ or S&P500). The property in question needs to be set from GetProperties(). The problem is that the first time ToString() is called is before the first call to GetProperties().

The actual sequence of NinjaTrader calls is:

• Initialize()
• NinjaTrader restores persistent properties
• ToString()
• GetProperties()

Since the new object is intended to be a logical copy of the old one, the property value should be unchanged from the old instance -- a perfect use for a persistent property. Fortunately persistent properties have their value restored in time.

Notes to NinjaTrader support folks:

1. Given NinjaTrader's design that frequently replaces an indicator object instance, persistent properties can be essential. I now know how they work, but it would be helpful to others if NinjaTrader documentation explained about them.
2. If what I am doing is not supported, then it should be unless you can explain a reasonable alternative supported way to handle persistent properties.
3. Please ensure that NinjaTrader 8 supports persistent properties. My way is a bit of a kludge -- it would be better to just support something like a [Persistent] property attribute.
4. I consider NinjaTrader calling ToString() so soon to be a design bug. Please ensure that NinjaTrader 8 does not make its first call to ToString() until at least after the first call to GetProperties(). In NinjaTrader 7 terms the first call to ToString() should be the last call before calling OnStartup().
5. Note that it is not reasonable to assert that the initialization should be done in Initialize() -- the code would be a duplicate (because it would still have to be in GetProperties()) and no one wants duplicate code to maintain.

[NinjaTrader_Matthew]

NinjaTrader 8 will support overriding the Clone() method, where you can change the behavior of persisting property info/browseable attributes or not for your custom NinjaScript object. While the behavior of properties persistence has not changed, your concerns are valid, and have been acknowledged by our development team as NT8 during its development. Rather than changing base behavior, the idea to support overriding this method was decided to be a supported concept to provide flexibility for the more advanced developers who are impacted by our base behavior. As of this post, there is not any documentation for overriding Clone() in NinjaTrader 8 beta, but we will be expanding in this area in the near future.

[ETFVoyageur]

Matthew,

Thanks for the reply. I look forward to reading the Clone() documentation when it becomes available. Is there some way I can get notified when it is available for me to read?

=====

As noted in this thread, I currently set properties I want to persist as [Browsable(true)], implement the ICustomTypeDescriptor interface, and use GetProperties() to delete the persistent property from the collection.

Can you say anything about how using Clone() will compare to what I am currently doing?

Thanks,
EV

[ETFVoyageur]

OK -- just in case anyone else is interested, I now have a solution I am happy with. Persistent properties work fine for me, and adding a new one is now no harder than adding any other property. Here's what I have done:

• Define a new attribute [PersistentData] that takes an optional boolean, defaulting to true. (I thought it might be nice to be able to explicitly set a non-persistent property's persistence false, just as a matter of readability)
• Implement the ICustomTypeDescriptor interface
• In ModifyProperties(), which I call from GetProperties(), add a loop that looks at all properties with [PersistentData] and removes from the collection any property whose PersistentData attribute is true (so it will not display in the Indicators dialog)

Adding a new persistent property is now very straightforward. All I have to do is to declare the property as usual, with these attributes:

• [Browsable(true)] -- so NinjaTrader will persist the value
• [PersistentData(true)] -- so ModifyProperties() will prevent displaying it in the dialog

I would like to suggest that NinjaTrader 8:

• Define the PersistentData attribute
• When cloning, copy any property with PersistentData true (in addition to what it already does -- copying any property with Browsable true)
• Note: Persistent properties would no longer need to set [Browsable(true)], so no one would have to do anything to prevent them showing up in the dialog

That would make a developer's life very easy -- no need to override Clone() -- just provide the one attribute for any property that needs to be persistent. Also good readable code -- the property itself says whether it is persistent, so there is no need to go look at some Clone() method to find out.

This seems to me to be a very clean solution. Any comments from anyone? Matthew?

[NinjaTrader_Matthew]

Thanks for your suggestion, however we really do not see this being a use case that impacts a large portion of our user base. Exposing and supporting the virtual Clone() provides middleground and flexibility you're looking for and allows you to implement your own attributes/methods to satisfy the behavior you seek.

I've added your comments and suggestion to SFT-481 and will allow users to vote on this feature which will ultimately determine if this feature will be added.

[ETFVoyageur]

Matthew,

Thanks. I do agree that while the need is very real it is also not all that common.

One saving grace for my suggestion is that it should take very little time to implement -- a few hours at most for a competent developer. It does provide a very clean simple way to make an indicator that needs this -- I do not see how it could be any cleaner and simpler.

On the other hand, it is a narrowly targeted solution, not a more general capability. Since Clone() will be needed either way I'll be most interested to see just what is involved in overriding Clone(). I hope we can see that documentation pretty soon.

BTW: Will the method I am using continue to work in NinjaTrader 8?

[ETFVoyageur]

Matthew,

Perhaps a comment is in order about why I prefer my solution for persistent properties.

From a programming point of view it is best when things are clean, clear, simple, and elegant. Doing it right is important -- both aesthetically and practically -- usually code that is easier to read, has fewer bugs, requires less maintenance, etc. There are often other kludges that could work, but that does not make them good programming style.

What could be a cleaner simpler way to indicate a property is persistent than to just adorn it with a [PersistentData] attribute? I have nothing against a Clone() method, and I presume it is needed in a number of scenarios or else you would not be planning on supporting it. It does, however, fail this test.

As to who should support this attribute -- NinjaTrader should, because it is generally useful. It is not specific to my indicators or to any other specific indicators. Attributes that support specific indicators are the responsibility of the indicator developer. Attributes that make sense for the general system should be supported by NinjaTrader.

As to my assertion that the development effort is minimal:

1) Creating the property itself should be a few minutes work for one of your developers.

2) You already filter for [Browsable(true)] in the code where you persist property values. All you need to do is add one more case -- that [PersistentData(true)] also persists the property value.

3) I cannot imagine that adding this to the product adds much of any testing burden.

That's enough -- I'll shut up before becoming any more obnoxious. I wrote this because it occurred to me that I have never explained why I view this as worth doing. It's because it is the Right Thing. For me, personally, if NinjaTrader will not support this I can just continue to do as I am doing now.

--EV