Serialization revisited; and is there an alternative?

[Hyper]

OK, I've just gotten "my sea legs" in complex NinjaTrader development
implemented as multi-class assembly DLL's.

Now, I fully realize that you simply can't help the "custom developers"
in every aspect of their work; as NinjaTrader is so flexilble and
extensible. That makes the support effort unbounded and so I
completely sympathize. However...

From what I see, the re-activation of an Indicator, Strategy, or similar
within the NinjaTrader 8 system depends *entirely* upon successful
serialization and deserialization. If this is unsuccessful, then manual
re-activation of the facility is required *every time* NinjaTrader is
restarted.

First of all, is this correct? I think it is... So I am proposing that there
be *an alternative* method for complex assembly DLL's which
*does not depend upon serialization*.

That an Indicator, Strategy, etc. be able to specify reactivation
through a mechanism which is not serialization/deserialization.

Is there already such a facility? and, if not, then I propose that
you consider providing one.

I'm not trying to be irritating here, since I'm so impressed by the
platform; but one of the biggest stumbling blocks is the seeming
inability to satisfy the re-activation requirements through
the serialization mechanisms; and especially on update of
an assembly DLL, even though the same parameter interface
may be maintained, the inability for NinjaTrader 8 to correctly
activate it; thus requiring a manual intervention for activation
each time the platform is restarted.

So I'm just respectfully requesting that some alternative
mechanism be provided so that assembly DLL's with
arbitrary internal complexity; be able to be modified
in implementation, but activated through a mechanism
which does not depend upon the current serialization
constraints

Even a workaround at this point would be helpful; for
example, maybe a "stub" high level NinjaScript generated
object (which would be reliably serialized/deserialized)
could then internally "dynamically load" the underlying DLL
assembly. Maybe that would bypass my difficulties, and
provide a path for DLL assembly activation that might
resolve the issues? At this point, any such workaround would
be welcome ! Any solution would be welcomed. Thanks in advance...

[edit] Like if the "serializable stub loader" would dynamically allocate the
DLL and use its entry points as a "delegate" implementation.
I have a feeling that could work somehow...

hyperscalper

[Hyper]

[EDIT] this post should be discounted and, instead, the next post
describes a fully supported method for developing dynamically
bound "plugin" application components.

What looks like an organized "interface based" approach to this
is described here:
https://stackoverflow.com/questions/...-method-in-dll

I guess I'll try to do something like this; but if there were a provided
facility for safe method delegation, then others could use the approach.

Maybe someone from the community has already done this ?

hyperscalper

[Hyper]

I took a look further into this. No doubt NinjaTrader platform engineers are
aware of this, but I'll post links anyway. The best way to understand this facility
is that it's a fully supported .NET 4.5 facility to create a "plugin architecture"
by which subsystems can be loaded, bound to, and unloaded cleanly.

An application loader, or "stub" could use this mechanism to dynamically load
and bind to a DLL assembly which provides some or all of the functionality.
In this case, the goal would be NinjaTrader Indicators, Strategies, etc. whose
implementation lies in the "plugin".

the NuGet package is available here: https://www.nuget.org/packages/microsoft.composition

https://docs.microsoft.com/en-us/dotnet/framework/mef/mef-for-net-for-windows-store-apps?redirectedfrom=MSDN

That's probably enough to get someone started with this; and it's clearly for Advanced requirements
only. I'll give it a shot when I get time; but it seems this could be a path toward better
integrating complex DLL Assemblies as "plugins" in a fully supported manner.

hyperscalper

[NinjaTrader_ChelseaB]

Hello hyperscalper,

Serialization is converting object values to strings so these can be saved in xml files such as workspaces and templates, and then de-serializing is reading from the xml file (workspace or template) and converting back to an object.

Inputs are basically all that would be serialized and saved, and only if they are types that are not easily converted to string such as numbers, bools, and enums.

Below is a link to the help guide on serializing brushes with an example.

https://ninjatrader.com/support/helpGuides/nt8/index.html?working_with_brushes.htm#Userdefinedbrushes

As far as referencing other classes, this would be outside of what is supported by NinjaTrader, but I have an example of shared static classes.

https://ninjatrader.com/support/forum/forum/historical-beta-archive/version-8-beta/83018-partial-class-does-not-work?p=712245#post712245

Below is a link to a video on exporting shared classes.

https://drive.google.com/file/d/1D9qMVu1Ahb58Z82IVT0R7caTu0icg9nC/view?usp=sharing

Dlls must be referenced for the code to usable.

https://ninjatrader.com/support/helpGuides/nt8/using_3rd_party_indicators.htm

[Hyper]

Thanks very much for the info, and for addressing the issue. Originally, I thought
in this way: "For a Strategy, the "root" or "top level" class would be the one
that would need serialization/deserialization, and all other referenced
objects would have their fields marked no serialize"

As I developed my Strategy, in another post, you'll recall we had all of those
"clone instances" to untangle; where only the Strategy whose state advanced
_beyond Configure_ was the instance that was going to survive as the
actual running Strategy subclass instance.

I'm just suggesting whether NinjaTrader might implement an alternative
approach which activates the "root object", e.g. the MyStrategy just once
and, of course, uses Reflection to determine any Properties.

Surely there are excellent reasons for doing it the way NinjaTrader 8
does it; but it seems to me right now, that an alternate method of
activation and querying the object could be provided, that would be
more suited to complex multi-class solutions packaged in assembly
DLL's, and make it easier for the developer.

Let me say that I just _love_ NinjaTrader 8 and, having overcome
much of the initial learning curve, it continues to impress me !

This is just a suggestion of something which would help those of
us who use the Visual Studio platform to do quite complex facilities,
and that's why I was suggesting that guidelines, based upon the
supported "MEF" framework could really benefit developers.

Of course these complex entities wouldn't be supported; but the
basic means by which a "plugin" type development could be done,
would have supported guidelines, and a framework within which
it could be done.

Coming back to the basics, and my understanding is simple, the
idea here is that only the "top level" NinjaScript like class would
even be known to the NinjaTrader system. Therefore, that
would be the only class where serialization/deserialization would
be performed. But that class would use MEF mechanisms to
activate one or more of its "plugins" which represent much
more complex subsystems.

Just offering it as an idea to extend the power of NinjaTrader
even further for trader/developers like me.

hyperscalper

[NinjaTrader_ChelseaB]

Hello hyperscalper,

I'm not certain of the specific code you are suggesting.

Are you sure you are referring to serializing strings for saving in xml?

Are you requesting that NinjaTrader does not create multiple instances of a script for different purposes like populating the strategies list with default properties?

[Hyper]

Hi.

Well, I'm thinking of "heavyweight indicators or strategies" and how it is that
these are deserialized for reactivation once they are added to a
chart entity, etc., within NinjaTrader at the moment.

It seems to me that the design does not consider complex objects
with many classes, and which allocate many resources.

If, for whatever reason, NinjaTrader decides it needs to revisit
properties, say, for an Indicator, it basically assumes that the
Indicator really doesn't have any significant "state" and that
it doesn't have any resources allocated, or additional threads
running; or a wide range of situations; and that can significantly
interfere with the stability of a really "heavyweight" and
complex object instance network.

So it doesn't so much relate to the process of serialization with
xml or any other representation; so much as how object state
can be disrupted and the consequences on complex dependent
code; as most NinjaScript single class file type objects don't
really have such state.

I realize that complex systems are "unsupported" because
of their very complexity; but if I were able to use a single
class possibly NinjaScript generated type of single class
"stub loader" and then activate and bind properly to one
or more DLL assemblies using the supported MEF
interface based C# approach; then it might make this
easier to manage; than if an Assembly DLL is presented
to the system, which it then tries to explore and
serialize. That dependent assembly's details would
be unknown.

I'm going down a rathole here, most likely, but the basic
idea would be for an Indicator or Strategy to be a very
"thin" object instance which NinjaTrader would know
about; but which would bind to an (ideally persistent)
dependent "server interface" so that...

Even if the runtime system blew away the top level
Indicator or Strategy object instance; it could be
re-created arbitrarily, and it could then reconnect
to its already activated "service object" which would
be unaffected by the destruction of the top level
object instance; but could simply reconnect to
its service, reinitialize it, and remain stable.

So I'm just throwing spaghetti on the wall and trying
to think of a runtime structure which would use
MEF to support "plugin" complex service structures,
which interface to NinjaTrader by a "thin" layer,
and which use MEF mechanisms to enhance
stability...

PROBABLY GETTING TOO COMPLEX FOR THE
SUPPORT FORUM, BUT I've certainly had lots of
trouble understanding the Strategy activations, and
then when these are running, having been attached
to a chart, the chart may decide to create another
instance, in such a way that it disrupts the functioning
and actually crashes all of NinjaTrader in the process.

So I'd like to be able to isolate the "guts" of a heavyweight
strategy, from the process of activation or re-activation
or serialization... whatever... of the "top level"
NinjaScript-like object which has all of those
resource dependencies. And the MEF structure
looks like the way to go, to achieve many
of the goals in a supported way.

I COULD BE WAY OFF TRACK, SINCE I've only been
at this for a few months myself.

Maybe when I get some time, I can look at how such
a thing could be structured, and how MEF could
make it possible to develop complex DLL Assemblies
which integrate better into the Indicator or Strategy
handling mechanisms of NT8.

hyperscalper

[NinjaTrader_ChelseaB]

Hello hyperscalper,

I really am not following.

When you server, are you talking about the connection?

In an active instance data methods are run from data events on a thread in the core of NinjaTrader. NinjaTrader will then update bar information and then run OnBarUpdate() / OnMarketData() overrides in your script.

Are you wanting partial classes or static properties?

https://ninjatrader.com/support/helpGuides/nt8/index.html?code_breaking_changes.htm#ImplementationChangesOverview

https://ninjatrader.com/support/help...ghtsub=partial

https://ninjatrader.com/support/forum/forum/historical-beta-archive/version-8-beta/83018-partial-class-does-not-work?p=712245#post712245

While there is some wpf binding going on behind the scenes, there is no binding on NinjaScript properties supported.

Can you, in as few words as possible, tell me the issue or error you experiencing you are trying to resolve?
Can you provide an export script that isn't compiling or is having an error that removes all of the logic?

I'm hoping for a single sentence with a feature request I may submit for you.

[bltdavid]

Well, I'm thinking of "heavyweight indicators or strategies" and how it is that
these are deserialized for reactivation ...

I COULD BE WAY OFF TRACK, SINCE I've only been
at this for a few months myself.

My friend, you are way off track.
You are getting the ideas behind serialization completely wrong.

Let's start with this simple concept:
Serialization is a fancy word to mean 'saving and storing parameters',
it has absolutely nothing to do with activation.

By 'saving and storing parameters', I mean the variables that you see
in the property grid when setting parameters for an Indicator or Strategy.

Those parameters in the property grid are actually exposed as public
properties using C# -- the property grid is the C# way of presenting a
standard organized GUI interface to set the public properties for an
instance of a class.

So, what is serialization?
It is the act of saving those public properties of a class instance to an
Xml file, such that the instance data persists on disk.

De-serialization is the opposite -- it is the act of initializing the public
properties from a previously saved Xml file -- ie, it loads parameters
by reading the Xml file and initializing the class variables.

That's it!

Like I said, the word serialization is a fancy word to describe saving
parameters to an Xml file -- and de-serialization describes loading
those parameters from an Xml file.

[In the finer details, serialization involves converting the property value
to a string, deserialization converts that string back to the type of the
property -- think of colors, the color red might be serialized to the string
"n:Red" which is then saved to the Xml file -- deserialization converts
the string "n:Red" back to the internal bit pattern needed to define the
specified color.]

Saying 'deserialized for activation' makes no sense. An instance of
a class is created by code, and in many cases NinjaTrader wants the
many many variables of that instance to be initialized in a certain way.
Well, de-serialization will initialize those values from an Xml file, so the
many many instance variables are simply borrowing the Xml file as
a container to 'persist' those instance values -- make sense?

Serialization is a simple concept. It has nothing to do with activation.
It is used throughout NinjaTrader to persist internal instances of many
different C# classes to disk as an Xml file ... that's all it does.

The terms Lightweight and Heavyweight, these terms have nothing
to do with serialization.

The Xml file contains values for the class variables, so that when a
new instance of that class is created, it's values can be re-populated
from previously saved values. 'Serialization' is the fancy name that
both C# and Java use to describe the code that saves those values
to the Xml file on disk.

De-serialization is the reverse, it loads the Xml file and re-initializes
the instance variables from the values in the Xml file.

Does that help?

[bltdavid]

From what I see, the re-activation of an Indicator, Strategy, or similar
within the NinjaTrader 8 system depends *entirely* upon successful
serialization and deserialization. If this is unsuccessful, then manual
re-activation of the facility is required *every time* NinjaTrader is
restarted.

No, it does not.
First off, 'activation' is not a programming term(*) in C# -- so I presume
you mean the creation of the Indicator or Strategy object.

Serialization and deserialization having nothing to do with the creation
of these objects, or any C# object. Nothing at all, nada, zilch, nyet.

Serialization and deserialization are optional techniques that only
come into play after an object is created.

Generally speaking,
Serialization means saving the instance variables to a file on disk.
Deserialization means initializing the instance variables from a file on disk.

Serialization and deserialization are techniques used for persistence
of instance data. These techniques are optional to the developers,
but once employed, if the Xml files saved to disk are corrupted, yes,
this could completely bring NinjaTrader to its knees.

'Manual re-activation' never occurs anyways. All objects of all classes
are recreated automatically, as needed, every time NinjaTrader.exe
starts up -- the many and varied Xml files are used to initialize the many
instance variables of these classes, as needed -- again, all automatically.

Make sense?

(*) - Ok, well, google "C# Activation" and the term does exist, but it has
a very specific meaning which varies depending on the context, so I now
say that the contextual use of that word (ie, as it relates to the serialization
and deserialization techniques) doesn't exist, since 'activation' is not a term
used by those techniques.

[bltdavid]

So I'm just respectfully requesting that some alternative
mechanism be provided so that assembly DLL's with
arbitrary internal complexity; be able to be modified
in implementation, but activated through a mechanism
which does not depend upon the current serialization
constraints

By 'activation', do you mean unloading and reloading of an assembly?

First off, it sounds like you're talking about a developer issue.
The most correct way to unload and reload an assembly that you
have just recompiled is to exit NinjaTrader and restart it.

If you want to dynamically load and unload your own assemblies,
well, go ahead, you can do that, but that is an advanced C#/.NET
concept and is not a supported part of the NinjaScript framework.

[EDIT: For all of my 30+ years of software engineering experience,
and my expertise with NinjaTrader, I've never had to care about
writing advanced code to unload & reload a freshly re-compiled
assembly. Why? Because it's a waste of my time caring about it.
I just exit NinjaTrader and restart it -- this solves all my problems
all the time, fast & easy, no mess, no fuss, no fancy knowledge
required.]

Are you having a specific problem that you're experiencing after
you recompile your assembly? What is that problem?

[Hyper]

Thanks for the comments from so many knowledgeable developers !

I have a couple of extremely complex both Indicators and Strategies,
and I'm able to add them to NinjaTrader on a restart of the entire
platform; but I am not currently able to allow NinjaTrader to restart
them automatically either because of serialization/deserialization
issues with the Assembly DLL, _or_ because of programming
errors perhaps on my own part.

Rest assured I know what serialization and deserialization are, and
what object instances are, and all of those things which you've
all kindly mentioned above. I appreciate that you've taken the time
to reply.

I'll take a pause on this; and investigate further, most likely using
an MEF implementation which will use a "thin" top level class
for the Indicator, or the Strategy, and which will then activate
the Assembly DLL through MEF mechanisms; and see whether
I am better able to get stability and behaviors which single-class
NinjaScript style Indicators and Strategy should exhibit.

Then once I have a handle on all of this, I'll try to report a
framework for this approach, or at least some guidelines
which might help others who are also working with complex
"heavyweight" Indicator or Strategy instances.

Again, I really appreciate your comments; and this isn't so
much a "support" issue; but I had the idea that it could ultimately
result in some guidelines for development of complex multi-class,
multi-threaded, etc. components within an Indicator or Strategy
framework; which could make things a lot easier for
developers.

And I realize that NinjaTrader platform support can't really
support complex developments like this; but there are plenty of
cases when you go "above and beyond the call of duty" and
so many of you give guidelines to at least point developers
in the right direction; and that is what makes NinjaTrader
Support stand out as really exceptional !! So Thanks for
all you do; it is very much appreciated!

[edit] So what I'm saying is that the great NinjaScript approach is
really based upon a single class file idea; where there isn't much
"state" in the Indicator or Strategy element being developed.
It seems to me that maintaining a "thin" single class for very
much more "complex" applications is the best approach, in line
with the way NinjaScript works; but when extremely heavyweight
facilities are involved; those should be separate Assembly DLL's
which represent "plugins" using the MEF framework; and which
the "thin" "top-level" class instance then activates using the MEF
interface-based framework; and that aspect of the functionality
is really hidden from the top-level activation class which
NinjaTrader sees... if that makes any sense. Well, I know it
does make sense; and I think it's the key to better management
of heavyweight Assembly DLLs; but it's up to me or someone
to come up with guidelines and show how that may be true (or not)


hyperscalper

[bltdavid]

I have a couple of extremely complex both Indicators and Strategies,
and I'm able to add them to NinjaTrader on a restart of the entire
platform; but I am not currently able to allow NinjaTrader to restart
them automatically either because of serialization/deserialization
issues with the Assembly DLL, _or_ because of programming
errors perhaps on my own part.

Unfortunately, I'm still confused.

I hope I'm not wasting my time trying to understand your problem.
I mean, you have my interest, but I still don't understand what it is
you're trying to solve ... and yet, I'd like to help you solve it.

It sounds deep, it sounds interesting, like I might learn something new,
but your generous writings don't provide much in the way of specifics.

You need to be able to get specific.
Otherwise, how can anyone help you?

I feel like a police interrogator, which is not a good thing.
So, forgive me, I'm not being mean or difficult.
I'm still trying to understand.

By 'add them to NinjaTrader', do you mean manually adding your
Indicator and/or Strategy to a chart?
What do you mean by 'restart them automatically'?
What 'serialization/deserialization issues' are you referring to?

Does the Log tab show any error messages?
Do you have any example code that illustrates the problems you're experiencing?

Most public properties of most Indicators and Strategies are values of simple
types, such as an enum, bool, int, double, etc. Do you have any public properties
that are reference types?

Try this experiment: After adding the Indicator to your chart, are you able to
successfully save a chart template with your indicator?

The act of saving a chart template requires serialization to an Xml file, and if
any Indicator on the chart cannot be successfully serialized, then this procedure
should produce an error message. Do you get any error messages?

[bltdavid]

I'll take a pause on this; and investigate further, most likely using
an MEF implementation which will use a "thin" top level class
for the Indicator, or the Strategy, and which will then activate
the Assembly DLL through MEF mechanisms; and see whether
I am better able to get stability and behaviors which single-class
NinjaScript style Indicators and Strategy should exhibit.

By MEF implementation, are you referring to Managed Extensibility Framework?

Why do you need to bother to do that?
Isn't that overkill? What problems are you trying to solve that you
think using MEF is the way to go?

Isn't adding all your assemblies as references good enough?
After you do this, are you still getting some kind of error?
What errors are these? When do they occur?

Have you read this and this?

PS:
When I was first learning NT7, I used to get an import error for my assembly.
Turns out it was related to my Indicator having a public property of an enum type,
which was defined in a separate namespace. To solve the error, I had to qualify
the enum type name with the full namespace name.

The aggravating thing about that error was this problem never manifested until
after I exported the assembly and tried to import for the first time into a fresh
NT installation.

I presume you're also seeing something weird or strange?
Can you describe it more specifically?
Can you take some screenshots?
Can you make a screencast video?
Can you provide example code to recreate it?

[Hyper]

Thanks so much for the comments. Like I said, since the issue of writing complex
Indicators and Strategies is itself a complex issue; I'm going to take a pause, but
I promise to update on any solution I come up with which makes these complex
entities "play well" with NinjaTrader's natural abilities to serialize, and then
subsequently on restart, to naturally re-activate these facilities as expected.

Major issues are: 1) complex state and resource management, 2) potentially
wanting to "unload" and "reload" an assembly implementation without having
to restart NinjaTrader, and 3) just guidelines as to how the MEF "plugin"
architecture is likely to be the solution to many of these issues.

Right now; my stuff runs incredibly well; however, when NinjaTrader is
restarted, these facilities cannot be reactivated; so I have to add them
to their respective charts manually, each time. That's not a huge issue
since everything runs fine. But it isn't the ideal situation, so I'll be
looking into the "plugin" approach provided by the MEF architecture
which should help in addressing many of these issues.

So I do promise to share any findings when I can get time to
work on the issues. I think this MEF approach is likely to
make NinjaTrader custom developments more stable in
many ways; and I think that's obvious once we look
into it further; even for ordinary custom developers
who would be able to follow guidelines as to how to
structure their code.

Restarting NinjaTrader really isn't a great requirement,
for example, when an Assembly implementation
changes. Ideally, when running automated strategies,
an implementation "plugin" should be able to be
replace/unloaded/reloaded dynamically without
having to stop the host NinjaTrader process. And
that's a big part of the goal for me.

hyperscalper