Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Serialization revisited; and is there an alternative?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Serialization revisited; and is there an alternative?

    OK, I've just gotten "my sea legs" in complex NinjaTrader development
    implemented as multi-class assembly DLL's.

    Now, I fully realize that you simply can't help the "custom developers"
    in every aspect of their work; as NinjaTrader is so flexilble and
    extensible. That makes the support effort unbounded and so I
    completely sympathize. However...

    From what I see, the re-activation of an Indicator, Strategy, or similar
    within the NinjaTrader 8 system depends *entirely* upon successful
    serialization and deserialization. If this is unsuccessful, then manual
    re-activation of the facility is required *every time* NinjaTrader is
    restarted.

    First of all, is this correct? I think it is... So I am proposing that there
    be *an alternative* method for complex assembly DLL's which
    *does not depend upon serialization*.

    That an Indicator, Strategy, etc. be able to specify reactivation
    through a mechanism which is not serialization/deserialization.

    Is there already such a facility? and, if not, then I propose that
    you consider providing one.

    I'm not trying to be irritating here, since I'm so impressed by the
    platform; but one of the biggest stumbling blocks is the seeming
    inability to satisfy the re-activation requirements through
    the serialization mechanisms; and especially on update of
    an assembly DLL, even though the same parameter interface
    may be maintained, the inability for NinjaTrader 8 to correctly
    activate it; thus requiring a manual intervention for activation
    each time the platform is restarted.

    So I'm just respectfully requesting that some alternative
    mechanism be provided so that assembly DLL's with
    arbitrary internal complexity; be able to be modified
    in implementation, but activated through a mechanism
    which does not depend upon the current serialization
    constraints

    Even a workaround at this point would be helpful; for
    example, maybe a "stub" high level NinjaScript generated
    object (which would be reliably serialized/deserialized)
    could then internally "dynamically load" the underlying DLL
    assembly. Maybe that would bypass my difficulties, and
    provide a path for DLL assembly activation that might
    resolve the issues? At this point, any such workaround would
    be welcome ! Any solution would be welcomed. Thanks in advance...

    [edit] Like if the "serializable stub loader" would dynamically allocate the
    DLL and use its entry points as a "delegate" implementation.
    I have a feeling that could work somehow...

    hyperscalper
    Last edited by Hyper; 09-19-2020, 10:21 AM. Reason: more info

    #2
    [EDIT] this post should be discounted and, instead, the next post
    describes a fully supported method for developing dynamically
    bound "plugin" application components.

    What looks like an organized "interface based" approach to this
    is described here:
    https://stackoverflow.com/questions/...-method-in-dll

    I guess I'll try to do something like this; but if there were a provided
    facility for safe method delegation, then others could use the approach.

    Maybe someone from the community has already done this ?

    hyperscalper
    Last edited by Hyper; 09-19-2020, 04:12 PM. Reason: superceded

    Comment


      #3
      I took a look further into this. No doubt NinjaTrader platform engineers are
      aware of this, but I'll post links anyway. The best way to understand this facility
      is that it's a fully supported .NET 4.5 facility to create a "plugin architecture"
      by which subsystems can be loaded, bound to, and unloaded cleanly.

      An application loader, or "stub" could use this mechanism to dynamically load
      and bind to a DLL assembly which provides some or all of the functionality.
      In this case, the goal would be NinjaTrader Indicators, Strategies, etc. whose
      implementation lies in the "plugin".

      the NuGet package is available here: https://www.nuget.org/packages/microsoft.composition

      https://docs.microsoft.com/en-us/dot...ectedfrom=MSDN

      That's probably enough to get someone started with this; and it's clearly for Advanced requirements
      only. I'll give it a shot when I get time; but it seems this could be a path toward better
      integrating complex DLL Assemblies as "plugins" in a fully supported manner.

      hyperscalper

      Comment


        #4
        Hello hyperscalper,

        Serialization is converting object values to strings so these can be saved in xml files such as workspaces and templates, and then de-serializing is reading from the xml file (workspace or template) and converting back to an object.

        Inputs are basically all that would be serialized and saved, and only if they are types that are not easily converted to string such as numbers, bools, and enums.

        Below is a link to the help guide on serializing brushes with an example.
        https://ninjatrader.com/support/help...definedbrushes

        As far as referencing other classes, this would be outside of what is supported by NinjaTrader, but I have an example of shared static classes.
        https://ninjatrader.com/support/foru...245#post712245

        Below is a link to a video on exporting shared classes.
        https://drive.google.com/file/d/1D9q...ew?usp=sharing

        Dlls must be referenced for the code to usable.
        https://ninjatrader.com/support/help...indicators.htm
        Chelsea B.NinjaTrader Customer Service

        Comment


          #5
          Thanks very much for the info, and for addressing the issue. Originally, I thought
          in this way: "For a Strategy, the "root" or "top level" class would be the one
          that would need serialization/deserialization, and all other referenced
          objects would have their fields marked no serialize"

          As I developed my Strategy, in another post, you'll recall we had all of those
          "clone instances" to untangle; where only the Strategy whose state advanced
          _beyond Configure_ was the instance that was going to survive as the
          actual running Strategy subclass instance.

          I'm just suggesting whether NinjaTrader might implement an alternative
          approach which activates the "root object", e.g. the MyStrategy just once
          and, of course, uses Reflection to determine any Properties.

          Surely there are excellent reasons for doing it the way NinjaTrader 8
          does it; but it seems to me right now, that an alternate method of
          activation and querying the object could be provided, that would be
          more suited to complex multi-class solutions packaged in assembly
          DLL's, and make it easier for the developer.

          Let me say that I just _love_ NinjaTrader 8 and, having overcome
          much of the initial learning curve, it continues to impress me !

          This is just a suggestion of something which would help those of
          us who use the Visual Studio platform to do quite complex facilities,
          and that's why I was suggesting that guidelines, based upon the
          supported "MEF" framework could really benefit developers.

          Of course these complex entities wouldn't be supported; but the
          basic means by which a "plugin" type development could be done,
          would have supported guidelines, and a framework within which
          it could be done.

          Coming back to the basics, and my understanding is simple, the
          idea here is that only the "top level" NinjaScript like class would
          even be known to the NinjaTrader system. Therefore, that
          would be the only class where serialization/deserialization would
          be performed. But that class would use MEF mechanisms to
          activate one or more of its "plugins" which represent much
          more complex subsystems.

          Just offering it as an idea to extend the power of NinjaTrader
          even further for trader/developers like me.

          hyperscalper


          Comment


            #6
            Hello hyperscalper,

            I'm not certain of the specific code you are suggesting.

            Are you sure you are referring to serializing strings for saving in xml?

            Are you requesting that NinjaTrader does not create multiple instances of a script for different purposes like populating the strategies list with default properties?
            Chelsea B.NinjaTrader Customer Service

            Comment


              #7
              Hi.

              Well, I'm thinking of "heavyweight indicators or strategies" and how it is that
              these are deserialized for reactivation once they are added to a
              chart entity, etc., within NinjaTrader at the moment.

              It seems to me that the design does not consider complex objects
              with many classes, and which allocate many resources.

              If, for whatever reason, NinjaTrader decides it needs to revisit
              properties, say, for an Indicator, it basically assumes that the
              Indicator really doesn't have any significant "state" and that
              it doesn't have any resources allocated, or additional threads
              running; or a wide range of situations; and that can significantly
              interfere with the stability of a really "heavyweight" and
              complex object instance network.

              So it doesn't so much relate to the process of serialization with
              xml or any other representation; so much as how object state
              can be disrupted and the consequences on complex dependent
              code; as most NinjaScript single class file type objects don't
              really have such state.

              I realize that complex systems are "unsupported" because
              of their very complexity; but if I were able to use a single
              class possibly NinjaScript generated type of single class
              "stub loader" and then activate and bind properly to one
              or more DLL assemblies using the supported MEF
              interface based C# approach; then it might make this
              easier to manage; than if an Assembly DLL is presented
              to the system, which it then tries to explore and
              serialize. That dependent assembly's details would
              be unknown.

              I'm going down a rathole here, most likely, but the basic
              idea would be for an Indicator or Strategy to be a very
              "thin" object instance which NinjaTrader would know
              about; but which would bind to an (ideally persistent)
              dependent "server interface" so that...

              Even if the runtime system blew away the top level
              Indicator or Strategy object instance; it could be
              re-created arbitrarily, and it could then reconnect
              to its already activated "service object" which would
              be unaffected by the destruction of the top level
              object instance; but could simply reconnect to
              its service, reinitialize it, and remain stable.

              So I'm just throwing spaghetti on the wall and trying
              to think of a runtime structure which would use
              MEF to support "plugin" complex service structures,
              which interface to NinjaTrader by a "thin" layer,
              and which use MEF mechanisms to enhance
              stability...

              PROBABLY GETTING TOO COMPLEX FOR THE
              SUPPORT FORUM, BUT I've certainly had lots of
              trouble understanding the Strategy activations, and
              then when these are running, having been attached
              to a chart, the chart may decide to create another
              instance, in such a way that it disrupts the functioning
              and actually crashes all of NinjaTrader in the process.

              So I'd like to be able to isolate the "guts" of a heavyweight
              strategy, from the process of activation or re-activation
              or serialization... whatever... of the "top level"
              NinjaScript-like object which has all of those
              resource dependencies. And the MEF structure
              looks like the way to go, to achieve many
              of the goals in a supported way.

              I COULD BE WAY OFF TRACK, SINCE I've only been
              at this for a few months myself.

              Maybe when I get some time, I can look at how such
              a thing could be structured, and how MEF could
              make it possible to develop complex DLL Assemblies
              which integrate better into the Indicator or Strategy
              handling mechanisms of NT8.

              hyperscalper

              Comment


                #8
                Hello hyperscalper,

                I really am not following.

                When you server, are you talking about the connection?

                In an active instance data methods are run from data events on a thread in the core of NinjaTrader. NinjaTrader will then update bar information and then run OnBarUpdate() / OnMarketData() overrides in your script.

                Are you wanting partial classes or static properties?
                https://ninjatrader.com/support/help...hangesOverview
                ​​​​​​https://ninjatrader.com/support/help...ghtsub=partial
                https://ninjatrader.com/support/foru...245#post712245

                While there is some wpf binding going on behind the scenes, there is no binding on NinjaScript properties supported.

                Can you, in as few words as possible, tell me the issue or error you experiencing you are trying to resolve?
                Can you provide an export script that isn't compiling or is having an error that removes all of the logic?

                I'm hoping for a single sentence with a feature request I may submit for you.
                Chelsea B.NinjaTrader Customer Service

                Comment


                  #9
                  Originally posted by Hyper View Post
                  Well, I'm thinking of "heavyweight indicators or strategies" and how it is that
                  these are deserialized for reactivation ...

                  I COULD BE WAY OFF TRACK, SINCE I've only been
                  at this for a few months myself.
                  My friend, you are way off track.
                  You are getting the ideas behind serialization completely wrong.

                  Let's start with this simple concept:
                  Serialization is a fancy word to mean 'saving and storing parameters',
                  it has absolutely nothing to do with activation.

                  By 'saving and storing parameters', I mean the variables that you see
                  in the property grid when setting parameters for an Indicator or Strategy.

                  Those parameters in the property grid are actually exposed as public
                  properties using C# -- the property grid is the C# way of presenting a
                  standard organized GUI interface to set the public properties for an
                  instance of a class.

                  So, what is serialization?
                  It is the act of saving those public properties of a class instance to an
                  Xml file, such that the instance data persists on disk.

                  De-serialization
                  is the opposite -- it is the act of initializing the public
                  properties from a previously saved Xml file -- ie, it loads parameters
                  by reading the Xml file and initializing the class variables.

                  That's it!

                  Like I said, the word serialization is a fancy word to describe saving
                  parameters to an Xml file -- and de-serialization describes loading
                  those parameters from an Xml file.

                  [In the finer details, serialization involves converting the property value
                  to a string, deserialization converts that string back to the type of the
                  property -- think of colors, the color red might be serialized to the string
                  "n:Red" which is then saved to the Xml file -- deserialization converts
                  the string "n:Red" back to the internal bit pattern needed to define the
                  specified color.]

                  Saying 'deserialized for activation' makes no sense. An instance of
                  a class is created by code, and in many cases NinjaTrader wants the
                  many many variables of that instance to be initialized in a certain way.
                  Well, de-serialization will initialize those values from an Xml file, so the
                  many many instance variables are simply borrowing the Xml file as
                  a container to 'persist' those instance values -- make sense?

                  Serialization is a simple concept. It has nothing to do with activation.
                  It is used throughout NinjaTrader to persist internal instances of many
                  different C# classes to disk as an Xml file ... that's all it does.

                  The terms Lightweight and Heavyweight, these terms have nothing
                  to do with serialization.

                  The Xml file contains values for the class variables, so that when a
                  new instance of that class is created, it's values can be re-populated
                  from previously saved values. 'Serialization' is the fancy name that
                  both C# and Java use to describe the code that saves those values
                  to the Xml file on disk.

                  De-serialization is the reverse, it loads the Xml file and re-initializes
                  the instance variables from the values in the Xml file.

                  Does that help?
                  Last edited by bltdavid; 09-24-2020, 05:01 PM.

                  Comment


                    #10
                    Originally posted by Hyper View Post
                    From what I see, the re-activation of an Indicator, Strategy, or similar
                    within the NinjaTrader 8 system depends *entirely* upon successful
                    serialization and deserialization. If this is unsuccessful, then manual
                    re-activation of the facility is required *every time* NinjaTrader is
                    restarted.
                    No, it does not.
                    First off, 'activation' is not a programming term(*) in C# -- so I presume
                    you mean the creation of the Indicator or Strategy object.

                    Serialization and deserialization having nothing to do with the creation
                    of these objects, or any C# object. Nothing at all, nada, zilch, nyet.

                    Serialization and deserialization are optional techniques that only
                    come into play after an object is created.

                    Generally speaking,
                    Serialization means saving the instance variables to a file on disk.
                    Deserialization means initializing the instance variables from a file on disk.

                    Serialization and deserialization are techniques used for persistence
                    of instance data. These techniques are optional to the developers,
                    but once employed, if the Xml files saved to disk are corrupted, yes,
                    this could completely bring NinjaTrader to its knees.

                    'Manual re-activation' never occurs anyways. All objects of all classes
                    are recreated automatically, as needed, every time NinjaTrader.exe
                    starts up -- the many and varied Xml files are used to initialize the many
                    instance variables of these classes, as needed -- again, all automatically.

                    Make sense?

                    (*) - Ok, well, google "C# Activation" and the term does exist, but it has
                    a very specific meaning which varies depending on the context, so I now
                    say that the contextual use of that word (ie, as it relates to the serialization
                    and deserialization techniques) doesn't exist, since 'activation' is not a term
                    used by those techniques.
                    Last edited by bltdavid; 09-24-2020, 11:01 PM. Reason: Yes, the term 'activation' exists, but not in context of serialization.

                    Comment


                      #11
                      Originally posted by Hyper View Post
                      So I'm just respectfully requesting that some alternative
                      mechanism be provided so that assembly DLL's with
                      arbitrary internal complexity; be able to be modified
                      in implementation
                      , but activated through a mechanism
                      which does not depend upon the current serialization
                      constraints
                      By 'activation', do you mean unloading and reloading of an assembly?

                      First off, it sounds like you're talking about a developer issue.
                      The most correct way to unload and reload an assembly that you
                      have just recompiled
                      is to exit NinjaTrader and restart it.

                      If you want to dynamically load and unload your own assemblies,
                      well, go ahead, you can do that, but that is an advanced C#/.NET
                      concept and is not a supported part of the NinjaScript framework.

                      [EDIT: For all of my 30+ years of software engineering experience,
                      and my expertise with NinjaTrader, I've never had to care about
                      writing advanced code to unload & reload a freshly re-compiled
                      assembly. Why? Because it's a waste of my time caring about it.
                      I just exit NinjaTrader and restart it -- this solves all my problems
                      all the time, fast & easy, no mess, no fuss, no fancy knowledge
                      required.]

                      Are you having a specific problem that you're experiencing after
                      you recompile your assembly? What is that problem?
                      Last edited by bltdavid; 09-24-2020, 11:35 PM.

                      Comment


                        #12
                        Thanks for the comments from so many knowledgeable developers !

                        I have a couple of extremely complex both Indicators and Strategies,
                        and I'm able to add them to NinjaTrader on a restart of the entire
                        platform; but I am not currently able to allow NinjaTrader to restart
                        them automatically either because of serialization/deserialization
                        issues with the Assembly DLL, _or_ because of programming
                        errors perhaps on my own part.

                        Rest assured I know what serialization and deserialization are, and
                        what object instances are, and all of those things which you've
                        all kindly mentioned above. I appreciate that you've taken the time
                        to reply.

                        I'll take a pause on this; and investigate further, most likely using
                        an MEF implementation which will use a "thin" top level class
                        for the Indicator, or the Strategy, and which will then activate
                        the Assembly DLL through MEF mechanisms; and see whether
                        I am better able to get stability and behaviors which single-class
                        NinjaScript style Indicators and Strategy should exhibit.

                        Then once I have a handle on all of this, I'll try to report a
                        framework for this approach, or at least some guidelines
                        which might help others who are also working with complex
                        "heavyweight" Indicator or Strategy instances.

                        Again, I really appreciate your comments; and this isn't so
                        much a "support" issue; but I had the idea that it could ultimately
                        result in some guidelines for development of complex multi-class,
                        multi-threaded, etc. components within an Indicator or Strategy
                        framework; which could make things a lot easier for
                        developers.

                        And I realize that NinjaTrader platform support can't really
                        support complex developments like this; but there are plenty of
                        cases when you go "above and beyond the call of duty" and
                        so many of you give guidelines to at least point developers
                        in the right direction; and that is what makes NinjaTrader
                        Support stand out as really exceptional !! So Thanks for
                        all you do; it is very much appreciated!

                        [edit] So what I'm saying is that the great NinjaScript approach is
                        really based upon a single class file idea; where there isn't much
                        "state" in the Indicator or Strategy element being developed.
                        It seems to me that maintaining a "thin" single class for very
                        much more "complex" applications is the best approach, in line
                        with the way NinjaScript works; but when extremely heavyweight
                        facilities are involved; those should be separate Assembly DLL's
                        which represent "plugins" using the MEF framework; and which
                        the "thin" "top-level" class instance then activates using the MEF
                        interface-based framework; and that aspect of the functionality
                        is really hidden from the top-level activation class which
                        NinjaTrader sees... if that makes any sense. Well, I know it
                        does make sense; and I think it's the key to better management
                        of heavyweight Assembly DLLs; but it's up to me or someone
                        to come up with guidelines and show how that may be true (or not)


                        hyperscalper
                        Last edited by Hyper; 09-25-2020, 10:23 AM. Reason: some more

                        Comment


                          #13
                          Originally posted by Hyper View Post
                          I have a couple of extremely complex both Indicators and Strategies,
                          and I'm able to add them to NinjaTrader on a restart of the entire
                          platform; but I am not currently able to allow NinjaTrader to restart
                          them automatically
                          either because of serialization/deserialization
                          issues
                          with the Assembly DLL, _or_ because of programming
                          errors perhaps on my own part.
                          Unfortunately, I'm still confused.

                          I hope I'm not wasting my time trying to understand your problem.
                          I mean, you have my interest, but I still don't understand what it is
                          you're trying to solve ... and yet, I'd like to help you solve it.

                          It sounds deep, it sounds interesting, like I might learn something new,
                          but your generous writings don't provide much in the way of specifics.

                          You need to be able to get specific.
                          Otherwise, how can anyone help you?

                          I feel like a police interrogator, which is not a good thing.
                          So, forgive me, I'm not being mean or difficult.
                          I'm still trying to understand.

                          By 'add them to NinjaTrader', do you mean manually adding your
                          Indicator and/or Strategy to a chart?
                          What do you mean by 'restart them automatically'?
                          What 'serialization/deserialization issues' are you referring to?

                          Does the Log tab show any error messages?
                          Do you have any example code that illustrates the problems you're experiencing?

                          Most public properties of most Indicators and Strategies are values of simple
                          types, such as an enum, bool, int, double, etc. Do you have any public properties
                          that are reference types?

                          Try this experiment: After adding the Indicator to your chart, are you able to
                          successfully save a chart template with your indicator?

                          The act of saving a chart template requires serialization to an Xml file, and if
                          any Indicator on the chart cannot be successfully serialized, then this procedure
                          should produce an error message. Do you get any error messages?

                          Comment


                            #14
                            Originally posted by Hyper View Post
                            I'll take a pause on this; and investigate further, most likely using
                            an MEF implementation which will use a "thin" top level class
                            for the Indicator, or the Strategy, and which will then activate
                            the Assembly DLL through MEF mechanisms; and see whether
                            I am better able to get stability and behaviors which single-class
                            NinjaScript style Indicators and Strategy should exhibit.
                            By MEF implementation, are you referring to Managed Extensibility Framework?

                            Why do you need to bother to do that?
                            Isn't that overkill? What problems are you trying to solve that you
                            think using MEF is the way to go?

                            Isn't adding all your assemblies as references good enough?
                            After you do this, are you still getting some kind of error?
                            What errors are these? When do they occur?

                            Have you read this and this?

                            PS:
                            When I was first learning NT7, I used to get an import error for my assembly.
                            Turns out it was related to my Indicator having a public property of an enum type,
                            which was defined in a separate namespace. To solve the error, I had to qualify
                            the enum type name with the full namespace name.

                            The aggravating thing about that error was this problem never manifested until
                            after I exported the assembly and tried to import for the first time into a fresh
                            NT installation.

                            I presume you're also seeing something weird or strange?
                            Can you describe it more specifically?
                            Can you take some screenshots?
                            Can you make a screencast video?
                            Can you provide example code to recreate it?

                            Comment


                              #15
                              Thanks so much for the comments. Like I said, since the issue of writing complex
                              Indicators and Strategies is itself a complex issue; I'm going to take a pause, but
                              I promise to update on any solution I come up with which makes these complex
                              entities "play well" with NinjaTrader's natural abilities to serialize, and then
                              subsequently on restart, to naturally re-activate these facilities as expected.

                              Major issues are: 1) complex state and resource management, 2) potentially
                              wanting to "unload" and "reload" an assembly implementation without having
                              to restart NinjaTrader, and 3) just guidelines as to how the MEF "plugin"
                              architecture is likely to be the solution to many of these issues.

                              Right now; my stuff runs incredibly well; however, when NinjaTrader is
                              restarted, these facilities cannot be reactivated; so I have to add them
                              to their respective charts manually, each time. That's not a huge issue
                              since everything runs fine. But it isn't the ideal situation, so I'll be
                              looking into the "plugin" approach provided by the MEF architecture
                              which should help in addressing many of these issues.

                              So I do promise to share any findings when I can get time to
                              work on the issues. I think this MEF approach is likely to
                              make NinjaTrader custom developments more stable in
                              many ways; and I think that's obvious once we look
                              into it further; even for ordinary custom developers
                              who would be able to follow guidelines as to how to
                              structure their code.

                              Restarting NinjaTrader really isn't a great requirement,
                              for example, when an Assembly implementation
                              changes. Ideally, when running automated strategies,
                              an implementation "plugin" should be able to be
                              replace/unloaded/reloaded dynamically without
                              having to stop the host NinjaTrader process. And
                              that's a big part of the goal for me.

                              hyperscalper

                              Comment

                              Latest Posts

                              Collapse

                              Topics Statistics Last Post
                              Started by John.Wehland, Today, 05:51 PM
                              0 responses
                              2 views
                              0 likes
                              Last Post John.Wehland  
                              Started by pennywise, Today, 05:47 PM
                              0 responses
                              3 views
                              0 likes
                              Last Post pennywise  
                              Started by averoos, Today, 03:51 PM
                              0 responses
                              11 views
                              0 likes
                              Last Post averoos
                              by averoos
                               
                              Started by crazyhorse2393, Today, 03:31 PM
                              2 responses
                              14 views
                              0 likes
                              Last Post crazyhorse2393  
                              Started by wooofad, Today, 02:56 PM
                              1 response
                              11 views
                              0 likes
                              Last Post NinjaTrader_Kate  
                              Working...
                              X