Object reference not set to an instance of an object

[anachronist]

Posting this to the 'General programming' thread because it involves both indicators and strategies.

I've been using NT7 for years and only now I run across this problem. It's related to running a strategy against a collection -- something I never needed to do before now.

I have written a library of several indicators with public properties. When I reference one of these from another indicator or from a strategy, it has worked very well for my code to follow this template:

// Variables
Foobar fb;    // Foobar() is an indicator that has a number of public properties I need

public override void Initialize() {
    // do stuff
}

public override void OnStartUp() {
   fb = Foobar(parameter1, parameter2, etc); // set handle to instance of Foobar()
   // do other stuff
}

public override void OnBarUpdate() {
    double val = fb[0];   // get primary output value of indicator for this bar
    double prop1 = fb.Property1;  // get a public property of Foobar() for this bar
    double prop2 = fb.Property2;  // etc.
    // do other stuff
}

I'm doing this because I want just one instance of Foobar being executed. Also, doing it this way allows me to have only one place where parameters are passed to the external indicator Foobar().

Now THIS HAS WORKED VERY WELL, for both strategies and indicators, until....

...I try to run a strategy against a collection. Works fine against a single instrument. Against a collection, I get this error message in the output window:

Error on calling 'OnStartUp' method for indicator 'MyIndicator': Object reference not set to an instance of an object.

If I understood what is going on when running a strategy against a collection versus a single instrument, I could figure out what to do.

I may need to correct/update some stuff I've uploaded to the file sharing area too, but I don't see a way to change anything I've submitted once uploaded.

(I also need to provide corrections to some highly optimized statistical indicators I contributed that NinjaTrader has informed me will be adopted for NT8.)

Thanks.
Alex

[NinjaTrader_PatrickH]

Hello Alex,

Thank you for your post.

We will look into this further and follow up with you shortly.

[anachronist]

Update: I have isolated the problem to the Add() method.

When I Add() an indicator to a strategy, and that indicator in turn references another indicator's properties in the way I describe above, it works fine for a single instrument, but I get that error message when I try to run the strategy against a collection.

The solution is to comment out those lines, since Add() is used (in my case) just as a visual aid in the strategy.

I learned something else about running a strategy against collections: don't rely on the "variables" region or Initialize() to initialize any variable values on each iteration through the collection. All variables that absolutely must start out with a specific value should be initialized in OnStartUp().

-Alex

[conanmishler]

Update:
I learned something else about running a strategy against collections: don't rely on the "variables" region or Initialize() to initialize any variable values on each iteration through the collection. All variables that absolutely must start out with a specific value should be initialized in OnStartUp().

-Alex

Very interesting, Alex. Thanks for sharing that!
That might explain the problem I'm seeing, which I suspect is rooted in this behavior of NT.

I'm not running backtest against a collection of instruments like you. Instead, I'm running optimization on a single instrument, giving the optimizer a variety of parameters to test.

The curious thing is that the optimization runs fine when I don't ask the optimizer to test a variety of parameters. It also runs fine when I *do* ask it to test a variety of parameters, but on a computer with only one or two parallel processors. But when I ask the optimizer to test a variety of parameters on a computer with many parallel processors, then I get the infamous, "object reference not set to an instance of an object."

I believe what is happening in both of our cases is explained, at least in part, by this quote from the NT manual:

"Multi-threading basically allows NinjaTrader to take advantage of multi-core CPUs commonplace in modern computing to do multiple tasks at the same time. What this means for an optimization run is that multiple iterations can be tested by utilizing the various cores available to the computer. Should you be using custom resources like text files, static members, etc. it is important to protect your resources from concurrent access."

I am declaring some "static" variables in my code, but I would think each parallel processor would be working from a completely different area of memory, having copied the entire executable of my strategy to a different area of the stack when the new process was forked. Just using "static" variables in one's code shouldn't break multi-threaded optimization, should it?

In my case, different combinations of parameters are running in parallel, and in your case, different instruments are running in parallel. But somehow, perhaps due to what you discovered about lack of consistent initialization in the "variables" section and the Initialize() section, we both got the same error.

It seems odd that I should be required to do all my critical variable initializations in OnStartUp(). Is that a feature or a bug?

Thanks again for your post, Alex. I'll give the OnStartUp() initialization idea a try.

I also hope someone from NT will read this post and answer the questions I posed. I am more than happy to let a NT Techie remotely log in and watch me reproduce the problem.

-Conan

[koganam]

Very interesting, Alex. Thanks for sharing that!
That might explain the problem I'm seeing, which I suspect is rooted in this behavior of NT.

I'm not running backtest against a collection of instruments like you. Instead, I'm running optimization on a single instrument, giving the optimizer a variety of parameters to test.

The curious thing is that the optimization runs fine when I don't ask the optimizer to test a variety of parameters. It also runs fine when I *do* ask it to test a variety of parameters, but on a computer with only one or two parallel processors. But when I ask the optimizer to test a variety of parameters on a computer with many parallel processors, then I get the infamous, "object reference not set to an instance of an object."

I believe what is happening in both of our cases is explained, at least in part, by this quote from the NT manual:

"Multi-threading basically allows NinjaTrader to take advantage of multi-core CPUs commonplace in modern computing to do multiple tasks at the same time. What this means for an optimization run is that multiple iterations can be tested by utilizing the various cores available to the computer. Should you be using custom resources like text files, static members, etc. it is important to protect your resources from concurrent access."

I am declaring some "static" variables in my code, but I would think each parallel processor would be working from a completely different area of memory, having copied the entire executable of my strategy to a different area of the stack when the new process was forked. Just using "static" variables in one's code shouldn't break multi-threaded optimization, should it?

In my case, different combinations of parameters are running in parallel, and in your case, different instruments are running in parallel. But somehow, perhaps due to what you discovered about lack of consistent initialization in the "variables" section and the Initialize() section, we both got the same error.

It seems odd that I should be required to do all my critical variable initializations in OnStartUp(). Is that a feature or a bug?

Thanks again for your post, Alex. I'll give the OnStartUp() initialization idea a try.

I also hope someone from NT will read this post and answer the questions I posed. I am more than happy to let a NT Techie remotely log in and watch me reproduce the problem.

-Conan

In C#, only ONE instance of a static variable may exist. That is almost the entire and only reason to ever declare a static variable. There will be no copies of a static variable. To quote from one of the many M$ articles on the subject: "Only one copy of a static member exists, regardless of how many instances of the class are created. Static methods and properties cannot access non-static fields and events in their containing type, and they cannot access an instance variable of any object unless it is explicitly passed in a method parameter." (emphasis mine).

ref: https://msdn.microsoft.com/en-us/library/79b3xss3.aspx

[conanmishler]

In C#, only ONE instance of a static variable may exist. That is almost the entire and only reason to ever declare a static variable. There will be no copies of a static variable. To quote from one of the many M$ articles on the subject: "Only one copy of a static member exists, regardless of how many instances of the class are created. Static methods and properties cannot access non-static fields and events in their containing type, and they cannot access an instance variable of any object unless it is explicitly passed in a method parameter." (emphasis mine).

ref: https://msdn.microsoft.com/en-us/library/79b3xss3.aspx

Thanks for the reply. However, I really don't think static variables in one process, or thread, are accessible by another thread. My understanding of multi-threaded systems is that each thread operates in a completely separate area of memory from the other threads. The answers in the following post support this understanding:

http://stackoverflow.com/questions/5880723/question-on-multiprocessing-and-static-variables-in-c

The questioner in this post is asking about communicating between processes in C, but I would guess the answers apply as much to C#.

[koganam]

Thanks for the reply. However, I really don't think static variables in one process, or thread, are accessible by another thread. My understanding of multi-threaded systems is that each thread operates in a completely separate area of memory from the other threads. The answers in the following post support this understanding:

http://stackoverflow.com/questions/5880723/question-on-multiprocessing-and-static-variables-in-c

The questioner in this post is asking about communicating between processes in C, but I would guess the answers apply as much to C#.

C is not C#. It has nothing to do with threading. It has to do with the class.

[conanmishler]

Thank you, Koganam!

I did some more reading and found that C#, unlike C, does in indeed share static variables across threads.

I eliminated all of the static variables from my strategy (except "static readonly" variables) and now I no longer get the "Object reference not set to an instance of an object" error when I optimize on a multi-processor computer.

Thanks again!

Conan