How to wait till last bar on Multitimeframe Strategy

[NeoAkira]

Hello JC,

just to clarify if you maybe misunderstood what i am asking. The DataSeries in the primary chart is fine and seems to work perfectly with my local History Data.
Only the Periods I add with Add(PeriodType.Day, xxx); are showing much lesser Data in my results.

Thanks again for your efforts

[NeoAkira]

Hi JC,
Here is another Indicator that your statement can't be right and that local historical Data will be used for the Add(Period) Function. I just found it via Visual Studio Debugging mode when I inspected my BarsArray (this Values stay also the same till I reach my last Bar)

BarsArray[0] = {instrument='$EURUSD' from='1990-01-28' to='2014-04-09' period=Daily - Bid splitAdjusted=False dividendAdjusted=False bars=5211}
BarsArray[11] ={instrument='$EURUSD' from='1990-01-28' to='2014-04-09' period=Daily splitAdjusted=False dividendAdjusted=False bars=3403}

BarsArray[0] is the Data on my current chart (Daily) starting from 1993 and has 5211 Bars (this is from my local historical DB)
BarsArray[11] is Data added via "Add(PeriodType.Day, 1);" and has also Daily Data but has only 3403 Bars. This must be from the DataProvider DB.

This has nothing to do with my logic or anything else you can check it for yourself.
There are objectively lesser Bars in the Bars Array added via Add.Period then in the original DataSeries.

Ok. but this doesn't help me any further...my question is if there is any possibility to change this behavior and enable Add(Period) to use my local history DB.
Why would anybody program something like this. You should always look first in your local history DB and if the Data is not there then get the Data from your Provider(s).

[NinjaTrader_JC]

Hello NeoAkira,

Let me write up a sample script to better demonstrate what I am talking about. I think this would be a good medium to show what is expected to happen and to get on the same page.

[NeoAkira]

Hello JC,

Let me write up a sample script to better demonstrate what I am talking about. I think this would be a good medium to show what is expected to happen and to get on the same page.

I'm really interested to see your script or what you want to prove.

Now something for your information regarding the above topic.
Since my results from the calculated Values for my Indicators from the Periods that I got via the Add(Period) Function and the Values on my chart were way off i deleted all my local history Data (especially the Dailys I'm working with) and loaded everything via the Historical Data Manager/Download Function inside NT from the FXCM Servers since they seem to offer more Data (starting from 2002) then the NT-Servers.

The Result now is that my local historical DB is the same as the Data from my Data Provider. And now my Indicator Values (calculated in my script and seen on my Chart) are also in sync.

The bad side of this Solution is that any Period that is loaded via Add(Period) seems to download or sync its Data somehow from/with the Data Provider and it is useless to import any local history Data that has a longer timespan (even if it is from the same source) if you want to do MultiTimeFrame backtesting and using the Add(Period) Function unless you are only working with the limited Data that is provided only from the Servers of your Data Provider.

I'm still eager to see your solution to this problem and hope that your script will show me how to do it right.

Thanks in advance

[NinjaTrader_JC]

Hello NeoAkira,

Thanks for your patience.

So here is a sample script that may help see what is going on.

This will show in the output window if you disable the added DataSeries and add them how NinjaTrader is going to process the OnBarUpdate. So with BarsRequired = 0 it will wait until 150 daily bars a built. With BarsRequired = 1 is 300.

[NeoAkira]

Hallo JC,

thanks for your effort and your script. But your script proves exactly my point...
I try to describe it as best as possible.

My initial situation:

• I have history data downloaded from FXCM that spans the Date from 1993 till today.
• I imported this data into my history Database according to their documentation.
• I can see that the history Data is in my Historical Data Manager on my local installation of NT.
• I open a new Chart with Period Type ( Bid/Day/1) and select a custom Range from 1990 till today.
• The Chart shows all Bars correctly from 1993 till today.
• I add your Script to the Chart with right click on the Chart and then click on Strategies
• When I run your script with AddDataSeries = False I get following outputs.

Primary Chart is 1Day

Primary Series
Time: 11.05.1993 23:00:00
Count: 5248
CurrentBar: 0
You didnt add any Daily Data Series

Primary Chart is 5Day

Primary Series
Time: 18.05.1993 23:00:00
Count: 1050
CurrentBar: 0
You didnt add any Daily Data Series

Primary Chart is 50Day

Primary Series
Time: 09.08.1993 23:00:00
Count: 105
CurrentBar: 0
You didnt add any Daily Data Series

Primary Chart is 150Day

Primary Series
Time: 08.02.1994 23:00:00
Count: 35
CurrentBar: 0
You didnt add any Daily Data Series

I added following lines to your script:
In Initialize()

        protected override void Initialize()
        {
            if(AddDataSeries)
            {
                //BarsInProgress = 1
                Add(PeriodType.Day,150);
                //BarsInProgress = 2
                Add(PeriodType.Day,50);
                //BarsInProgress = 3
                Add(PeriodType.Day,5);
[B]                //BarsInProgress = 4
                Add(PeriodType.Day,1);
[/B]
            }

And in OnBarUpdate() at the End of your code

            // Because there is 150, 1 Day bars in a 150 Day chart
            if(BarsInProgress == 4 && CurrentBars[4] == 150)
            {
                Print("1 Day Series");
                Print ("Time: "+Time[0]);
                Print("Count: "+Count);
                Print("CurrentBar: "+CurrentBar);
                Print("");
            }

I now restarted your Script with AddDataSeries = True
And I get this result

150 Day Series
Time: 30.04.2002 23:00:00
Count: 24
CurrentBar: 0
Since this is the largest series it has to wait unit it has data before processing a Strategy.

1 Day Series
Time: 01.05.2002 23:00:00
Count: 3521
CurrentBar: 150

5 Day Series
Time: 06.05.2002 23:00:00
Count: 705
CurrentBar: 30

50 Day Series
Time: 02.07.2002 23:00:00
Count: 71
CurrentBar: 3

I put the Time and the Bar Count in Bold so that you see the difference explicitly.

Every Data Series that is put via Add(PeriodType) in the Script is not using my local history Data. This is Data that is loaded from the Servers of the Data Provider I'm connected to when I started the Strategy.
How else could the discrepancy in Time and Bar Count occur?

And this is my Problem. I can not make Back-tests with the Data and time range I downloaded. Also I can not properly check and compare my collected Indicator Values from the script since the start Dates of the Real Chart I'm looking at and the Values that are calculated in my Script having different starting times and therefor different Bars and I also assume a different Session Template.

The only workaround I found was to delete all my local history Data and load the Data from my Data Provider I'm connecting to.
If I do this then the bar counts and Time is in Sync and my calculated indicator values are also in sync with my local chart.
The problem herin is only that my Data Provider is not providing me enough Data on their normal Servers to make the Tests I want.

I hope I could describe my problem better this time.

This behavior of the Add(PeriodType) function makes any Baktesting with local history Data impossible and therefor any Multi Time Frame testing impossible.

I want to know if there is any possibility to use local history Data with the Add(PeriodType) function.

Thanks again for all your effort and I'm still hoping that we can find a solution for this problem.

[NeoAkira]

Hi JC,

me again...

I just wanted to show you how the values from your script are when i use only historical Data from my DataProvider. This means my local history Database has only been updated via the Historical Data Manager/Download Function. No import from separate downloaded Data Files.
Everything else is exactly the same as in my post before

Again first the Data with AddDataSeries=False
I change the primary Data Series inside the Chart with the Keyboard (1D, 5D, 50D,150D)

Primary Series = 1D

Primary Series
Time: 22.10.2001 23:00:00
Count: 3523
CurrentBar: 0
You didnt add any Daily Data Series

Primary Series = 5D

Primary Series
Time: 26.10.2001 23:00:00
Count: 705
CurrentBar: 0
You didnt add any Daily Data Series

Primary Series = 50D

Primary Series
Time: 27.12.2001 23:00:00
Count: 71
CurrentBar: 0
You didnt add any Daily Data Series

Primary Series = 150D

Primary Series
Time: 30.04.2002 23:00:00
Count: 24
CurrentBar: 0
You didnt add any Daily Data Series

And now the Output with with AddDataSeries=True

150 Day Series
Time: 30.04.2002 23:00:00
Count: 24
CurrentBar: 0
Since this is the largest series it has to wait unit it has data before processing a Strategy.

Primary Series
Time: 01.05.2002 23:00:00
Count: 3523
CurrentBar: 150

1 Day Series
Time: 01.05.2002 23:00:00
Count: 3523
CurrentBar: 150

5 Day Series
Time: 06.05.2002 23:00:00
Count: 705
CurrentBar: 30

50 Day Series
Time: 02.07.2002 23:00:00
Count: 71
CurrentBar: 3

As you can see the Bar count is completely in sync. The Timestamps are a little off but this is not the problem and it is only a couple of months and related to the BarsInProgress & CurrentBars[] Logic.

I just wanted to show you that the missing bars and Data has nothing to do with the BarsInProgress & CurrentBars[] Logic.

Still hoping to hear from you and that you/we can find a solution for this problem.

[NeoAkira]

Hi JC,
just wanted to bump this and ask if you have something new on this topic.

Thanks in advance

[NeoAkira]

Hi JC and/or any other Support Member

I am really getting a little impatient here ( I don't want to use the word angry).

It took me almost three weeks to prove to you that your Assumption that the Add(Period) Function is using local historical Data has been wrong and that instead it is using historical Data it is downloading from the Data Provider (every time the function is called).

Now the communication is suddenly canceled from your side and I hear nothing from you. Not even if you are working on a fix or something else.

My client and I need a solution for this problem. Why is there a local historical DB if it is not used by your own functions? Is there a setting I can use to switch this behavior?

Please be so kind and answer my questions.

Thanks in Advance.

[NinjaTrader_JC]

Hello NeoAkira,

Sorry for the delay.

Lets try to clear up one thing. If you are not connected to any data provider and you import your Historical Data into NinjaTrader from FXCM do you see the same results as you are now?

From all of my tests I have not seen this behavior. NinjaTrader will overwrite Historical Data if you have NinjaTrader automatically set to "Get data from server (if available)" under Tools -> Options -> Data tab.

Have you tried using the Kinetick - End of Day (Free) for Daily as they should have at least 10 years of Historical Data and testing it out on my end for the $EURUSD and $EURJPY they have data back until 1993?

Also, if your primary series is set to 1 day could you clarify why you are adding it again?

Happy to be of further assistance.

[NeoAkira]

Hi JC,

Lets try to clear up one thing. If you are not connected to any data provider and you import your Historical Data into NinjaTrader from FXCM do you see the same results as you are now?

If I'm not connected to any Provider I can not select and start a Strategy therefore I can not run the script on the chart (also a really odd behaviour).
I also tried to connect to the simulated feed but it will use the same data range as my last real connection. If my last connection was to FXCM Servers I have data from 2002 till today. If I was last connected to NT-Servers I have only Data from 2008 till today.

From all of my tests I have not seen this behaviour. NinjaTrader will overwrite Historical Data if you have NinjaTrader automatically set to "Get data from server (if available)" under Tools -> Options -> Data tab.

As I wrote in a previous post I already tried this option with no difference in the results and it will only overwrite historical Data if it is not already available in your local DB or you start a reload manually.

Have you tried using the Kinetick - End of Day (Free) for Daily as they should have at least 10 years of Historical Data and testing it out on my end for the $EURUSD and $EURJPY they have data back until 1993?

This is really not a solution because I will also need minute Data at a later phase of this project. If you have the data available just download and import the data from kinetic and then connect to another provider who has a smaller data range on their servers and then run your script from there. You will see the same results as me.
If you always use the same server and the same Data and Data Range it is only logical that you can not recreate this problem because it will not happen.

Also, if your primary series is set to 1 day could you clarify why you are adding it again?

It was only to show you that if I load the same time series as the Primary Series via Add(Period) that there is a difference in the BarCount if your local history DB is larger then the history that is used in Add(Period) from your Data Provider.
On the other Hand I never know in what Time Series the Strategy will be loaded, To have a consistent Result I load all Data Series I need via Add(Period) and run all my calculations only on this Periods.

Happy to be of further assistance.

JC, I really appreciate your support. I really think that I proved with all the previous Posts and the Data I have provided that the Add(Period) Function is definitely not using the local History DB.
It seems to using the local history DB because most People have the same Data in their local DB as the Data Provider they are using. But when you have imported your Data from a different Data source or a wider time range you will see that this is not the case and that this Function is using the Data from your Provider.

What more prove do you need?
It is not working for me...It is not working with your script and it is not working with my script. It only works when I use the limited Data from the Servers of my Data Provider.

I need a solution for this. If you can't find a solution for this than please ask your colleagues or your development team or whoever might know more about this topic.

We are talking about this problem for almost three weeks and nothing is going forward. To be able to recreate this problem you should at least recreate the situation i have described previously. you need to import a data range in your local history DB that is larger than the data your Data Provider is delivering from his servers. Than you will be able to recreate this problem.

Thanks in advance.

[NinjaTrader_JC]

Hello NeoAkira,

So what I believe the difference you are seeing is when you select the "Reload All Historical Data" from inside of the Chart, which is forcing NinjaTrader to match the Historical Data on your Local PC that is changing your Output statements.

To see this you do not have to run a script just import your Historical Data into NinjaTrader, open up a chart and you should see your data back to 1993 as you expect. Now connect to FXCM, even if you have "Get data from server (if available)" checked you should see the chart go back to 1993, but if you right click inside of the Chart and select "Reload All Historical Data" you will see the Chart only load back to 2002 or whatever date the server is going back to.

Now to get around this there are only a few ways.

1. You may go to the FXCM Account connection and disable the Historical Data option

2. Connecting to the Kinetick - End of Day (Free) connection first and then Connect to FXCM will request all of the Historical Daily Data from Kinetick and then all of the Intraday data from FXCM.

3. You may instead right click inside of the Chart and select "Reload NinjaScript" to not overwrite the Historical Data that you have imported.

I believe option 2 may be the best way to go but let me know if it works for you.

[NeoAkira]

Hello JC,

So what I believe the difference you are seeing is when you select the "Reload All Historical Data" from inside of the Chart, which is forcing NinjaTrader to match the Historical Data on your Local PC that is changing your Output statements.

I'm not a native English Speaker so maybe I have misspelled or miswritten something somewhere.

How on Earth do you come to the assumption that I did a manual reload of the historical Data? I did this error once and it replaced all my history Data with Data downloaded from the Servers of my Data Provider (FXCM). I needed to reimport the Data I downloaded with the History Download Tool from FXCM. I wrote multiple times that my Data in my local history DB is correct and I have the Data available from 1993 till today.

The Print Statements from your Script that I posted in http://www.ninjatrader.com/support/f...2&postcount=21 also clearly shows that the time range on the primary chart is from 1993 till the day I ran your script (this is the Data that is in my local history DB).
Additionally the Print Statements from your Script where I used The Add.(Period) Function inside the same Chart show clearly that it starts in 2002 and not in 1993 as it should. You also see that there is a significant difference in the BarCount between the Output from the primary Chart and the output of the Values that are generated with the Add(Period) Function. This alone is prove that Add(Period) is not using the local history Database and instead is always loading the Data fresh from my connected Data Provider.

In the Post http://www.ninjatrader.com/support/f...2&postcount=22 I cleared all my local history DB and downloaded all the Data fresh from my Data Provider I'm connected to with the Historical Data Manager from NT. In this Post you see that the Time Range and the Bar Count is in sync (the same) between the primary Chart (which is Data from my local history DB) and the Values that where printed with the Add(Period) Function.
That should be another proving factor that the Add(Period) Function is not working with Data in my local history DB and is using the Data it downloads from the Data Provider because my local history DB is in this case the same as the Data Provider I'm connected to. This also proves that it has nothing to do with BarsInProgress or anything else. Only the time is a little of (a couple of months) but this is OK and also logical sound.

To see this you do not have to run a script just import your Historical Data into NinjaTrader, open up a chart and you should see your data back to 1993 as you expect. Now connect to FXCM, even if you have "Get data from server (if available)" checked you should see the chart go back to 1993, but if you right click inside of the Chart and select "Reload All Historical Data" you will see the Chart only load back to 2002 or whatever date the server is going back to.

I know of this odd Behavior of the "Reload All Historical Data" Feature therefore I only used it once when I started with NT and never touched it again. This "Feature" does delete the data in the local history DB for the instrument of the current Chart and reloads all the Data fresh from the Data Provider I'm connected to. This "Feature" should only be available in the "Historical Data Manager" because that is what a Data Manager does and not screwing around your local history DB in the Chart. When I want to reload the historical Data on the Chart it should do this from my local history DB and not from the Data Provider. This should be logical. Why on earth do you have a local history DB when the Functions grab their Data from the Data Provider every time?
As I wrote multiple Times my local history DB is NOT the Problem. The behavior of the Add(Period) Function is my Problem.
I don't know if I talk Swahili Language or why you don't understand my Problem and come up with Problems I don't have.

Now to get around this there are only a few ways.

1. You may go to the FXCM Account connection and disable the Historical Data option

Not really an Option since we will need the Data from the Broker we will trade with when the algorithms and the strategy are finished (Algorithm and Strategy need to be able to have the Data available in local history DB).
Also it was a prerequisite for us to have a broker that can provide us with 20 Years of Data and a Platform to be able to make proper Backtesting and walk Forward testing with this Datasets.

2. Connecting to the Kinetick - End of Day (Free) connection first and then Connect to FXCM will request all of the Historical Daily Data from Kinetick and then all of the Intraday data from FXCM.

Not an Option since we will also need Minute Data at some Point that is not provided via Kinetic Free.

3. You may instead right click inside of the Chart and select "Reload NinjaScript" to not overwrite the Historical Data that you have imported.

I know this and I use this function some times but again you are talking about a complete different Problem. I don't have a Problem with my local history DB I have a Problem with the Add(Period) Function.

I believe option 2 may be the best way to go but let me know if it works for you.

NO, It does not work for me. What will work for me is when the Add(Period) Function is using the local history DB to get its Data.

Thanks in advance
and please try to stay on topic that is the Add(Period) function or involve one of your colleagues that knows something about this topic.

[NeoAkira]

Solution for Add(Period) misbehavior

Hi JC,

I think I finally found a solution for the Add(Period) misbehavior.

It seems that when you use only

Add(PeriodType.Day, 50)

it is not working with the local history DB and instead is loading the Data or the Date Range that is available for the Script from your Data Provider you're connected to.

When I use

Add(Instrument.FullName, PeriodType.Day, 50, MarketDataType.Bid);

It is using my local DB and the Date Range i selected on the Chart and I can finally see/work with my local Data.

I rewrote your Script in Initialize and added it as an attachment:

            if(AddDataSeries)
            {
                //BarsInProgress = 1
                Add(Instrument.FullName, PeriodType.Day, 150, MarketDataType.Bid);
                //BarsInProgress = 2
                Add(Instrument.FullName, PeriodType.Day, 50, MarketDataType.Bid);
                //BarsInProgress = 3
                Add(Instrument.FullName, PeriodType.Day, 5, MarketDataType.Bid);
                //BarsInProgress = 4
                Add(Instrument.FullName, PeriodType.Day, 1, MarketDataType.Bid);

            }

And now my Output looks like this:

AddDataSeries=false
Primary Series
Time: 11.05.1993 23:00:00
Count: 5256
CurrentBar: 0
You didnt add any Daily Data Series

AddDataSeries=true
150 Day Series
Time: 08.02.1994 23:00:00
Count: 36
CurrentBar: 0
Since this is the largest series it has to wait unit it has data before processing a Strategy.

Primary Series
Time: 09.02.1994 23:00:00
Count: 5256
CurrentBar: 150

1 Day Series
Time: 09.02.1994 23:00:00
Count: 5256
CurrentBar: 150

5 Day Series
Time: 16.02.1994 23:00:00
Count: 1052
CurrentBar: 30

50 Day Series
Time: 09.05.1994 23:00:00
Count: 106
CurrentBar: 3

This is how it should have looked like in the first place.
The BarCount is in Sync with my primary Series and the DateTime is only a couple of months off which is OK with me and a logical behavior regarding how BarsInProgress and CurrentBars works.

I just wanted to inform you and other Users about this Solution.
In hope that if this Problem effects somebody else in the Future he doesn't need to talk 3 weeks about loading/reloading/deleting/importing history Data and instead uses the longer or more explicit Version of Add(PeriodType).

Thanks again for all your efforts JC but next time try to read and understand the Problem first and don't try to assume something that nobody has written/talked anything about. I worked also in 2nd/3rd Level Support for sometime and know it is not always easy to communicate with customers who have no clue what they are doing.

But when somebody shows you in an analytical sound way that something is not working as it should then you should also listen to him/her. And the first thing to find a solution to a Problem is trying to recreate this Problem what you actually never did and thats why you always came up with solutions that had nothing to do with the real Problem regardless how many times I wrote that the Add(Period) Function is my main Problem.

Thanks again

NeoAkira