Charts and Analyzer unreal disconnection issue from Database

[pstrusi]

NT support.

Once again happens this issue: The internal working data for Charts and Strategy Analyzer is "UNSYNCED" with the real and original registered in the database. This occurs for the EURUSD.

The real-time data is properly registered in the database as seen in charts, BUT eventually, as soon I closed NT and restart it again, then the inexplicable alterations at once ( same time ) appears in charts and Strategy Analyzer. This kind of sudden and big price variations that are NOT REAL , affects totally my Analysis Work, cause I got different and unreliable results.
I've checked the price action registered in the database and it's different than shown on charts and Strategy Analizer's tests.
CHARTS AND STRATEGY ANALYZER ARE BUILDING ON UNREAL DATA, BECAUSE IT'S NOT WHAT THERE'S IN THE DATABASE.
Please consider to check the issue price action in charts and the extract of the database attached in two files to this post.

This is happening for quite some time, as you can check in this forum's several post, but still I continue to suffer it with not solutions. I don't know what to do in order to use the real info registered in the database.

How could I reset the sync between the original and genuine database and the working historical data within the folders to be used by Charts and Strategy Analyzer?

THOUGHTS
1. I'm watching the internal database folders in order to figure out which one are the original database and which is actually the working historical data for charts and analyzer. I won't touch anything until you advice me how to address this. The two folders lines:

- Documents/NinjaTrader7/db/tick/$EURUSD: here there are bid,ask and last .ntd files
- Documents/NinjaTrader7/db/data/20160411 : here there is $EURUSD.ntm file

2.I've tried to use the Download historical manager searching "reset" the historical data but not results. If I Exported and Imported the database could be reset it?

Looking forward for your help

[NinjaTrader_ChelseaB]

Hello pstrusi,

Thanks for your note.

I'd like to reproduce on my end to take a closer look.

Are you saving your live data as historical data or is this downloaded from the historical data servers when your chart is refreshed or when the connection is reconnected?

To check this, click Tools -> Options... -> Data.
Is 'Save chart data as historical' checked?

Also, what is 'Quote currencies (FX) in:' set to in this window?

Who are you connected to?
(This is indicated in green in the lower left corner of the Control Center)

What time zone are you in?

On the $EURUSD instrument what is the Tick Size and Point Value set to?
(Tools -> Instrument Manager... -> search EURUSD and look at the results Tick Size and Point Val columns)

Can this be reproduced at any time by disconnecting and reconnecting or restarting?

[pstrusi]

Hi Chelsea, thank a lot for assistance. I'll try to answer your questions for your initial approach to this issue.

Are you saving your live data as historical data or is this downloaded from the historical data servers when your chart is refreshed or when the connection is reconnected?

Yes, I always save my live chart data as historical, cause I work with IB. And of course 'Save chart data as historical' is permanently checked.

Also, what is 'Quote currencies (FX) in:' set to in this window?

Always in HalfPips

Who are you connected to?
(This is indicated in green in the lower left corner of the Control Center)

Interactivebrokers. That's why I save my live data as historical.

What time zone are you in?

My whole platform is set to US E.T

On the $EURUSD instrument what is the Tick Size and Point Value set to?
(Tools -> Instrument Manager... -> search EURUSD and look at the results Tick Size and Point Val columns)

ticksize------> 0.00005
Pointvalue--> 100.000

Can this be reproduced at any time by disconnecting and reconnecting or restarting?

No. This issue happens "randomly" after closing NT and restart it again. typically after weekends.

Chelsea, my NT configuration is been set correctly the whole time as you can see. So this must be other unknown reason. Whatever it is causing, my priority to finish my backtests is RESETING or SYNCING the historical working data used by charts and Analyzer with the correct internal database.

As you might see in the two files attached to this post, the database data is different to the chart data shown.

Thanks for your help and I'll be waiting for your practical suggestions in how to perform a reload of my historical data from the internal database itself which is right fortunately.

[NinjaTrader_ChelseaB]

Hi pstrusi,

The data in the file ends at 10:05:00 while the chart begins at 10:08:02. Are you wanting me to compare the data in the exported file with the data on the chart?

If so, may I have an export of the data from 10:07 to 10:09?

[pstrusi]

Chelsea, actually the database extract file is backwards, the most recent is upper. You could go up in the data and searching around 10:08:12 to 10:08:15. I've download once again the two files here and they are correct. Please let me know any further doubts

[NinjaTrader_ChelseaB]

Hi pstrusi,

That was my mistake.

I am attempting to reproduce this behavior as we speak. I will let you know if I am able to reproduce using these settings.

[pstrusi]

Thanks Chelsea.

Have in mind that this issue might not rise up before you right now, As I've told you before I could work happily for two months without any issue. After you do your research, one of the most important aspects in your help to me, is actually how to sync again the Historical working database with the original database which is untouched. That's why I put certain suggestions or thoughts in the original post. Please tell me what you think about them.

Thanks in advance

[NinjaTrader_ChelseaB]

Hi pstrusi,

I'd like to you to clear the chart cache and see if this fixes the chart. (Don't delete the saved historical data though)

Please use the steps I have provided below to delete your chart cache and historical data.

• Close NinjaTrader
• Open (My) Documents\NinjaTrader 7\db\cache\
• Delete all files within this folder
• Restart NinjaTrader
• Connect

Open a new chart without any indicators or templates applied to see if you are able to load a chart.

If this does not fix the chart, please send an email to platformsupport [at] ninjatrader [dot] com so that we may schedule a call and I can assist.

[pstrusi]

Thanks for your suggestion about clearing chart cache.

Despite I did this very same procedure on weekend ( which it didn't solve the issue ), I will do it again today after market close, so I don't lose any current live data. I'll let you know the result.

I have not problem to schedule a call but I should say that my spoken english is not fluid enough to do an effective technical solution.

Please allow me to repeat myself in this point: Could you consult to developers team, how to "reset" or "sync" my historical working data with my internal database which is on good state?. In my original post, at final you could find a couple of thoughts hopefully they might help to performance this procedure.

Thanks

[NinjaTrader_ChelseaB]

Hello pstrusi,

Historical data is saved in individual files and not saved in the NinjaTrader.sdf database file.

This historical data is located at:
Documents\NinjaTrader 7\db\ in the folders tick, minute, and day.
This is the same data that appears in the Tools -> Historical Data Manager... window, as it is read from these files.

When starting NinjaTrader the historical data will be read from these files to build your chart. If the chart has already been built once without being modified, this will exist in the chart cache and will load from the chart cache.

The chart should be loading what is in the Historical Data Manager.

From your initial post, you mentioned that the data in the Historical Data Manager is the correct information, is this correct?

If so, I am having you clear the cache so that the chart is rebuilt from the historical data in Documents\NinjaTrader 7\db\tick folder.

Are you asking how to reload the historical data that appears in the Historical Data Manager?

[pstrusi]

Hello pstrusi,

Historical data is saved in individual files and not saved in the NinjaTrader.sdf database file.

This historical data is located at:
Documents\NinjaTrader 7\db\ in the folders tick, minute, and day.
This is the same data that appears in the Tools -> Historical Data Manager... window, as it is read from these files.

When starting NinjaTrader the historical data will be read from these files to build your chart. If the chart has already been built once without being modified, this will exist in the chart cache and will load from the chart cache.

The chart should be loading what is in the Historical Data Manager.

From your initial post, you mentioned that the data in the Historical Data Manager is the correct information, is this correct?

If so, I am having you clear the cache so that the chart is rebuilt from the historical data in Documents\NinjaTrader 7\db\tick folder.

Are you asking how to reload the historical data that appears in the Historical Data Manager?

Hi Chelsea, I appreciate your close assistance, hopefully this will be addressed soon.

Yes, the database extract that actually you saw here was taken from the "Historical data manager" which is what I refer to as the right internal database. My initially thought was that there was a different set of cache data to the original historical, and perhaps this was corrupted somehow.

Since the historical data is ok. there won't be necessary to reload it. Having say this, I wonder Why this right historical data was not read correctly by charts or even more critical: Strategy Analyzer, after I performed the Cache Clear procedure?

I'll repeat it today after market close, and I'll report it to this post right away.

Thanks

[koganam]

Hi Chelsea, I appreciate your close assistance, hopefully this will be addressed soon.

Yes, the database extract that actually you saw here was taken from the "Historical data manager" which is what I refer to as the right internal database. My initially thought was that there was a different set of cache data to the original historical, and perhaps this was corrupted somehow.

Since the historical data is ok. there won't be necessary to reload it. Having say this, I wonder Why this right historical data was not read correctly by charts or even more critical: Strategy Analyzer, after I performed the Cache Clear procedure?

I'll repeat it today after market close, and I'll report it to this post right away.

Thanks

As long as your live data is snapshot data instead of raw data, your historical data will NEVER match your live data after the data is reloaded, so anything that uses historical data will differ from what happened live. IB sends snapshot data.

[pstrusi]

As long as your live data is snapshot data instead of raw data, your historical data will NEVER match your live data after the data is reloaded, so anything that uses historical data will differ from what happened live. IB sends snapshot data.

Hi Koganam.

Since my data provider is my very same broker ( Interactive brokers ) I have to save the live data that I receive from them to my historical database, and I think that regardless the type of live data whether is snapshot or raw data, I should be able to reload from historical as seen previously as live data from IB.

Thanks always for your thoughts, suggestions and help. This issue is the more difficult that I deal with in NT.

[pstrusi]

Chelsea, I must ask you:

What is the difference between the "historical data saved in individual files" and the NinjaTrader.sdf file? I thought that the .sdf file had all the basic tick data.

I'd like to know this in order to think logically more angles to solve this error.

[NinjaTrader_ChelseaB]

Hi pstrusi,

The NinjaTrader.sdf file is the SQL database file.
This contains ATM Strategy templates, Session templates, Instrument Lists / Custom Instruments, and trade execution data.

This file is meant to be very small.

Historical data is not kept in an SQL database and is instead kept in historical data files designed to hold historical data.
These files while small themselves, add up to a very large amount of files and data.