NT and data snooping

[entropy]

Hello NT support:

I'm dealing with a very perplexing issue--I'm doing simulation testing with the free trial of zenfire. My strategy and all indicators involved are set CalculateOnBarClose = true, so very confused as to why this might happen:

When I run my strategy on live data for a while and then refresh, the indicator paints differently over the same price data. The data looks like it's coming in from zenfire corrupted and then fixes when I reload the data that just was saved in real time.

Is it at all possible that NT is looking forward during the backtest or indicator painting over the historical data? I thought this was impossible. Is data snooping possible with NT? If so how would I do it/avoid it?

If I can get this figured out and get my indis to paint live like they do over historical data, I'll be buying NT for sure, please help, thanks!

[entropy]

By the way, my strategy accesses timeframes smaller than the current chart for example the 5 minute from the 30, could this corrupt anything? Thanks!

[NinjaTrader_Matthew]

Hello entropy,

You're likely experiencing issues with discrepancies between real-time and historical price data.

From our Help Guide article - http://www.ninjatrader.com/support/h...are_built.htm:

As ticks come into NinjaTrader in real-time, they are time stamped based on your local PC time if they do not already have an associated time stamp that is provided from the real-time data source. The majority of our supported brokerage feeds DO NOT time stamp ticks, where most of our supported market data vendor feeds do provide time stamped ticks. NinjaTrader then builds bars based on the time stamp of the incoming tick and displays these bars in your chart in real-time.

Let's say you have a tick (tick "A") with a time stamp of 10:31:00 AM which gets packaged into the 10:32:00 AM bar and happens to be the high of that bar. An hour later, you reload historical data from your historical data provider into NinjaTrader. This process will overwrite the existing data. The 10:32:00 AM bar now looks different since the high made by TICK "A" is now part of the prior bar, 10:31:00 AM. How is this possible?

•Your PC clock could have been off so the time stamp is delayed

•Your internet may have been lagging so the tick came in slightly delayed and therefore the time stamp is delayed

•Due to standard latency, even 50ms delay (which is normal) could be the difference between a 10:30:59 and 10:31:00 time stamp

•There is no way of knowing how the historical data provider packages their bars

The only way to ensure that data always looks the same is if every connectivity provider sent ticks with time stamps AND that all vendors synchronized on time stamps. Unfortunately, this is just not a reality nor plausible scenario.

For your issues, I would suggest ensuring your local PC clock is in sync with a windows time server.

You can sync your PC clock by double clicking on the clock in the lower right corner of your desktop. Once you have done that, click on Internet Time tab and then click Update. Your PC clock should now be updated.


Since Zen-Fire is naively time stamped on their servers, you can check for discrepancies in real-time by opening up a Time and Sales window (File--> New) and then comparing the time stamps you see on the T&S from Zen-Fire, can comparing your local PC clock. You will want to check this down to the second.

[cirion]

Hi,

I am working with entropy on this strategy and as I know he is asleep I thought I would add a part of the issue visually.

When you run just a chart nothing on it the candles are fine (normal).

When we run our strategy the candles start to break.and are fixed if you refresh the chart. (then start breaking again)

for nt - broken.png is an example of how the candle starts to look and over time all the candles do this.

for nt - normal.png is what the candle looks like without the strategy.

Thanks

CIrion

[NinjaTrader_Matthew]

Hello cirion,

Thank you for your additional information.

Do you see errors from the output window or from the Log tab of the Control Center around this time?

[cirion]

Thanks for your fast reply

No errors in log.

In your strategy we pull in other time frames would this effect the candles on the chart?

Can you think of anything we should try or add to help us debug?

Part of my thinks it is just bad data but it is hard to point the finger with out more debugging

Cheers

[NinjaTrader_Matthew]

Would it be possible to remove the additional data series to isolate it to a single series? This would depend on how it is coded, but simplifying the code usually speeds up this process.

Also you can check the integrity of the data that is being used by going to Tools--> Historical Data Manager-- > Edit tab.

You should check the values of data when currently experiencing issues and the values once the issue has resolved. If these are changing, it is likely the core data. If this data stays the same, it would likely be an issue in the custom code.

[cirion]

Thanks for your help we solved the issue

was to do with how we were pulling in other timeframes and what array the time frame was in

Cheers