Corrupt Minute Bars

[aslane]

The last couple of weeks I have been noticing corrupt bars during the trading day. Everything seems to be going along fine, and then all of a sudden when a new bar starts the high/low values are corrupt and do not recover. This typically happens for a single bar or two, and then subsequent bars are fine.

The first attachment shows what I mean. You can see the open and close ticks and the high/low portion of the bar are separate, and the data box shows the bad values.

If I simply change the interval and change it back, then the bars are correct. The second attachment shows what it looks like after going 15M -> 5M -> 15M.

I have spent some time narrowing it down, and this happens on charts with no indicator/strat, just a straight 5-15 minute data series. I have seen on multiple symbols (ES, NQ, CL, 6A). The same data series shows up fine in a different chart, so it is just a single chart data series that is affected, not the underlying data. This happens once or twice a day.

I believe this happens when I am connected to both Continuum and IB (data is sourced from Continuum). When I am only connected to Continuum I do not see the issue. I have also only seen when multiple large workspaces are open.

There is nothing in the logs, so please do not start with a stock support request. This is an internal NT corruption issue.

[NinjaTrader_RyanM]

Hello aslane,
Can you please clarify if reloading the historical data initial resolves this item as well? To reload historical data please right click on your chart > "Reload all historical data".

[aslane]

Yes, if I reload data it also resolves the issue, but as I said that is not required (just switching intervals fixes it). So, I do not think it is a bad tick flowing in (also other charts with same symbol and same interval are just fine).

When I observe it happen, it literally happens when the new bar is started. Seems like bad initial state is loaded into bar.

[NinjaTrader_RyanM]

Hello aslane,
This item is currently being reviewed by our Product Management team.
Please send me your log and trace files so that I may look into what occurred to platformsupport[at]ninjatrader[dot]com.

- Open your NinjaTrader folder under My Documents.
- Right click on the 'log' and 'trace' folders and select Send To> Compressed (zipped) Folder.
- Send the 2 compressed folders as attachments to this email.
- Once complete, you can delete these compressed folders.

Within the body of the email please add:
ATTN RyanM
https://ninjatrader.com/support/foru...d.php?t=108266

[aslane]

I am posting back here because this problem is NOT resolved, and Ninja has said they wont be fixing, so others should have the info.

Problem: when using standard minute bars and standard chart styles, you will occasionally get bad bars displayed with nonsensical high/low values.With candlesticks, this results in long tails where the symbol never traded (open/close are typically outside the high/low range). Example is attached. This occurs when a new bar is first formed. The issue is the high/low values used to render are corrupt, and these bad values are also displayed in the data window. The actual underlying bar data delivered to indicators is correct though and has been verified via an indicator. This problem happens maybe once a day.

Quick Fix: Hitting the Home key followed by the End key, will resolve the bad cached data and the bars are rendered correctly (see second image).

I spent a long time trying to work with support to address this issue, and Ryan did a good job. The NT devs though seem incapable to figure out without a simple scenario that reproduces (typical). They zeroed in on it potentially being a custom chart style issue, but I have disproved that as well by running with standard chart styles.

This has been reproduced with:

* standard min bars
* standard candlestick chart style
* low cpu util
* low gpu util
* plenty of memory
* moderate workspace with 7 windows, 12 total tabs, 4 total symbols, min num of indicators
* single data connection to Continuum
* happens when feed utilization is low (pre-market)

Last response from NT:

Currently there are known bar caching limitations. The current bar caching improves the overall stability of the platform and is not a feature that we are going to revised at this time. It is recommended that if customers that plan to frequently change bar series with the custom chart styles to restart NinjaTrader.

Honestly, I am not sure the devs have a clue. At this point, I am done with this. Hopefully, NT devs can get their act together.

[NinjaTrader_RyanM]

Hello aslane,
Thank you for your response. At this time we are unable to reproduce this item. Currently the behavior is only appearing within your local PC and with the use/after the use of the custom bar type. I have forwarded this item to the Product Management team.

[aslane]

Actually, it does not require any custom bar types as I mentioned in post.

Also, when we last talked, you confirmed that at least one other person was seeing this issue.

[NinjaTrader_RyanM]

Hello Aslane,

To clarify a few items since maybe we’re on the wrong page. We spent many hours debugging and trying to reproduce the case, as we truly would want to solve an issue of this type.

We so far have not yet reproduced the case even running your custom chartstyles and believe the issue is an environmental issue at this time. Additionally we do NOT have any other reported cases of this type (the differentiating factor being this is the last bar in the chart which is a ‘realtime’ bar and not a cached bar). Per code review and testing combinations on our side we simply cannot see where the issue could be coming from. We have narrowed / isolated any caching logic as we never cache the last bar on the chart. The only two explanations left are, NT truly randomly had a 1 minute bar come through with that level when AddBar was called for this load, or deeper ring buffer problem deep in NinjaTrader which if that was the issue we would have more reports. Since the issue left us scratching our heads we then wanted to take the next step which would be to switch to a clean NT instance and see if we could reproduce there, and if not then start introducing things back one at a time until the problem comes back.

However since you had indicated that you did not want to do that we’re out of next steps at this time. I really appreciate the patience so far you’ve put in.

[aslane]

Yes, we have spent a lot of time trying things, and that is why I mentioned you have done a good job. The issue I have in trying to debug this for NT, is you want me to run full time with a clean Ninja. This is an issue because I actually trade and don't have the resources to do this on the side. Also, I truly believe you wont see it on a vanilla setup, so you are chasing something that wont happen, not to mention it only happened twice in the last week.

I think you can rule out the bad data arrived item. The reason being I am only running with a single data connection now (previously we thought you needed a second connection to force issue), and I have verified that all underlying data is correct when the issue occurs. I have a study that runs on each chart that verifies that every bar is correct in terms of the open/close are contained within the high/low, and it never triggers when a bad rendered bar shows up. Also, the same data is present on many other charts and it is always correct on other charts.

I doubt your ring buffer is an issue or you would likely have thousands of complaints, not to mention you would be able to easily reproduce.

I am fairly certain this is some kind of threading/race condition. Generally, things work, but somewhere there is something that knocks the normal rhythm off and the render thread is able to get ahead of the bar data being generated. For example, the BarTimer indicator has a timer running that causes a forced refresh. That refresh is not causing the issue, but is something that would run right around same time as a minute bar AddBar call.

Not knowing the actual implementation, it is hard for me to theorize too much, but somehow the Render gets/makes a copy of the new bar before it is truly updated (the Home-End work around picks up a new correct copy). These two processes are on different threads. This interaction is where I would be looking.

I can try new builds and help narrow the issue, but can not play the run from scratch game. If I am the only one seeing, feel free to write it off, but I suspect you will start seeing the issue more in the future, so I wanted to doc the info for anyone seeing issue.