Level1 Market Replay from Historical Tick Data Cache

[whitmark]

The improvements in Market Replay speeds are very useful but the greatest bottleneck now is the availability of recorded Market Replay data. It can be very hard to capture contiguous streams of data for a full day much less for weeks at a time with internet feed disconnects, application crashes, and other undesirable events that interrupt the recording of this data. It would be very desirable to be able to load the data cache with tick data (bid, ask, and last price/volume) and then be able to populate a Level1 Market Replay file for playback. I have used a similar feature in another trading package with my eSignal feed and I find it invaluable in enabling my forward testing activities with the assurance that I have a complete sample of tick history.

Perhaps the Market Replay infrustructure cannot accommodate this type of "forced recording" in which case I would recommend an alternative data source of "Data Replay" be considered. Unfortunately, Level2 will not be available for historical download unless the respective exchange makes it available in an easy to use format, so personal recording is the only viable option for now.

Any additional thoughts on this idea are most welcome.

Regards,

Whitmark

[NinjaTrader_Dierk]

Thanks for sharing your thoughts.

[KBJ]

If the data formats that NinjaTrader uses for Market Replay file datastreams are available, it would be possible to make some standalone utilities which would merge datastreams from various sources (e.g., different users' saved datastreams).

Then perhaps a data repository for the community could be developed and people could pool their data.

Ideally, we'd also want to have a means for comparing saved data as well, so that we could verify that saved datastreams are consistent to one another which might help prevent data errors.

[NinjaTrader_Chris]

Suggestions and Feedback

Hello,

Thank you for your suggestions. I have forward them to development and will have them add it to our future enhancment list.

[zoltran]

Didn't NT used to keep daily replay files arround?

Why was this stopped? I'm guessing some sort of legal or support issue.

[NinjaTrader_Dierk]

Last we had some sample replay files for a few days. However, they never where up to current which is why the stop providing these files.

[Red Fish]

Replaying historical data or converting historical data for replay

Whatever happened to this idea? To be honest when I first signed up to Ninja Trader that is what I thought I was getting, a way of replaying history so that I could hone my skills of entering the market in the same time period every day...my trading method is time of day dependent which means it takes years to develop the skill and experience necessary, something backtesting never really gives you.

I was in search of a simulator that might pack in years of trading experience into months, but that doesn't seem what NT is about. A shame because I think its popularity would be far greater with that facility. Can't we get a working converter to at least convert text files into the .ntm files NT uses?

[NinjaTrader_Ray]

There is no such converter at this time.

[hemlock]

I would be willing to write a converter for everyone to use if the format of the tick replay files was released to the public.

Ken

[whitmark]

I had heard the guys at Money Flow Bars had a "History Player" built for them but didn't realize until now that they are offering it for sale.

http://www.moneyflowbars.com/historyplayer.html

Anyone have any experience with it?

[hemlock]

Whitmark, Had not seen the MoneyFlow app. Here is another:

http://www.ninjatrader-support.com/vb/showthread.php?t=6147

I don't think either replays multiple instruments. That would be quite useful for my purposes.

[whitmark]

Thanks Hemlock, I had not noticed VitimanC's utility before. Kudos to VitimanC for making this generous contribution to the forum. The existence of more than one tool to do this does suggest a need for this type of functionality and potentially a very useful platform extension.

In the attached exhibit is my "old school" way of doing this, albeit using a tick server from another platform to show there are several ways to do this. However, there are currently some limitations in Ninja from fully enabling this functionality.

In the attached example, I am simultaneously pushing ticks for the ES and YM continuous contracts from a data cache that was updated with eSignal bid/ask/last data. Shown are two source charts (but this could also be setup as a single source chart) and does require the use of a custom indicator to push ticks to NinjaTrader across the Ninja ATI. On the left are the two source charts set to tick update and the tick player (simulation server). On the right are two Ninja charts that are being updated simultaneously with bid/ask/last data tick by tick via the External Data Feed. The SampleMACrossOver strategy is used on both charts.

When operating in this manner to facilitate larger scale forward testing runs when Market Replay data is not available or not convenient to work with, it’s important to keep in mind:

• The External Data Feed was designed to accept realtime data and therefore it has some issues when working with historically datetime stamped data. It has been my experience that only tick data will work in this mode, and not minute or daily data. It is my understanding from a historical post that you can change your PC clock’s time to fake out Ninja into thinking the data is realtime vs historical data, but I have no firsthand experience doing this. See the attached link:
  
  http://www.ninjatrader-support.com/v...ead.php?t=3618

• When streaming tick data via the External Data Feed, the data cache for the symbol you are updating will be modified. So either, backup your data or better yet, stream the data to another symbol that you have created in the Instrument Manager exclusively for this purpose.

• If you rerun the historical data over the same period multiple times at faster replay speeds, it’s likely you will get duplicate key errors in your data cache that you would have to repair which is another reason to back up your primary data and only use a throwaway symbol for this replay purpose.

• I have been getting inconsistent trade marker display behavior, sometimes plotting, sometimes not, that I have not been able to pin down yet. At one point, since Ninja thought it was historical data, I was getting the historical green and red position lines on “realtime” entries and exits.

Obviously, use this or similar approaches at your own risk, since its likley unsupported operations. I would certainly welcome more direct platform support for this type of operation and sharing of experiences with others who are would like to see longterm, uninterruped forward testing possible.

Regards,

Whitmark

[whitmark]

In NT7, will there be the ability to drive NinjaTrader with external historical bid/ask/last data through the ATI for testing purposes and not require a rollback of the computer clock?

Thanks

[whitmark]

I did some tests using 6.5.1000.7 and it looks like its working fine with the Ninja chart correctly building both tick- and time-based bars. Thank you!

[DangerBoy]

Mark,

Funny how we all end up down the same paths

I was just about to start myself a big project of writing an external simulation server to feed ninja as I've come to realise the current market replay feature is not sufficient for my needs due to the constant interruptions in the replay data as well as the recorder player having limited flexibility in navigating the recorded data.

Luckily I stumbled across this post of yours beforehand and the information in it is invaluble. I can see we face similar challenges but you've gone the route of using an existing sim server to feed Ninja which I actually think is a great option since it means not having to write all the feeder code from a custom data base etc.

I had a couple of questions for you, as I've never used Neo, how is the flexiblity of the recorder navigation bar, does it manage to do everything you need.

Specifically for debugging purposes I would like to do things like set repeat play loops and be able to define stops and starts, navigate to a specific point in time by typing in the actual date and time. I can see from the picture that you are able to navigate forward one time period at a time tick/minute etc. which looks great as I need that also. Also with the Nero Sim server can one import your own bid/ask tick level information?

Anything you would say to look out for? I'm thinking in the end this may be the solution for myself also.

Also a question to NT guys regarding the design of this replayer. I'm asking the question more from a point of view that perhaps there are some challenges in this set up that I have overlooked and you have already faced and it's the reasoning behind the current design.

Why do we have a different repository for replay data from actual data?

Surely ideal scenario is we have one data repository that stores all necessary information including Bid/ask etc., this repository can be loaded via the NT import feature. The replay feature will then allow replay of this data from any point in time to any point in time with a flexible recorder bar that allows good navigation of data, as the Nero Sim bar seems to do. Can you give a bit more detailed direction of where NT7 is going with this feature, as I would hate to go to all the work of figuring out Nero and writing external feeder connections, Neo indicators etc., when NT7 will be sorting out this area of weakness.

I have seen in Rays update notes that continuous playback will be possible over multiple days but no mention of whether the recorder will be abolished and we'll be able to import our own Bid/Ask info and whether the existing recorder navigator bar is going to be polished up a bit?

Thanks for your help.

Cheers
Ross