Playback Connection Freezes

[koganam]

Playback had to be major implementation changes in beta 3 so any old reports are not of value since a lot of the code for playback had to be updated to fix some critical bugs.

As such we must treat any new issue on playback with Beta 3 as unrelated issue from prior betas.

For BernWillChris post this is expected if there is no data to playback. The time in the top right of the playback control will scrub through the day even if there is no data to play. Make sure you have data available for that time by right clicking on the playback control and selecting "Available Data". If it still is not playing then please let me know what instrument, time frame, what window you have selected and if you downloaded data from NinjaTrader historical data servers and any other data that might help reproduce so we could isolate further.

For NJA_MC post this is very strange and I'm never able to reproduce that the time in the playback control never updates. If you sent logs to Patrick can you please also send your contact information so we could try a remote session to look into that issue further since that is a very strange issue we wanted to look into with priority.

That pretty much explains about my post. Thanks for taking the time to respond, given how busy you must be.

[NJA_MC]

Sorry,

I got tied up at work, hope to do a clean run so I can get the log's together for you for Playback. I am likely to be around on Friday if you want to log in to test some things out.

[NJA_MC]

I have had a chance to work with Playback connection again. What I suspected may have happened is I may have had bad data imported from NT7, I know I was getting an error from NT7 at times on some "historical data loads" under replay. It might be that NT8 doesn't report and might fail transparently.

So with that concept, I deleted all the older data (manually from the DB folders). I tried downloading historical data from NTBroker so I can use the Tick Replay (Historical) setting.

I started up NT8 and connected to Playback and the playback dialog opened (Set to Historical data). The Chart was a 1 Range Chart. I chose a time range I knew was in the system downloaded from NTBroker before. First thing I noticed is the CHART didn't reset like it used to in the past. Once I selected a time range the CHART would jump to this time. There was historical data available prior to the time I chose and that data still showed on the CHART (Picture attached). What remained visible on the chart was the same historical data from yesterday (didn't reset).

I pressed play and time did start moving forward! So this is major progress which means there may have been bad Historical/Replay data in the database. The issue here is the CHART displayed data from Yesterday (didn't update) but the Data Range printed on the chart is the earliest Historical Data in the database. And the playback was actually ignoring the START date I had set.

As it was ignoring my Start Date, I tried to go to the Market Replay setting as I had downloaded a few random days. The CHART updated to the earliest day and started playing at high speed! Chart was scrolling along with it!!! (Happy dance)

I think clicked back to the Historical and now the Playback is responding as expected, update the start date on the historical and the chart changes but only if the "Start Date" is OUTSIDE the visible range of the chart. If it is already on the screen it seems to wait until you "PLAY" before the screen really updates.

The next and most telling test occurred next, I disconnected from "Playback". In the past, once I did this, the Control Center showed the disconnection, but the Playback dialog box remained open. If you tried to open another connection it would refuse stated that I had to close the Playback connection before I could open a different connection. Ultimately, I had to use the task manager to terminate NT8 and restart to do anything else on the platform. So this implies to me, the problem I was having before was because the Playback system had crashed but not reported the problem.

I think exited NT8 and restarted to see if I get the original startup behavior or if the system seems to be working correctly. I am happy to state that the Playback now seems to be working. It seemed like clearing out old data (maybe corrupt data) has stopped the PLAYBACK from transparently crashing on my system. This may have left NT8 BETA3 in an unknown state until I managed to clear that state with fresh Replay Data and a short Replay run.

So I will take a few more steps and watch this feature to make sure it doesn't recur. I am going to move back to my 15 instrument test setup to see if I can crash the Playback. It is possible a strategy or other exception put Playback in this state so I will play around now I can get it moving. I will try something other than 1 Range chart as well.

Some thoughts for Support/Development to check:
- Make sure Historical/Replay Data is properly parsed and report errors
- Playback should not transparently Crash

[NJA_MC]

Date Range Reduced

I have selected a 10 Min chart for the CL and now my Playback Start Data Range has been clipped to July 16th.

There is clearly more historical data in the system...

Also, there are some issues with setting the starting date again. Not sure why but sometime when you select a new START date the chart doesn't update, nor the Playback time shown on the top row of the Playback Dialog.

[NinjaTrader_AlexC]

Hi NJA_MC,

Would you mind sending in a copy of your log and trace and workspace files from this time to platformsupport[AT]ninjatrader[DOT]com for further investigation?

[cls71]

I also have problems with the Playback.

1. Downloaded ES 03-14 15/01/2014.
2. Setting Start and End DatePickers in the control at 15/01/2014.
3. It is not possible the replaying. The cursor is at the end and I cannot move it.
   
   
   
4. Also, I cannot already change the date from the date pickers. The datapickers are frozen and I cannot change the dates though I own data from others dates.
   
   
   

[NJA_MC]

Hi Alec,

I have sent the logs off, I think Patrick has them.

[NJA_MC]

Hummm.... Happened again...

Darn,

It happened again! The Playback box doesn't seem connected to the Control Center or charts. Sometimes when I Disconnect the playback it disappears, but other times it is clearly hung. You can see in the attached images I have disconnected from Playback, but the Dialog is still open, and then I try to open the NTBrokerage and reports an error.

One thing showed in the middle of the PLAYBACK session in the Trace file (Playback did not hang, I could exit Playback mode):

 2015-08-01 20:47:54:561 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=CashValue currency=UsDollar value=*****
2015-08-01 20:47:54:801 () Cbi.NinjaTraderProxy.OnSocketCompleted.SocketError: socketError=NotConnected
[B]2015-08-01 20:48:07:533 Cbi.Instrument.RequestBars (to Provider): instrument='CL 05-15' from='04/14/2015 00:00:00' to='04/16/2015 23:59:59' period='1 Tick'
2015-08-01 20:48:15:261 ERROR: Connection to HDS server lost
[/B]2015-08-01 20:48:24:304 () Cbi.NinjaTraderProxy.OnSocketCompleted.SocketError: socketError=NotConnected

I will email the logs to support for review. I will continue to test addons, etc to see if custom ninjascript or something is causing this issue...

EDIT: I restarted NT8 and tried a cleaner start, just a single 1 range chart and went to playback. Got these errors in the log immediately (Playback Dialog Hung on this error group):

2015-08-01 21:04:56:385 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=BuyingPower currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=CashValue currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=Commission currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=Fee currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=GrossRealizedProfitLoss currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=InitialMargin currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=IntradayMargin currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=LongOptionValue currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=LookAheadMaintenanceMargin currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=LongStockValue currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=MaintenanceMargin currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=NetLiquidationByCurrency currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=PositionMargin currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=RealizedProfitLoss currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=ShortOptionValue currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=ShortStockValue currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=SodCashValue currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=SodLiquidatingValue currency=UsDollar value=*****
2015-08-01 21:04:56:390 (Playback Connection) Cbi.Account.AccountItemCallback: account='Playback101' accountItem=TotalCashBalance currency=UsDollar value=*****
2015-08-01 21:04:58:496 Cbi.Instrument.RequestBars (to Provider): instrument='CL 06-15' from='04/19/2015 00:00:00' to='04/20/2015 23:59:59' period='1 Tick'
2015-08-01 21:05:03:056 ERROR: Connection to IS server lost
2015-08-01 21:05:12:098 () Cbi.NinjaTraderProxy.OnSocketCompleted.SocketError : socketError=NotConnected
2015-08-01 21:05:17:054 ERROR: Connection to HDS server lost
2015-08-01 21:05:26:097 () Cbi.NinjaTraderProxy.OnSocketCompleted.SocketError : socketError=NotConnected

[NJA_MC]

I tried Safe Mode to see if playback mode would work. Same problem, got just the HDS error.

[NJA_MC]

More data:

The common thread around this issue after disabling all of my code/addons as well as using safe-mode:

- Once Playback connection is selected Playback Dialog opens always in HISTORICAL
- The Chart Shows "Loading..." and this message doesn't clear
- The Playback box shows no Time/date at the top while "playing"
- Chart time doesn't change to Start time upon opening the Playback dialog or when start time is modified
- Trace always shows: ERROR: Connection to IS server lost
- Trace always shows: ERROR: Connection to HDS server lost
(This shows after attempt to load selected instrument on chart) EG: Cbi.Instrument.RequestBars (to Provider): instrument='CL 05-15' from='04/15/2015 00:00:00' to='04/19/2015 23:59:59' period='1 Tick'

Just for fun, I changed time-zones to make sure the consistent 8PM No historical data bug wasn't causing this, changing time-zone didn't seem to matter.

[NJA_MC]

Found a way to force Market Replay in Playback

I removed the UI.XML file which defaulted the Playback mode to "Market Replay". I could then use the Market Replay connection, I don't have much data loaded but it ran fine on the available data.

The problem changed, but might make the issue more visible when I clicked on "Historical". At that point the "Loading..." showed in the chart and the Playback box FROZE.

<?xml version="1.0" encoding="utf-8"?>
<NinjaTrader>
<RecentInstruments>
<RecentInstrument>CL 09-15</RecentInstrument>
</RecentInstruments>
<PinnedInstruments />
<WindowSizes>
<NinjaTrader.Gui.Data.PlaybackControlCenter>
<Height>128</Height>
<Width>330</Width>
</NinjaTrader.Gui.Data.PlaybackControlCenter>
</WindowSizes>
<PlaybackControl>
<Location>1402;882</Location>
<Size>330;128</Size>
<IsDataSourceHistorical>False</IsDataSourceHistorical>
</PlaybackControl>
</NinjaTrader>

Setting the above "IsDataSourceHistorical" to "False" got me to Market Replay. (True is Historical).

This might mean the "Historical" database is still bad on my system? I am 7Zipping the files now from the DB directory incase you want to use them to debug this issue. I don't have a place I can put the 1GB file so if you have an FTP or service, let me know and I will upload it.

[NJA_MC]

NTBroker Historical Data Server down?

Is it possible the NT Historical Data Server has been down this evening?

Since I Deleted the historical data for testing, I don't seem to be able to pull any new data in to populate charts or historical data download. I am getting the HSD disconnected statement shortly after startup when I try to load a chart. It is also stating the IS connection is lost.

I don't know if this has cause the problem I documented earlier... I do use NT Brokerage.

[NJA_MC]

Is it possible the NT Historical Data Server has been down this evening?

Since I Deleted the historical data for testing, I don't seem to be able to pull any new data in to populate charts or historical data download. I am getting the HSD disconnected statement shortly after startup when I try to load a chart. It is also stating the IS connection is lost.

I don't know if this has cause the problem I documented earlier... I do use NT Brokerage.

I think we are getting closer to a cause of this problem!

After I realized the HSD server in the error was (Historical Data Server) and saw this error on every attempt to start market Playback, I decide this morning when the HDS came back on-line to copy the "OLD BAD DATA" back into the DB folder.

Sure enough, I connected to the Playback after reloading the data and "Market Replay" worked as it always did (now I know how to get back to it if needed). I then selected "Historical" and after a few second, it came right up!

A quick check of the Trace File and there are NO ERRORS for the SI or HDS!

It appears that the "Playback -> Historical" needs the NT HDS server live. This doesn't seem to make sense as all the required data should already be downloaded. I tend to use the system on the weekends when the HDS server is down for maintenance...