Order Execution Performance in Real Time

[iantg]

I have been trying to optimize my speed performance for a while now and I have a few observations and a few questions on this subject.

I am co-locating near the exchange I am trading on and near the data provider I use. I have tested a variety of strategies, some just a few lines of code, and some as much as ten thousand lines of code. I am starting to see a correlation where the more memory being used, the more complex the code base and the harder I push the ninjatrader the slower my execution speeds tend to be. Now I am not sure if this is truly a measurement of the order execution speed or just the reporting of the speed. Here is what I am measuring first:

1. Submit an order: Grab the time stamp: with something like:
Print("Order Submitted" + "/" +
order.Name + "/" +
order.OrderState + "/" +
Time[0] + "/" +
DateTime.Now.Millisecond + "/" +
DateTime.Now.Ticks);

2. Then on the OnOrderUpdate subroutine I collect the timestamps of every state change with a similar print:
Print("Order State Change" + "/" +
order.Name + "/" +
order.OrderState + "/" +
Time[0] + "/" +
DateTime.Now.Millisecond + "/" +
DateTime.Now.Ticks);

What I am measuring is the time it takes to move through these events

1. Order first submitted from my entry signal block
2. Initialed State
3. Submitted State
4. Accepted State
5. Working State

With my particular co-location At my best I have been able to get this entire sequence from 1 -5 down to < 1 millisecond. Which for a retail trader, is extremely fast. But the caveat is that I have only hit that speed with a simple test script with only about 15 lines of code, no charts running, memory very low, etc. When I run my real production code my average time to go through these order state events (1-5) is around 10 to 25 milliseconds. In my production use case, I run lots of prints collecting meta-data and I run a 1 tick time series and most of my code executes off of OnMarketDepth. I also have several streams open from OnMarketDepth that are tracking volume updates at all times. So there is a lot going on here.

I have never been able to get past 10 milliseconds with my production code, and I can hit < 1 milliseconds 3 out of 5 times with my speed test code that is around 15 lines. So this leads me to believe that I am hitting some performance issues at full scale. I am not sure if the performance issues are:

1. By using lots of memory intensive processes and running prints, collecting level 2 data, etc. I am not able to submit my orders fast enough, and moving through various order states NinjaTrader is not able to capture these events.
2. By using lots of memory intensive processes, and running prints, collection level 2 data etc. I am not able to track my statistics on order state in real time. There are lags and my print times are firing late because NT is running slow.

So 1 would be more of a true performance issue, whereas 2 would be more lags in the printing and outputting of capturing these events.

For reference my sample size is around 30 orders with my production code ranging from
Best: 10 Millseconds
Worse: 200 Milleseconds
Average: 25 to 50 Millseconds

My Speed test code (15 lines) No other processes running:
Best < 1 Millseconds
Worst: 50 Milliseconds
Average: 5 Milliseconds

Also for reference even in my worse case scenario NT never runs more than 500 MB of memory and the total memory of my VPS is usually never higher than 20%. So I am not my any means maxing this out.

So here are my questions:

1. Am I correct in assuming that at a certain threshold (Lines of Code, Open streams to Level 2 data, running 1 tick time series, etc.) NT will hit performance issues that can impact order execution or reporting of order executions.
2. All things considered do you think I am hitting real execution lag, or just lag reporting the executions? For reference my Log File time stamps match my Output Window Prints.
3. What would be the best advice to improve my performance? Are there any ideas you can provide beyond just the following?

A: Reduce the lines of code
B: Close level 2 data feeds, or use conservatively
C: Don't print to the output window
D: Don't use a 1 tick data series

A-D may be deal breakers for my production code, and I may not be able to get around these. So I am looking for any other performance enhancing ideas:

Thanks,

Ian

[NinjaTrader_Jesse]

Hello iantg,

Thank you for the post.

1. Am I correct in assuming that at a certain threshold (Lines of Code, Open streams to Level 2 data, running 1 tick time series, etc.) NT will hit performance issues that can impact order execution or reporting of order executions.

There is no specific threshold per se, you can trigger performance problems even with simple syntax. Your logic and its efficiency will play a large role here. You can have very large amounts of code and only utilize a small fraction of the overall code so the question is reliant on what code is actually being executed. NinjaTrader will execute the C# code as it was written so if that code can cause performance impacts it likely will. This question also heavily relies on the computers hardware and its processing ability. Depending on what your code is doing, you could certainly see performance impacts in the platform. We have a note about some effects you can see when executing problematic NinjaScript code here.

2. All things considered do you think I am hitting real execution lag, or just lag reporting the executions? For reference my Log File time stamps match my Output Window Prints.

Although this is a lot of information about the situation, I really couldn't say without a specific example of what logic is used in each test. You would likely need to debug the overall script further and isolate what the difference is between the simple and complex test to know what is happening in more detail. If you are executing some syntax that is in someway slowing the platform down, you may see impacts in other areas of the platform which may lead to items like the increase in time between the prints or slowness in the UI.

3. What would be the best advice to improve my performance? Are there any ideas you can provide beyond just the following?
A: Reduce the lines of code
B: Close level 2 data feeds, or use conservatively
C: Don't print to the output window
D: Don't use a 1 tick data series

The items you have noted are good places to review to see if they make differences in your testing. Item A would be an obvious choice wherever possible but not mandatory. Excessive printing or writing to disk should also likely be avoided if it can be. The less going on in the user interface ultimately will allow NinajTrader to have more resources available to execute logic.

I couldn't say that removing tick series would be a solution unless your logic doesn't require the tick series. If removing a whole series that would affect your logic controls the performance, you may look at other ways to mitigate that like reducing the amount of data loaded or what is being done on that BarsInProgress.

In addition to what you mentioned, I would suggest to further debug the script to find what total syntax is required to make the difference in prints. If it is known specifically what the requirement is to see the difference, that syntax can be further explored to understand why.

I look forward to being of further assistance.