Building a customized stock screener (monitoring 5,000+ stocks)

[DimaFridrich]

Hi team,
I have a question that was on my mind for a long time - is it possible to build a customized screener of the entire US stock market (at least for the backtest).

The Need and the existing problem:
My strategy is based on first screening stocks in live mode using Trade Ideas (TI) and then importing (loading) this list to my NT 7 based strategy for further execution (intraday trading).
Now, the list of traded stock symbols changes on a daily basis.
For me now. to perform a backtest I have to manually save the list of the stocks traded on each day (that I export from Trade Ideas). This process is very ineffective and time consuming.

Possible solution - building customized screener in NT 8.
Instead of using TI, why not to build a customized stock screener that would screen all the US stocks each day based on the same criteria I use in TI. Even if I am able to pull this off retroactively (just for the backtest) it will be a great win for me.

What is the following problem - Not sure that NT 8 or Kinetick (my data provider) can technically allow that. I'll explain:
NT 8 - Since I know that in NT 7 you cannot add instrument to already running strategy, will I be able to do it in NT 8 (on a code level) for such a huge amount of instruments (around 6,000)?
For example, I would like my strategy to look only at specific stock out of 6,000 on specific date (based on the initial filtering criteria), while on another day I would like my strategy to pick another batch of symbols based on the same 6,000 stock list. Will I be able to do so? If yes. what would be the most effective way? Will I have to load the whole symbol list regardless during "OnstateChange. Configure" stage?

Data Provider limitation - Kinetick in my case. They say that you can pull info of 100 symbols simultaneously.What does it mean? For example, if I now decide to backtest and load the entire 6,000 stock list will they limit me to only first 100 or will they allow me to do that, but the loading time will be longer as they would allow loading 100 stocks at a time?
If the first correct, how then I can at all resolve this whole situation? Will I have then to devide my backtest to 100 symbol batches and basically do 60 separate backtest? (hope there is a better solution).

Please advise.
Thanks in advance!

[NinjaTrader_Jim]

Hello DimaFridrich,

Thanks for your post.

I may suggest the following:

1) Building a multi time frame strategy that loads information of added symbols using a StreamReader. The StreamReader would read the symbols, and the strategy would call AddDataSeries for each added symbol.

2) Building a Market Analyzer Column or indicator as a "scanner" and use the Market Analyzer to deploy this column/indicator that will be your symbol scanner. This scanner should then use a StreamWriter that will list all symbols that should be backtested or should have the strategy deployed with.

Caveats:

Symbol limits and performance issues will be a bottleneck. Requesting data for 100's of instruments is a cumbersome operation in addition to having the strategy code process on all of those instruments. NinjaTrader itself does not have limits on data requests, but your data provider's limits on simultaneous data requests will be limiting. You are correct that downsizing the task into smaller chunks would be the way to work through this limitation.

As mentioned above, working with smaller chunks would be the way to get past heavy performance impact and exceeding your Market Data Provider's request limits. I.E. have several (as many as it takes) Market Analyzer templates that handle separate sections of your overall list to scan. The Market Analyzer would be run with the scanner Indicator/Column configured for the specific list that should be updated. After all lists are updated, you can then run the strategy which adds the instruments associated with this list and perform your backtesting.

It is technically unsupported to dynamically add additional data series, but if the list is loaded from file, the approach should work around this stipulation.

If you use OnBarUpdate analysis for your scanner, This will require a data series with at least 1 day of data to be requested/loaded for each instrument. This will bring a performance penalty when used in excess.

Examples for using a StreamReader/StreamWriter can be found below.

SampleStreamReader - https://ninjatrader.com/support/help...o_read_fro.htm

SampleStreamWriter - https://ninjatrader.com/support/help...o_write_to.htm

Thinking of any other ways this could be done would involve enabling the strategy against an instrument list of all symbols that the strategy should be deployed against. (Data Provider limitations will still be prevalent here.) We do not have a way to programmatically build instrument lists, but we are tracking interest in the ability to import symbols as an instrument list. If this sort of functionality were implemented, you could create a scanner that creates an Instrument List that could be imported, and then you could deploy your strategy against that Instrument List in the Strategy Analyzer or the Control Center. This ticket ID is SFT-1603 and I have added a vote on your behalf.

Please let me know if I can be of further assistance.

[DimaFridrich]

Hey Jim,
Appreciate you extensive answer.

Indeed, the direction of action is now clearer to me. The idea of Market Analyzer implementation is especially interesting as I wasn't really using this tool until now. Will have to explorer it further.
Also, I wasn't aware of the fact that it is possible to load additional symbols (even by reading an external file) when the strategy is already running. These are great news!

Will try to implement the above and will let you know if any follow up questions arise.

Thanks again!

[Sailor]

Hi Dima, Hi Jim,
I just read your post and interestingly I am looking for the same thing.
@ Jim, I have already solved the screener problem with Tradingview. I use their screener and download every 10 sec. the results into a database. Depending on your criteria, price ,rvol, float etc. the list is short max 10-50 stocks. In addition during the day, other stocks can pop up and will be added the the list of the day.
So I have two questions Jim,
1) can I also use the Sample reader to dynamicly update the Listt of symols where the stategy runs intraday? Can this be done automaticly? if the file is updated, the system should releod and run the strategy?
2) How about backtesting, is it possible to backtest a strategy, for a set of stock for a day? Example: Backtest only monday with the stocklist for monday, tuesday with the list for tuesday and so on backwards.

Regards Lars

[NinjaTrader_Jim]

Hello Lars,

Thanks for your questions.

1) can I also use the Sample reader to dynamicly update the Listt of symols where the stategy runs intraday? Can this be done automaticly? if the file is updated, the system should releod and run the strategy?

A StreamWriter would be used to write to file. You will have control over when the file should be opened/written to so it is certainly possible to update the list throughout the day. The strategy will need to be reloaded to process State.Configure and load the symbols with AddDataSeries.

2) How about backtesting, is it possible to backtest a strategy, for a set of stock for a day? Example: Backtest only monday with the stocklist for monday, tuesday with the list for tuesday and so on backwards.

If you set up instrument lists, you could run a basket test of X symbols against X day. Instrument lists must be created manually as there is no supported way to build instrument lists programmatically.

Working with Instrument Lists - https://ninjatrader.com/support/help...ment_lists.htm

Basket tests - https://ninjatrader.com/support/help...asket_test.htm

Please let me know if I can be of further assistance.

[Sailor]

Dear Jim,
Thanks for your answer, so i can use the Streamreader in a strategy to update the symbollist. But is it also possible to update the market analyzer window based on a Symbollist file and dynamicly updated?

[NinjaTrader_Jim]

Hello Sailor,

You could have a MarketAnalyzerColumn add a timer that checks your file for any changes and then updates CurrentValue to what you would like seen in the Market Analyzer.

Please let us know if we can be of further assistance.

[Sailor]

Thanks Jim, but how can i update the symbols in the market Analyzer window from the file?

[NinjaTrader_Jim]

Hello Sailor,

You would need to make Instrument Lists that contain your target symbols, and then load that Instrument List in the Market Analyzer. There is currently no supported way to programmatically create an Instrument List.

Working with Instrument Lists - https://ninjatrader.com/support/help...ment_lists.htm

Please let me know if you have any questions.

[mcosta72]

Hi NinjaTrader_Jim ,

I am looking for a Timer exactly the same you published for MarketAnalyzerColumn but running on my AddOn. I can't figure out the right class method.
Is there any way?

Thanks

[NinjaTrader_Jim]

Hello mcosta72,

I have attached an example that adds a timer in an AddOn tab page. Note that we would not be using TriggerCustomEvent because there would not be any OnBarUpdate or built in indexing functionality where it would be required.

Please let us know if we can be of further assistance.

[mcosta72]

That's great, thank you very much!