this should be interesting to all, i have been evaluating nt's optimization engine.

[rtwave]

good day to everyone,



i have had very strange results when using the optimization engines both on nt 7 and nt 8 before. consequently, i decided to do some very general testing on the performance of the optimization engines themselves and i have some preliminary results now.


i have some suggestions to make and questions to ask of the people at nt, and would also appreciate if other users shared their experiences and advice using nt's optimization engine.


so far i have run all my tests on nymex's cl contract with end date 31-XII-2014 optimizing for largest long + short net profit. i'll explain my tests in more detail further below.


- the first situation i would like to point out is that running optimization processes in ninjatrader is unnecessarily cumbersome. one can only run one single optimization at a time and there's no way to create a set of several optimizations so that they all run one after another. in nt one has to wait until the optimization in process is finished before being able to run another one and this is inefficient, tedious and excessively time consuming. in other platforms i have used, one can run one optimization per data chart and have as many charts as one wants so that one can prepare an unlimited number of optimization processes and they will run unattended one after another for hours or days.

- the second improvement i would suggest has to do with being able to copy the optimal settings for the parameters of a strategy that has been optimized to another chart with a different symbol or a different time period. in other platforms i have used one can optimize a strategy up to a certain date on a chart leaving data out of sample and then just change the end date on the chart, and the data that had been left out of sample will be added to the chart. in this way it is easy to evaluate how strategies would have done on real live data. one can copy and paste these charts that include optimized strategies and subsequently change their symbol, start and end date, and all other properties like values for slippage, commissions, etc. this allows for great flexibility and ease when evaluating and optimizing strategies but nt completely lacks these features and everything must be done typing all the optimized settings by hand.



the following is a composite image i have created from the reports of several different strategies i evaluated. i started by evaluating the sample sma cross over strategy that is included in the nt platform for largest long + short net profit. i ran optimizations for this strategy disabling the exit on session close feature on daily, 1440 minute and 360 minute bars and seemingly all these variations worked correctly. i then evaluated the same strategy on the same bars but enabling the exit on session close feature and seemingly all worked fine as well. i then optimized the same strategy on the same bars just adding a stop loss order for every position in the strategy and results were also positive (these cases are the ones with green charts on the composite image below).


in all of these cases, it seems like the strategies functioned as expected and the optimization engine also did its job. the size of the profits is not so relevant and i don't think these strategies could be traded with real money, but it seems like the strategies and the optimization engine were working acceptably.


the first unexpected result came when i tried to optimize the sample sma cross over strategy on the same bars incorporating a trailing stop order (calculated on ticks) for every position. in this case this variation generated losses in all kind of bars i tried. the blank reports in the composite image are of these cases with trailing stops, as the platform reported that the best results were when both sma periods were identical and no trades were generated, therefore the optimal result was 0.







and then the biggest issue came when trying to optimize any kind of strategy on range bars. every single combination of range size (30, 40 or 50 ticks), strategy, exit on session close, stop orders or no stop orders i tried generated zero trades. this must be a malfunction of the platform as it was not like the case with trailing stops where the optimal parameters generated zero trades and the rest were all losing settings, in this case there are zero results. the following report illustrates this situation.





these are my results so far. i'm fairly experienced at developing automated strategies so i think these unexpected results with trailing stop orders and range bars must be malfunctions with the nt platform.


the following two screengrabs illustrate how the same sample sma cross over does work and generate trades on a cl range 50 chart and how the sma cross over with a trailing stop also works on a cl range 50 chart and should generate profits if the optimization engine worked properly.







i will now do similar tests on other bar types like renko, tick and volume and will report back as soon as i have any results. i'm not inclined to spend any time right now evaluating more complicated strategies until these initial malfunctions have been solved and i better understand the limits of the nt platform and the optimization engine.


very well, thanks, regards.

[NinjaTrader_PatrickH]

Hello rtwave,

Thank you for your post.

I will submit your two requests to development. One for optimizations to run one after another and the other to allow for templates and copying the parameters of optimization results to a new strategy.

Concerning the runs of your backtests with no results I am unable to reproduce this behavior.

Can you attach the strategy you test for the Trailing Stops?

And please attach the strategy you test for the Range bars if not just the SampleMaCrossover.

In addition, please provide a screenshot showing the full parameters section of the Strategy Analyzer so I may test with the same settings.

I look forward to your response.

[rtwave]

Patrick,


thanks.


those are excellent news. it is indeed an urgent necessity to be able to queue multiple optimization processes at once in nt instead of having to wait out every single one of them in order to be able to start another. being able to copy and paste optimized strategies with optimal parameters to different data charts and other windows is also a greatly needed and immense improvement on the current state of the nt platform.



now, if you say you didn't get the same aberrant results i am reporting when you tried to optimize with trailing stops or on range bar charts those would be excellent news as well.


the following are the strategies i have been using to test the optimization engine. the first one is the sample sma cross over strategy just as it is included in the nt platform, the others are identical strategies i created with the wizard to be able to include an optimizable input parameter for position size, as well as stop loss and trailing stop size in ticks. in all cases i optimized for maximum net profit, with the genetic optimizer over 100 generations with 100 generation size. for both sma periods the parameters were optimized over this range (3;100;2), the position size was always left unchanged at 1, the stop loss was optimized over (50;200;10) and the trailing stop over (30;200;10).


as i reported, when trying to optimize on range bars, all of these strategies returned blanks and the trailing stop strategy reported losing results in all the tests i carried out. i will leave out the declarations and wizard settings sections in all cases to keep these posts from getting too long.

//This namespace holds strategies in this folder and is required. Do not change it.
namespace NinjaTrader.NinjaScript.Strategies
{
    public class SampleMACrossOver : Strategy
    {
        private SMA smaFast;
        private SMA smaSlow;

        protected override void OnStateChange()
        {
            if (State == State.SetDefaults)
            {
                Description    = NinjaTrader.Custom.Resource.NinjaScriptStrategyDescriptionSampleMACrossOver;
                 Name        = NinjaTrader.Custom.Resource.NinjaScriptStrategyNameSampleMACrossOver;
                Fast        = 10;
                Slow        = 25;
                // This strategy has been designed to take advantage of performance gains in Strategy Analyzer optimizations
                // See the Help Guide for additional information
                IsInstantiatedOnEachOptimizationIteration = false;
            }
            else if (State == State.DataLoaded)
            {
                smaFast = SMA(Fast);
                smaSlow = SMA(Slow);

                smaFast.Plots[0].Brush = Brushes.Goldenrod;
                smaSlow.Plots[0].Brush = Brushes.SeaGreen;

                AddChartIndicator(smaFast);
                AddChartIndicator(smaSlow);
            }
        }

        protected override void OnBarUpdate()
        {
            if (CurrentBar < BarsRequiredToTrade)
                return;

            if (CrossAbove(smaFast, smaSlow, 1))
                EnterLong();
            else if (CrossBelow(smaFast, smaSlow, 1))
                EnterShort();
        }

        #region Properties
        [Range(1, int.MaxValue), NinjaScriptProperty]
        [Display(ResourceType = typeof(Custom.Resource), Name = "Fast", GroupName = "NinjaScriptStrategyParameters", Order = 0)]
        public int Fast
        { get; set; }

        [Range(1, int.MaxValue), NinjaScriptProperty]
        [Display(ResourceType = typeof(Custom.Resource), Name = "Slow", GroupName = "NinjaScriptStrategyParameters", Order = 1)]
        public int Slow
        { get; set; }
        #endregion
    }
}

//This namespace holds strategies in this folder and is required. Do not change it.
namespace NinjaTrader.NinjaScript.Strategies
{
    public class tesmacrossover : Strategy
    {
        private SMA smaFast;
        private SMA smaSlow;

        protected override void OnStateChange()
        {
            if (State == State.SetDefaults)
            {
                Description    = NinjaTrader.Custom.Resource.NinjaScriptStrategyDescriptionSampleMACrossOver;
                 Name        = "tesmacrossover";
                Fast        = 10;
                Slow        = 25;
                P01s        = 1;
                // This strategy has been designed to take advantage of performance gains in Strategy Analyzer optimizations
                // See the Help Guide for additional information
                IsInstantiatedOnEachOptimizationIteration = false;
            }
            else if (State == State.DataLoaded)
            {
                smaFast = SMA(Fast);
                smaSlow = SMA(Slow);

            }
        }

        protected override void OnBarUpdate()
        {
            if (CurrentBar < BarsRequiredToTrade)
                return;
            
            if (smaFast[0] > smaSlow[0])
            {
                EnterLong(Convert.ToInt32(P01s), @"p01");
            }
            
            if (smaFast[0] < smaSlow[0])
            {
                EnterShort(Convert.ToInt32(P01s), @"p01");
            }

        
        }

        #region Properties
        [Range(1, int.MaxValue), NinjaScriptProperty]
        [Display(ResourceType = typeof(Custom.Resource), Name = "Fast", GroupName = "NinjaScriptStrategyParameters", Order = 0)]
        public int Fast
        { get; set; }

        [Range(1, int.MaxValue), NinjaScriptProperty]
        [Display(ResourceType = typeof(Custom.Resource), Name = "Slow", GroupName = "NinjaScriptStrategyParameters", Order = 1)]
        public int Slow
        { get; set; }
        
        [NinjaScriptProperty]
        [Range(1, int.MaxValue)]
        [Display(ResourceType = typeof(Custom.Resource), Name="P01s", Description="position size 01", Order=3, GroupName="NinjaScriptStrategyParameters")]
        public int P01s
        { get; set; }
        #endregion
    }
}

[rtwave]

and these are the stop loss and trailing stop variations of the sample sma cross over strategy:

//This namespace holds Strategies in this folder and is required. Do not change it. 
namespace NinjaTrader.NinjaScript.Strategies
{
    public class tesmacrossoversl : Strategy
    {
        private SMA SMA1;
        private SMA SMA2;

        protected override void OnStateChange()
        {
            if (State == State.SetDefaults)
            {
                Description                                    = @"trial error sma crossover stop loss";
                Name                                        = "tesmacrossoversl";
                Calculate                                    = Calculate.OnBarClose;
                EntriesPerDirection                            = 1;
                EntryHandling                                = EntryHandling.UniqueEntries;
                IsExitOnSessionCloseStrategy                = true;
                ExitOnSessionCloseSeconds                    = 50;
                IsFillLimitOnTouch                            = false;
                MaximumBarsLookBack                            = MaximumBarsLookBack.Infinite;
                OrderFillResolution                            = OrderFillResolution.Standard;
                Slippage                                    = 3;
                StartBehavior                                = StartBehavior.AdoptAccountPosition;
                TimeInForce                                    = TimeInForce.Gtc;
                TraceOrders                                    = false;
                RealtimeErrorHandling                        = RealtimeErrorHandling.StopCancelCloseIgnoreRejects;
                StopTargetHandling                            = StopTargetHandling.PerEntryExecution;
                BarsRequiredToTrade                            = 1000;
                // Disable this property for performance gains in Strategy Analyzer optimizations
                // See the Help Guide for additional information
                IsInstantiatedOnEachOptimizationIteration    = true;
                Fsma                    = 5;
                Ssma                    = 10;
                P01s                    = 1;
                Ss01                    = 8;
            }
            else if (State == State.Configure)
            {
                SetStopLoss(@"p01", CalculationMode.Ticks, Ss01, false);
            }
            else if (State == State.DataLoaded)
            {                
                SMA1                = SMA(Close, Convert.ToInt32(Fsma));
                SMA2                = SMA(Close, Convert.ToInt32(Ssma));
            }
        }

        protected override void OnBarUpdate()
        {
            if (CurrentBars[0] < 1)
            return;

             // Set 1
            if (SMA1[0] > SMA2[0])
            {
                EnterLong(Convert.ToInt32(P01s), @"p01");
            }
            
             // Set 2
            if (SMA1[0] < SMA2[0])
            {
                EnterShort(Convert.ToInt32(P01s), @"p01");
            }
            
        }

        #region Properties
        [NinjaScriptProperty]
        [Range(1, int.MaxValue)]
        [Display(ResourceType = typeof(Custom.Resource), Name="Fsma",  Description="fast sma", Order=1,  GroupName="NinjaScriptStrategyParameters")]
        public int Fsma
        { get; set; }

        [NinjaScriptProperty]
        [Range(1, int.MaxValue)]
        [Display(ResourceType = typeof(Custom.Resource), Name="Ssma",  Description="slow sma", Order=2,  GroupName="NinjaScriptStrategyParameters")]
        public int Ssma
        { get; set; }

        [NinjaScriptProperty]
        [Range(1, int.MaxValue)]
        [Display(ResourceType = typeof(Custom.Resource), Name="P01s",  Description="position 01 size", Order=3,  GroupName="NinjaScriptStrategyParameters")]
        public int P01s
        { get; set; }

        [NinjaScriptProperty]
        [Range(1, int.MaxValue)]
        [Display(ResourceType = typeof(Custom.Resource), Name="Ss01",  Description="stop size 01", Order=4,  GroupName="NinjaScriptStrategyParameters")]
        public int Ss01
        { get; set; }
        #endregion

    }
}

//This namespace holds Strategies in this folder and is required. Do not change it. 
namespace NinjaTrader.NinjaScript.Strategies
{
    public class tesmacrossoverts : Strategy
    {
        private SMA SMA1;
        private SMA SMA2;

        protected override void OnStateChange()
        {
            if (State == State.SetDefaults)
            {
                Description                                    = @"trial error sma crossover trailing stop";
                Name                                        = "tesmacrossoverts";
                Calculate                                    = Calculate.OnBarClose;
                EntriesPerDirection                            = 1;
                EntryHandling                                = EntryHandling.UniqueEntries;
                IsExitOnSessionCloseStrategy                = true;
                ExitOnSessionCloseSeconds                    = 50;
                IsFillLimitOnTouch                            = false;
                MaximumBarsLookBack                            = MaximumBarsLookBack.TwoHundredFiftySix;
                OrderFillResolution                            = OrderFillResolution.Standard;
                Slippage                                    = 3;
                StartBehavior                                = StartBehavior.AdoptAccountPosition;
                TimeInForce                                    = TimeInForce.Gtc;
                TraceOrders                                    = false;
                RealtimeErrorHandling                        = RealtimeErrorHandling.StopCancelCloseIgnoreRejects;
                StopTargetHandling                            = StopTargetHandling.PerEntryExecution;
                BarsRequiredToTrade                            = 1000;
                // Disable this property for performance gains in Strategy Analyzer optimizations
                // See the Help Guide for additional information
                IsInstantiatedOnEachOptimizationIteration    = true;
                Fsma                    = 5;
                Ssma                    = 10;
                P01s                    = 1;
                Ss01                    = 8;
            }
            else if (State == State.Configure)
            {
                SetTrailStop(@"p01", CalculationMode.Ticks, Ss01, false);
            }
            else if (State == State.DataLoaded)
            {                
                SMA1                = SMA(Close, Convert.ToInt32(Fsma));
                SMA2                = SMA(Close, Convert.ToInt32(Ssma));
            }
        }

        protected override void OnBarUpdate()
        {
            if (CurrentBars[0] < 1)
            return;

             // Set 1
            if (SMA1[0] > SMA2[0])
            {
                EnterLong(Convert.ToInt32(P01s), @"p01");
            }
            
             // Set 2
            if (SMA1[0] < SMA2[0])
            {
                EnterShort(Convert.ToInt32(P01s), @"p01");
            }
            
        }

        #region Properties
        [NinjaScriptProperty]
        [Range(1, int.MaxValue)]
        [Display(ResourceType = typeof(Custom.Resource), Name="Fsma",  Description="fast sma", Order=1,  GroupName="NinjaScriptStrategyParameters")]
        public int Fsma
        { get; set; }

        [NinjaScriptProperty]
        [Range(1, int.MaxValue)]
        [Display(ResourceType = typeof(Custom.Resource), Name="Ssma",  Description="slow sma", Order=2,  GroupName="NinjaScriptStrategyParameters")]
        public int Ssma
        { get; set; }

        [NinjaScriptProperty]
        [Range(1, int.MaxValue)]
        [Display(ResourceType = typeof(Custom.Resource), Name="P01s",  Description="position 01 size", Order=3,  GroupName="NinjaScriptStrategyParameters")]
        public int P01s
        { get; set; }

        [NinjaScriptProperty]
        [Range(1, int.MaxValue)]
        [Display(ResourceType = typeof(Custom.Resource), Name="Ss01",  Description="stop size 01", Order=4,  GroupName="NinjaScriptStrategyParameters")]
        public int Ss01
        { get; set; }
        #endregion

    }
}

[rtwave]

i have carried out some more tests and the aberrant results when trying to optimize on non minute bar charts have been a lot more of the same.






i once again got blank results when trying to optimize the sample sma cross over strategy on renko 50 tick, volume 300,000 and tick 300,000 bars. as in all the cases i have reported previously i tried to optimize for the period from january 2008 to december 2014, for maximum net profit, with the genetic optimizer over 100 generations with 100 generation size. for both sma periods the parameters were optimized over this range (3;100;2).


the composite image shows that the optimization did not work but if i apply the same sample sma cross over strategy to charts whose specifications are identical to what i'm trying to optimize on the strategy will execute without any issues. renko bar charts look very promising, i really hope these extensive malfunctions can be solved promptly.

[NinjaTrader_PatrickH]

Hello rtwave,

Thank you for your response.

I see the Optimization is being set to January 1, 2008 to December 31, 2014 and each of these Bars Types are tick based.

Who do you connect to for your historical tick data?

Please go to the NinjaTrader Control Center > Tools > Historical Data > expand (+) Historical > and then expand the 08 through 14 contracts for CL and advise if you see Tick data for these contracts and the dates you wish to test.

I look forward to your response.

[rtwave]

Patrick,


thanks.


right now i'm using a cqg demo connection.


i went to the historical data tab as you requested and it seems like only minute and day data is available.





all the other months look the same, tick data is only available from 04-2017 to date.

[NinjaTrader_PatrickH]

Hello rtwave,

Thank you for your patience.

Without the historical tick data the Range bars could not be built that far back and thus the results would return as 0.

Your suggestion for an option to run one optimization after another has been assigned the internal tracking id of SFT-1047.

Your suggestions for an option to set templates and copy parameters from Optimization results to new strategies without manually plugging in the values has been assigned the internal tracking id of SFT-1196.

Please let me know if you have any questions.

[rtwave]

Patrick,


it's good to read that nt will work on improving the functionality of the optimization engine. at this moment it is just ridiculously limited. in nt 7 it was possible to save individual optimization results but in nt 8 not even such a simple task can be accomplished. so far i have been testing strategies with 3 parameters at most, but i have others will a far larger number of parameters and it would be a hassle to have to create screengrabs for each individual optimization process and input everything by hand. i just hope this won't take years and until nt 9 is released to be solved.


now, getting back to the main issue i created this thread about, you correctly identified the issue and we fully agree that it is the lack of tick data that is makes it impossible to optimize over non minute bar types. the questions now are, ¿does ninjatrader only provide 12 months of tick data for all symbols? ¿where could i get historical tick data compatible with the nt platform?


thanks.

[NinjaTrader_PatrickH]

Hello rtwave,

Thank you for your response.

NinjaTrader supplies historical tick data back 360 days. Kinetick supplies the historical tick data back 180 days. I am not aware of any resources for tick data that would go back further in time.

Please let me know if you have any questions.

[ashmanz80]

Hi Patrick,

I have a connection to Kinetick. What did you mean when you said "Ninja Trader" can supply tick data going back 360 days? Did you mean if we request it?

Thanks in advance

[NinjaTrader_PatrickH]

Hello ashmanz80,

Thank you for your response.

NinjaTrader historical data servers are used in certain connections such as the NinjaTrader Continuum connection. This would be what I meant by NinjaTrader supplying the data.

Please let me know if you have any questions.