• If this is your first visit, you will have to register before you can post. To view messages, please scroll below and select the forum that you would like to visits. Questions? Be sure to check out the Forum FAQ.

Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Critical debugging: Account Position in live trading NT 7.0.1000.39

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Critical debugging: Account Position in live trading NT 7.0.1000.39


    In countless posts of the past, you've always said that NT is like a beacon of all info coming from the broker's account, so when I have a Strategy working in real-time-live trading, and I need to check for Market.Position or Position.Quantity actually from my broker's account only.

    So, I have this situation, very easy to understand. I have a simple Strategy that before doing anything, it needs to check Market.Position and Quantity.Position, so it will know if it has to close an opened position, opening a new position or doing nothing with the right accordingly orders. So following with the example, if I had an opened position and my Strategy detects a signal that indicates to reverse that position, first, the Strategy will check for actual position and submits an order to Flatten it, once it makes sure that it's actually flat by checking the new status info, it will submit a new order to open the desired new active position. Obviously, the info that the Strategy needs to know before acting, will be "always and all time" the info coming from my real-live-trading broker's account and NOT some temporary internal NT's strategy position. Let me explain it better with my real trading situation from last night so you know why this so critical for some of us:

    I have a simple account with Interactive Brokers, and I need to meet right margin requirements constantly in the instruments that I work with, that's why my Strategy works as it stated above.
    Last night, I started my Strategy, and in few moments it opened the first position with a simple trade, so far so good. Then, as expected, the Strategy detects a signal that indicates to reverse the actual position. So, once again, I repeat: the Strategy first checks the actual position ( Market.Position and Quantity.Position ) and submits a simple order to flat the actual Position. When this is done and checked, by verifying the Market.Position and Quantity.Position coming supposedly from "my broker's account", then the Strategy would submit a new order for the new desired position, pretty straightforward. Well, last night, I got eventually several margin warnings and rejected orders cause margin was not met. Since the Strategy always works with the right quantity to meet margins, I decided to investigate and I found that the problem was the Strategy was submitting the second order to open a new position WHILST Interactive Brokers still has not been yet updated with the actual and right Market.Position and Quantity.Position, so How was this possible??? Well, obviously the Strategy is getting some temporary-internal Strategy position from NT instead of my broker's account as it should be. Because, eventually when the broker's account is updated, the order is processed just fine.

    Having explained the issue, is there any way to check ONLY my broker's account info OR this is a critical bug that you need to fix immediately?

    Will I have to implement some snippet ( like below ) cause NT is not working properly with the info coming from the broker's account right away?

    Code:
    foreach (Account acct in Cbi.Globals.Accounts)                
                    {                
                        if ( acct.Name == "XXXXXXX")            
                        {            
                            PositionCollection positions = acct.Positions;        
                            foreach (Position pos in positions)        
                            {        
                                if (pos.Instrument.FullName == "ES XXXXXX")    
                                {    
                                    if ( Position.MarketPosition != pos.MarketPosition || Position.Quantity != pos.Quantity )
                                    {
                                        // Do something
                                    }
                                }    
                            }        
                        }            
                    }


    Looking forward to this urgent requirement. I have a lot of pressure on me and this kind of issues makes me unhappy. Please help.
    Thanks




    Last edited by pstrusi; 10-30-2018, 02:32 AM.

    #2
    Hello pstrusi,

    The strategy position and account position are separate. If these start out of sync (without the synchronize start behavior), they will never be in sync unless you place a manual to sync them.

    (edited after I noticed your inquiry was for NT7)

    Unfortunately, with NinjaTrader 7, your options are limited.

    You would need to use the Synchronize with either Wait until flat or Immediately submit, or you would need to manually place orders to synchronize the account to match the strategy.


    However, with NinjaTrader 8 there is increase functionality to handle these situations:

    If you want a strategy to start with the account position, use the AdoptAccountPosition start behavior.

    Below is a public link to the help guide.
    https://ninjatrader.com/support/help...rtbehavior.htm

    And a link to a video that demonstrates the start behaviors available.
    https://www.screencast.com/t/D2GRrGe5MfzJ

    If you want to only rely on the account position and ignore the strategy position, maybe it would be better to use the addon approach and not use a NinjaScript Strategy.
    Last edited by NinjaTrader_ChelseaB; 10-30-2018, 06:22 AM.
    Chelsea B.NinjaTrader Customer Service

    Comment


      #3
      Hello pstrusi,

      I am responding to your comments:

      I have a simple Strategy that before doing anything, it needs to check Market.Position and Quantity.Position, so it will know if it has to close an opened position, opening a new position or doing nothing with the right accordingly orders.
      if I had an opened position and my Strategy detects a signal that indicates to reverse that position, first, the Strategy will check for actual position and submits an order to Flatten it, once it makes sure that it's actually flat by checking the new status info, it will submit a new order to open the desired new active position. Obviously, the info that the Strategy needs to know before acting, will be "always and all time" the info coming from my real-live-trading broker's account and NOT some temporary internal NT's strategy position.
      So, once again, I repeat: the Strategy first checks the actual position ( Market.Position and Quantity.Position ) and submits a simple order to flat the actual Position.
      To confirm, you are not trying to do this when the strategy is started, but instead you are trying to do this after the strategy has been trading for a while?

      In other words, you are trying to work with orders that are coming from another source in real-time and not when when strategy is started, is this correct?

      Without using example situations or analogies, can you state using few words what you are trying to do?

      Are you trying to force the Strategy Position to change without using an order?

      (As a heads up, querying the account information directly is not documented or supported for NinjaTrader 7 but was expanded on and documented in NinjaTrader 8)
      Last edited by NinjaTrader_ChelseaB; 10-30-2018, 07:13 AM.
      Chelsea B.NinjaTrader Customer Service

      Comment


        #4
        Originally posted by pstrusi View Post
        In countless posts of the past, you've always said that NT is like a beacon of all info coming from the broker's account, so when I have a Strategy working in real-time-live trading, and I need to check for Market.Position or Position.Quantity actually from my broker's account only.

        So, I have this situation, very easy to understand. I have a simple Strategy that before doing anything, it needs to check Market.Position and Quantity.Position, so it will know if it has to close an opened position, opening a new position or doing nothing with the right accordingly orders. So following with the example, if I had an opened position and my Strategy detects a signal that indicates to reverse that position, first, the Strategy will check for actual position and submits an order to Flatten it, once it makes sure that it's actually flat by checking the new status info, it will submit a new order to open the desired new active position. Obviously, the info that the Strategy needs to know before acting, will be "always and all time" the info coming from my real-live-trading broker's account and NOT some temporary internal NT's strategy position. Let me explain it better with my real trading situation from last night so you know why this so critical for some of us:

        I have a simple account with Interactive Brokers, and I need to meet right margin requirements constantly in the instruments that I work with, that's why my Strategy works as it stated above.
        Last night, I started my Strategy, and in few moments it opened the first position with a simple trade, so far so good. Then, as expected, the Strategy detects a signal that indicates to reverse the actual position. So, once again, I repeat: the Strategy first checks the actual position ( Market.Position and Quantity.Position ) and submits a simple order to flat the actual Position. When this is done and checked, by verifying the Market.Position and Quantity.Position coming supposedly from "my broker's account", then the Strategy would submit a new order for the new desired position, pretty straightforward. Well, last night, I got eventually several margin warnings and rejected orders cause margin was not met. Since the Strategy always works with the right quantity to meet margins, I decided to investigate and I found that the problem was the Strategy was submitting the second order to open a new position WHILST Interactive Brokers still has not been yet updated with the actual and right Market.Position and Quantity.Position, so How was this possible??? Well, obviously the Strategy is getting some temporary-internal Strategy position from NT instead of my broker's account as it should be. Because, eventually when the broker's account is updated, the order is processed just fine.

        Having explained the issue, is there any way to check ONLY my broker's account info OR this is a critical bug that you need to fix immediately?

        Will I have to implement some snippet ( like below ) cause NT is not working properly with the info coming from the broker's account right away?

        Code:
        foreach (Account acct in Cbi.Globals.Accounts)
        {
        if ( acct.Name == "XXXXXXX")
        {
        PositionCollection positions = acct.Positions;
        foreach (Position pos in positions)
        {
        if (pos.Instrument.FullName == "ES XXXXXX")
        {
        if ( Position.MarketPosition != pos.MarketPosition || Position.Quantity != pos.Quantity )
        {
        // Do something
        }
        }
        }
        }
        }


        Looking forward to this urgent requirement. I have a lot of pressure on me and this kind of issues makes me unhappy. Please help.
        Thanks



        You are confusing NinjaTrader operations with Broker operations. In a really slow market, you would probably not have the issue. In a fast market, doing things the way that you have written, would expose this issue.

        You have to think of what an actual trade operation is like; not what position NinjaTrader is processing. To avoid this issue, you must use OnExecution(), to check for the execution being filled. That is when the broker has reported its execution of the trade, and NT has finalized its processing of the state of the account.

        In other words, before you start looking at what NT has recorded, and is processing as your position, first make sure that the broker has reported the 'fill/completion of processing' of orders to NT.
        Last edited by koganam; 10-30-2018, 09:42 PM.

        Comment


          #5
          Originally posted by pstrusi
          Hi Koganam, thanks for answering.

          What you state at the last " first make sure that the broker has reported the 'fill/completion of processing' of orders to NT. " is exactly what I informed in the post. I'm not confusing operations of NT with the broker, on the contrary: I'm aware that the problem is just that, I need to pull the market position info actually from the broker and not NT.

          Let me tell you some interesting details after searching the log, so you can see the issue is not solved by check executions, you just have to wait for the info coming from your broker.

          - I submitted the order,
          - The order was filled and makes my position flat.
          - IB records the transaction BUT still, my actual position is not updated at the broker. On the other hand, in NT my actual position is exactly right.
          - So, when my Strategy at NT is right and ready to submit a new order, yet the flat position is not reported from IB.
          - My order is submitted but for Margin requirements, it's rejected, logically.
          - Eventually, IB sends my actual position and then the new order is processed just fine.

          I think I did some workaround that might work better.

          Thanks again
          Regards
          If the broker has not yet reported the fill to NT, any actions that you take in NT based on the fill of the order can cause you problems. YES, it is that simple. Get the response from the broker before you do anything else based on position. The broker is going to act based on THEIR position. If they have not reported an execution yet to NT, any order that you make will be processed based on what the broker is recording as your position. NT position is not relevant until the broker reports back the results of any operation that NT has ordered to the broker.

          Instead of indulging in theoretical arguments as to what the broker should do, you will be better off tracking what the broker actually does, and acting on that. Have you even bothered to check if the broker has reported the execution, or are you just going to continue to argue with me that you do not need to do so?

          Here is your situation:
          1. You send an order to the broker.
          2. You do not know what the situation is with the broker, because the broker has not yet informed you of it.
          3. You decide that you will send another order, even though you have no information yet about what the broker has done.
          4. The broker may reject the order, because having no knowledge of what the broker has done, you cannot know what the broker has done yet.
          5. You want to fault NT when the broker rejects your order, which as far as the broker is concerned is invalid.
          6. I ask you to first check what the broker has done, before you send the next order.
          7. You refuse to do so, telling us that that is unnecessary.
          Does that make any sense? The computer really has no care or interest in what you think. As a programmer, it is your duty to provide the correct instructions to the computer. The program will process data as instructed, not as you wish.
          Last edited by koganam; 11-01-2018, 11:48 AM.

          Comment


            #6
            Originally posted by pstrusi
            NT Support team:

            When you're working under live trading with your real broker's account, it's important to have updated the broker's position info as soon as possible in order to know if you can proceed with further instructions. I always thought that this info would be updated fast and be available to be pulled from "OnPositionUpdate" while you have synced accounts, but I've realized that after an execution is recorded, NT is the first to update Strategy Position, then some milliseconds later, the broker will update the account position. So, in live trading, if you have a fast Strategy that needs to know Position Update to act, it will wrongly read NT Strategy Position ( which still lacks for the broker update ).

            According to Koganam, the only way to pull the last broker's account update info is programming some logic procedure using the method OnExecution() , which unfortunately is not stated clearly in the guide what it's really about it. Now I understand why you suggest to use OnExecution() for protective orders and not only OnOrderUpdate . This info is so critical that it should be changed in the guide and not a confusing explanation telling is only about an internal sequence of events. With the contribution of everyone is always possible to get a better product.

            Regards
            I do not disagree with you that they could elaborate on it a bit, but it is still, already, pretty clearly stated.

            ref: https://ninjatrader.com/support/help...and_onexec.htm

            Comment


              #7
              Originally posted by pstrusi
              Then


              Already a pretty clearly stated you say, ah ? hehehe this is just funny. This little extract is from NT7 guide:
              · CRITICAL: If you want to drive your strategy logic based on order fills you must use OnExecution() instead of OnOrderUpdate(). OnExecution() is always triggered AFTER OnOrderUpdate(). There is internal strategy logic that is triggered after OnOrderUpdate() is called but before OnExecution() that can adversely affect your strategy if you are relying on tracking fills within OnOrderUpdate().
              · As an example, the NinjaTrader core may have received "Working" and then "PartFilled" order state change events back from the broker API on thread "B" and at some point in time (MILLISECONDS LATER) the NinjaTrader core will take these events and trigger the OnOrderUpdate() method in the strategy on thread "A". Thus, when the strategy receives the first "Working" state for an order, the IOrder object passed in will reflect the "Working" state although the actual order is really in a state of "Part Filled" which is truly reflected in the original IOrder object returned in any of the order methods such as EnterLong(). Of course, the OnOrderUpdate() method will subsequently receive the event for "PartFilled" state.
              So in the first paragraph is told that OnExecution() is ALWAYS triggered AFTER OnOrderUpdate(), this is right and you were right about that this info is actually from the broker.
              Then in the second paragraph tells you that thread "B" that updates from the broker ( that you can check with OnExecution() ) is updated BEFORE than OnOrderUpdate, because, later it states:
              Of course, the OnOrderUpdate() method will subsequently receive the event for "PartFilled" state.

              There's an obvious contradiction about timing.


              Since I like to share when something that was driving me nuts, is solved, here what I did:

              If you want to keep your Strategy "healthy" go for OnOrderUpdate for mostly all updates and only use OnExecution() as a critical flag that comes really from the broker, that indicates some state that your Strategy waits before doing anything.

              Regards
              There is no contradiction: it is just unclear writing, not differentiating properly between how the events are occurring in a multi-threaded environment; especially and specifically when orders are only part-filled. You are still pretty clearly told to use OnExecution() rather than OnOrderUpdate() if the situation requires knowledge of broker fills.

              In any case, I use all 3: OnExecution(), OnOrderUpdate() and OnPositionUpdate() for order processing, gating and tracking. The 3 handlers have different purposes, but I always will act on new orders only after the broker has informed me of what happened to my last order. It is a matter of prudence. I see no reason ever to act before I know what the broker has done with my last order, even if I just want to cancel it.
              Last edited by koganam; 11-02-2018, 10:06 AM.

              Comment


                #8
                Originally posted by pstrusi
                Now markets are opened, I'm sorry to confirm that OnExecution() only gets the execution info and this doesn't imply that at that time the broker's account has already updated its position, not at all. I'm going to show with a simple sequence exactly what info I need:

                ORDER GETS FILLED
                11/4/2018 6:22:40 PM|1|32|Order='2062615117/xxxxxxx' Name='CLOSE SHORT' New state=Filled Instrument='ES 12-18' Action=Buy Limit price=2718......................................

                EXECUTION CONFIRMATION
                11/4/2018 6:22:40 PM|1|16|Execution='ES 12-18/0000fe5f.5bdf7a54.01.01' Instrument='ES 12-18' Account='xxxxxxx' Exchange=Globex Price=2718 Quantity=X Market position=Long....

                SINCE METHOD OnExecution() received the confirmation ( known as coming from the broker ) my Strategy thinks its position is flat so it submits an order to open Long
                11/4/2018 6:22:40 PM|1|32|Order='f7bd3bbe34384e198e381bed2e461f9d/XXXXXX' Name='SET LONG' New state=PendingSubmit Instrument='ES 12-18' Action=Buy Limit price=2717.75

                ORDER ACEPTED
                11/4/2018 6:22:40 PM|1|32|Order='f7bd3bbe34384e198e381bed2e461f9d/XXXXXX' Name='SET LONG' New state=Accepted Instrument='ES 12-18'

                ORDER REJECTED BY THE BROKER CAUSE THE BROKER'S ACCOUNT POSITION IS NOT FLAT
                11/4/2018 6:22:40 PM|1|32|Order='f7bd3bbe34384e198e381bed2e461f9d/xxxxxxxx' Name='SET LONG' New state=Rejected Instrument='ES 12-18' Action=Buy Limit price=2717.75
                Native error='Order rejected - reason:YOUR ORDER IS NOT ACCEPTED. IN ORDER TO OBTAIN THE DESIRED POSITION YOUR NET LIQ [xxxxx USD] MUST EXCEED THE MARGIN REQ [yyyyyy USD] (201)'

                THEN.....WHEN FINALLY THIS INFO ARRIVES, IT STATES THAT BROKER'S POSITION IS ALREADY FLAT AND THE NEW ORDER IS NOW PROCESSED FINE
                11/4/2018 6:22:41 PM|1|64|Instrument='ES 12-18' Account='xxxxxx' Avg price=0 Quantity=0 Market position=Long Operation=Remove Currency=UsDollar

                This last line above got from the log file, is the info that I need to get to be used by the Strategy, but HOW ??????

                I've tried unsuccessfully several ways. OnPositionUpdate, combinations of market.position, ...etc but all of them gets me a false Flat position from the broker.

                I hope you have the answer
                Thanks


                P
                Since some of your information is masked, I find it hard to say anything specific. However, in the first instance, the total of your messages is implying that you have exceeded the margin limits for the position.

                What kind of order are you using for "CLOSE SHORT"? A limit buy order? That is what information that you have not masked about it seems to be saying. Masking the account number makes absolute sense. Why are you masking position quantities and identities? In order to understand your flow, all the data must be seen. (The account info is not germane to the order flow, and should correctly be masked). Nothing else needs to be masked.

                Comment


                  #9
                  Hello pstrusi,

                  Thank you for your patience and additional information.

                  I would like to look into this matter further with your Log and Trace files.

                  Please send me your log and trace files so that I may look into what occurred.

                  You can do this by going to the Control Center-> Help-> Mail to Platform Support

                  Ensuring 'Log and Trace Files' is checked will include these files. This is checked by default.

                  Please list 'ATTN: Patrick H' in the subject line and reference this thread in the body of the email.

                  I look forward to assisting you further.
                  Patrick H.NinjaTrader Customer Service

                  Comment


                    #10
                    Hi Patrick, thanks for your attention and assistant offering.
                    By now, I think that it was possible for me to create a workaround based on what I said recently, working with PositionUpdate in an exact state.
                    If I find that I need further help, I'll let you know here.
                    Thanks

                    Comment


                      #11
                      As in many others issues, generally, the use of Flags to make sure that the Strategy has reached a certain state seems to solve problems that are impossible otherwise. I create a workaround that uses flags withing OnPositionUpdate so the Strategy acts over checked info. No need of fancy commands or complicated debugging. Simplicity is generally an answer to many problems.

                      Comment


                        #12
                        Originally posted by pstrusi


                        Koganam you're focusing in the most irrelevant part of the post and it draws my attention, having in mind that you're a very knowlegdable in NT, that masked info, really ? .

                        Since the first line of this post I've always said that I'm having an issue of margin requirement cause the Strategy reads a "Flat position" whilst the Broker's position is not yet updated to Flat. This occurs just for a few milliseconds but good enough to cause a problem. However, there's a key part; the last line of the log provides that real info that solves the issue:
                        • 11/4/2018 6:22:41 PM|1|64|Instrument='ES 12-18' Account='xxxxxx' Avg price=0 Quantity=0 Market position=Long Operation=Remove Currency=UsDollar
                        As you can see it reads: Quantity=0 Market position=Long, so I suspect that a possible workaround would be dealing with OnPositionUpdate and detecting this exact state within a logic sequence.

                        Regards

                        I can only understand what I see if I can see all the relevant information. That is a great part of your problem. You ask for help, then think that you can decide what is relevant and what is not. It does not work that way. I treat every problem as if it were occurring to me, which means that I need to see all the process flow information, not just what you deem important. I saw the very line that you quote here, and as it did not make sense to me, given what else was posted, I asked you to let me see the other information.

                        I really do not think that your telling me what you think is the most important is appropriate behavior when you ask for help. If you ask for help, let the helper help you, not have you deciding what the helper needs to know in order to help resolve the problem. Apropos of which, as you are the one who thinks that you know everything that is relevant, why are you asking for help in the first place? If you know it all, why bother?

                        Comment

                        Latest Posts

                        Collapse

                        Topics Statistics Last Post
                        Started by YevhenShynkarenko, Today, 02:55 AM
                        0 responses
                        14 views
                        0 likes
                        Last Post YevhenShynkarenko  
                        Started by pjsmith, Today, 02:20 AM
                        0 responses
                        5 views
                        0 likes
                        Last Post pjsmith
                        by pjsmith
                         
                        Started by faisamtrader, Yesterday, 10:03 PM
                        0 responses
                        7 views
                        0 likes
                        Last Post faisamtrader  
                        Started by johnnybegoode, Yesterday, 08:11 PM
                        0 responses
                        4 views
                        0 likes
                        Last Post johnnybegoode  
                        Started by Bionian, Yesterday, 02:47 PM
                        2 responses
                        25 views
                        0 likes
                        Last Post Bionian
                        by Bionian
                         
                        Working...
                        X