Fail of epic proportions!!

[dmking]

The experiences of this past few days has caused me to assess the situation with regard to Zen-Fire, Mirus and the forced upgrade to Ninja v18.

As with many of you, I trade for a living and regard myself as a "professional" retail trader. We rely on a fast, reliable and accurate data. We require an easy to use and flexible analysis platform. We require an easy to use and flexible order entry platform. And we require reasonable/low fees and commissions.

I'm not sure who's culpable here, but it looks like Zenfire is the main suspect. I know that when I signed up with Mirus one of the main factors I considered was the "free" real-time data provided by Zen-fire. At the time Zen was highly rated because of their reliability, accuracy and low latency ( at least for us retail traders).

Mirus offered competitive rates and access to the Ninja platform.

Ninja has considerably improved over the years and now offers a great set of features and functions and I am loathe to walk away from it.

In a former life I was responsible for the roll out of many mission critical enterprise and retail applications and have seen a lot and have dealt closely with customers and several levels and have never seen anything like this. I would never have let the Zen-Fire or Ninja upgrades into the wild without EXTENSIVE TESTING. Now, that may have happened, but this really seems like an alpha, not even beta, test program being passed off as a production release.

It seems like significant changes were made to Zen which had a huge impact on Ninja. Give that, it would have been prudent for Zen and Ninja to conduct more months of testing and integration than they obviously have to date, before forcing us to roll. I would have been fired if I had let any of my releases go out with this result.

Not that we will ever know, but I would like to know:
- How much testing Zen did before going LIVE?
- How much (and for how long) INTEGRATION testing was done between Zen and the Ninja platform before the December 29th mandatory upgrade?
-Why there was the rush to get an obviously under tested integration out the door?

It looks to me like Ninja and Mirus are collateral damage from a poorly planned and executed Zen-Fire rollout. Not being on the inside, but based on experience this is what it seems like.

There are things that we have in our power to control. I'm not ready to throw Ninja and Mirus over the side, but this has caused me to reassess what I need to do to conduct my business in an efficient and reliable manner. Doing something for free should not take precedence over doing business reliably and efficiently.

1. DTNIQ third party data feed work with Ninja and is excellent. It has a cost, but check it out.

2. eSignal Premier is a very good analysis/charting platform and the data feed is very good as well. There are costs associated here as well.

3. Multi-charts is a good platform.

4. For order execution I use ThinkorSwim. It's good and free from TD Ameritrade if you have an account with them.

5. TT X-Trader is excellent, but costly and would require a move away from Mirus.

The take away's here are to:

- Have a redundant, high speed and reliable data-feed - DAMN THE COST.
- Have a backup trading platform.
- There is a cost of doing business. Don't be "pennywise and pound foolish".

I made changes prior to the open today. I saw this train-wreck coming on the 29th and planned accordingly. Ask yourself this, "Did I miss the trade at 1822 on the ES, because I was afraid to trade or because all I got was "Connecting to Zen-Fire>."

The support guy/gals at Ninja are doing the best that they can, but you can't push on a rope.

[Trader_55]

I don't know the exact details of what happened, but it seems as though Rithmic made the decision to cut off Zen-Fire from licensing, essentially forcing Zen to find an alternative data feed. It was possible that this happened quickly enough that there wasn't enough time to do proper testing. I have a feeling that the brokers (Mirus) decided to keep relatively quiet about this and hope that everything went smoothly, a strategy which appears to have blown up in their face. If the cut off of the Rithmic feed was thrust upon Zen without much time to react then that is understandable, but trying to rush out an alpha release like this was a very bad idea, especially without adequate warning to their customers. The scary part? They may not be able to fix these problems easily. Who knows what the issues are. Who knows if even once it is working, this type of thing isn't going to be an intermittent problem. They obviously did very little testing, so even if it gets "fixed" for next week, that doesn't guarantee anything. That's why I'm leaving, I can't rely on their data feed any more. They should have had a clause in their contract with Rithmic which said that they needed more notice before getting cut off, if in fact that is what happened.

[Instamess]

Not that we will ever know, but I would like to know:
- How much testing Zen did before going LIVE?
- How much (and for how long) INTEGRATION testing was done between Zen and the Ninja platform before the December 29th mandatory upgrade?
-Why there was the rush to get an obviously under tested integration out the door?

I think I know the answer to the last two of your questions.

As far as testing goes, Ninja has already said they did their "standard" testing i.e. they were satisfied that Release 18 was customer ready. To that end they are culpable - it is obviously not customer ready and NinjaTrader screwed up here (in addition to ZenFire's blame for the shoddy release).

It was rushed because, as the rumours go, ZenFire was given the boot by Rithmic.

With respect to your first question about how much testing ZenFire did, we'll never know as ZenFire refuses to comment or engage.

[dmking]

I don't know the exact details of what happened, but it seems as though Rithmic made the decision to cut off Zen-Fire from licensing, essentially forcing Zen to find an alternative data feed. It was possible that this happened quickly enough that there wasn't enough time to do proper testing. I have a feeling that the brokers (Mirus) decided to keep relatively quiet about this and hope that everything went smoothly, a strategy which appears to have blown up in their face. If the cut off of the Rithmic feed was thrust upon Zen without much time to react then that is understandable, but trying to rush out an alpha release like this was a very bad idea, especially without adequate warning to their customers. The scary part? They may not be able to fix these problems easily. Who knows what the issues are. Who knows if even once it is working, this type of thing isn't going to be an intermittent problem. They obviously did very little testing, so even if it gets "fixed" for next week, that doesn't guarantee anything. That's why I'm leaving, I can't rely on their data feed any more. They should have had a clause in their contract with Rithmic which said that they needed more notice before getting cut off, if in fact that is what happened.

Hi Trader-55,

What you say may be true. But I find it inconceivable that Zen was caught unaware by actions that Rithmic was taking. This was a business relationship and contracts were probably in place. Zen should have seen this coming from a long way off.

If Mirus was aware of the issues with Rithmic and Zen and didn't say anything to us then shame on them, and I can't view them as a reliable business partner.

This whole thing smells bad all the way down the line. Zen, Ninja and Mirus had to have known a long time ago and failed to notify their clients and failed to plan a proper transition/integration of a new data feed. We have choices, and my guess is that many of us will vote with our feet.

[koganam]

I think I know the answer to the last two of your questions.

As far as testing goes, Ninja has already said they did their "standard" testing i.e. they were satisfied that Release 18 was customer ready. To that end they are culpable - it is obviously not customer ready and NinjaTrader screwed up here (in addition to ZenFire's blame for the shoddy release).

Hm. I fail to see how NT is at all culpable here. NT tested their program with all then existing connectors and passed it. After roll out, ONE and ONLY ONE feed/brokerage combo had a problem. I think it is clear who pooped the bed. NT is working just fine: I have made multiple trades with it: just using different feeds and brokers.

[Instamess]

Hm. I fail to see how NT is at all culpable here. NT tested their program with all then existing connectors and passed it. After roll out, ONE and ONLY ONE feed/brokerage combo had a problem.

So maybe the testing with that feed brokerage combo was deficient. NT and ZF are closely associated. NT knew that ZF was under pressure and had to launch Release 18 while the markets were quiet. I'm not saying they took shortcuts to help Mirus out, but they did have to approve R18 before it was foisted on us, and they did.

NT's support has always been great. That they provide support for ZF and answer ZF related queries is also appreciated. But that also provides protection for ZF and saves Mirus having to come to the forum and field questions. That's very convenient. NT represent but don't represent Mirus. It's a one way conversation - bits are dribbled to us when it suits Mirus but we can't speak directly with them - NT's involvement as a one way valve middleman prevents it. I find this frustrating.

[stocktraderbmp]

I don't know the exact details of what happened, but it seems as though Rithmic made the decision to cut off Zen-Fire from licensing, essentially forcing Zen to find an alternative data feed. It was possible that this happened quickly enough that there wasn't enough time to do proper testing. I have a feeling that the brokers (Mirus) decided to keep relatively quiet about this and hope that everything went smoothly, a strategy which appears to have blown up in their face. If the cut off of the Rithmic feed was thrust upon Zen without much time to react then that is understandable, but trying to rush out an alpha release like this was a very bad idea, especially without adequate warning to their customers. The scary part? They may not be able to fix these problems easily. Who knows what the issues are. Who knows if even once it is working, this type of thing isn't going to be an intermittent problem. They obviously did very little testing, so even if it gets "fixed" for next week, that doesn't guarantee anything. That's why I'm leaving, I can't rely on their data feed any more. They should have had a clause in their contract with Rithmic which said that they needed more notice before getting cut off, if in fact that is what happened.

http://www.ninjatrader.com/support/f...ad.php?t=62159 - I posted this in November - I can't say for definite but you may have hit the nail on the head - my suspicion is that they have called the 'BigTick' data feed the new 'ZenFire' when the take up on MirusTrader and BigTick was so poor. Incidently try googling 7Ticks and a guy called Kevin Bohren.
Also interesting that if you did not update by the last day of 2013 then you would be unable to access your account - adding to the theory that the contract between Rithmic and Mirus was revoked or at least not renewed.

[koganam]

...That they provide support for ZF and answer ZF related queries is also appreciated. But that also provides protection for ZF and saves Mirus having to come to the forum and field questions. That's very convenient. NT represent but don't represent Mirus. It's a one way conversation - bits are dribbled to us when it suits Mirus but we can't speak directly with them - NT's involvement as a one way valve middleman prevents it. I find this frustrating.

I agree 110%. However, that was not really what I was addressing. That NT has had to step into the breach because Mirus chooses to be remiss is so regrettable as to border on despicable.

I was just saying that NT 7.0.1000.18 seems to have no inherent issues, so we should not be blaming them for what their DataFeed/Broker partner does. That blame should be placed squarely where it belongs.

[Trader_55]

http://www.ninjatrader.com/support/f...ad.php?t=62159 - I posted this in November - I can't say for definite but you may have hit the nail on the head - my suspicion is that they have called the 'BigTick' data feed the new 'ZenFire' when the take up on MirusTrader and BigTick was so poor. Incidently try googling 7Ticks and a guy called Kevin Bohren.
Also interesting that if you did not update by the last day of 2013 then you would be unable to access your account - adding to the theory that the contract between Rithmic and Mirus was revoked or at least not renewed.

Thanks... looks like the BigTick data feed has been having problems for some time... definitely wouldn't trade with it now. The fact that they are remaining silent throughout this entire ordeal is telling me that they don't really know what's wrong, which is why they are trying to duck and cover. I wouldn't be surprised if there were still problems next week, but I guess we'll see.

[Instamess]

I agree 110%. However...

I was just saying that NT 7.0.1000.18 seems to have no inherent issues, so we should not be blaming them for what their DataFeed/Broker partner does. That blame should be placed squarely where it belongs.

To what extent is NT complicit? I don't know. When a question was asked the NT CEO gave a guarded reply that they did their "standard" tests before release. That doesn't mean much. Perhaps the tests did show some flaws and they approved the release nonetheless. Perhaps there are different versions of "standard" tests. Perhaps NT did suspect there would be problems and decided it was okay to beta test on live users because it was the Christmas holidays and they needed to help Mirus out of their predicament.

Or perhaps none of the above.

Ray, the NT CEO, reads all these posts. If he comes out and says that Mirus is 100% to blame and that there's nothing NT could have done to foresee any of the problems that occurred, I'll take back what I said about NT being partly at fault here. Further, I'll issue an apology for suspecting it in the first place.

I agree 110%. However, that was not really what I was addressing. That NT has had to step into the breach because Mirus chooses to be remiss is so regrettable as to border on despicable.

Stepped into the breach? I don't see it so much that way. As far as this forum is concerned they didn't step into any breach as much as they erected a fence to protect Mirus from the full fury of angry users.

[koganam]

...
Stepped into the breach? I don't see it so much that way. As far as this forum is concerned they didn't step into any breach as much as they erected a fence to protect Mirus from the full fury of angry users.

I do not think that NT is capable of protecting any entity other than itself. This is a pretty straight-forward situation. Your broker/datafeed vendor refuses to talk to you. NT's CEO goes out of his way to get Mirus to speak to him, and relays what they say. That is not a shield of Mirus: it is NT stepping in to get you information when the primary culprit was hiding. NT's only real responsibility here was to have told everybody that NT is working correctly, and shut up.

It is a saving grace that NT will even try to help at all, and it speaks to how much they care about their customers, no matter how sucky the NT partners with whom the NT customers choose to associate. Rather than blaming NT for trying to help when your broker ignored you, you might be better served by finding a broker who treats you with care and respect.

I am done.

[bce111]

I'm going to try to avoid wasting much time in these forums today as I've already wasted so much time this week. I'll check in later. I'm going to try and trade as that's what I need to do. This is of course with dire warnings that we shouldn't trade today.

This whole episode is such a crock. I was especially irritated to read the lame message from the Mirus CEO about, "Many of those we have spoken with have been extremely understanding of these technical issues" What a crock. That's only in your dreams. It's called spin.

The more I thought about this last night the more I wondered if this dysfunction was a known element before it even began. In the sense of knowing they had to do this conversion, it would cause massive disruptions and they just decided there was no way to avoid it and needed to bite the bullet and go through with it. Maybe they chose this week figuring it was a holiday week. They perhaps knew there would be fallout and even some lost customers and would manage that the best they could. Perhaps Ninja knew this was going to go down too.

Off to trade or try to even with a bad connection that could hurt me. Don't bother warning me Ninja, I've read it. The parties involved have put me in this situation.

I'll be back later to respond to comments. Have a good day traders.