NT can't access incoming OIF files

[maskiepop]

Hi.

I am generating orders externally and passing it to NT via OIF files. I generate a lot of orders during fast moving markets, so I write out each order to a separate OIF file, i.e OIF00001.txt, OIF00002.txt, OIF00003.txt, ... and so on. I close the file as soon as I have written out the an OIF record.

My problem is that I am getting a lot of read/delete errors from NT regarding the OIF files. A few orders does get through.

The data below is from the log created during Simulated trading.

10/10/2007 10:59:24 AM|3|1|OIF read error 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00001.txt' Exception 'The process cannot access the file 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00001.txt' because it is being used by another process.'

10/10/2007 10:59:36 AM|3|1|OIF read error 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00002.txt' Exception 'The process cannot access the file 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00002.txt' because it is being used by another process.'

10/10/2007 10:59:36 AM|3|1|OIF delete error 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00002.txt' Exception 'The process cannot access the file 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00002.txt' because it is being used by another process.'

10/10/2007 10:59:52 AM|3|1|OIF read error 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00003.txt' Exception 'The process cannot access the file 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00003.txt' because it is being used by another process.'

10/10/2007 10:59:52 AM|3|1|OIF delete error 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00003.txt' Exception 'The process cannot access the file 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00003.txt' because it is being used by another process.'

10/10/2007 10:59:54 AM|3|1|OIF read error 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00004.txt' Exception 'The process cannot access the file 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00004.txt' because it is being used by another process.'

10/10/2007 10:59:54 AM|3|1|OIF delete error 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00004.txt' Exception 'The process cannot access the file 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00004.txt' because it is being used by another process.'

10/10/2007 10:59:59 AM|3|1|OIF read error 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00005.txt' Exception 'The process cannot access the file 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00005.txt' because it is being used by another process.'

10/10/2007 10:59:59 AM|3|1|OIF delete error 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00005.txt' Exception 'The process cannot access the file 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00005.txt' because it is being used by another process.'

10/10/2007 11:00:00 AM|3|1|OIF read error 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00006.txt' Exception 'The process cannot access the file 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00006.txt' because it is being used by another process.'

10/10/2007 11:00:00 AM|3|1|OIF delete error 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00006.txt' Exception 'The process cannot access the file 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00006.txt' because it is being used by another process.'

10/10/2007 11:00:05 AM|3|1|OIF read error 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00007.txt' Exception 'The process cannot access the file 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00007.txt' because it is being used by another process.'

10/10/2007 11:00:05 AM|3|1|OIF delete error 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00007.txt' Exception 'The process cannot access the file 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00007.txt' because it is being used by another process.'

It's not clear to me how NT's access to the OIF is triggered, but if it's at File Creation time, then that's a bit too soon, isn't it.

TIA

[NinjaTrader_JoshP]

According to the description here (http://www.ninjatrader-support.com/H...heConcept.html) the OIF happens immediately at file creation. The only quick workaround I can think of right now is to create the file somewhere else and then just move it into the NT folder. I believe that shouldn't cause any access violation errors, but perhaps someone with more experience in OIFs could help you out more on this issue.

[NinjaTrader_Dierk]

NT can't access/read the files since your writing process still is accessing them.

Please make sure your writing process really has closed the file reference/pointer/handle.

If the issue still persists, then you should create the file in a different directory and only move it to the "incoming" directory as you have closed it.

[maskiepop]

NT can't access/read the files since your writing process still is accessing them.

Please make sure your writing process really has closed the file reference/pointer/handle.

If the issue still persists, then you should create the file in a different directory and only move it to the "incoming" directory as you have closed it.

Dierk,

Thanks for your reply. Following your suggestion, I developed two approaches. The first was basically to test whether an MSDOS bat copy would do the job. The second, involves the Kernel32's MoveFileA routine to move the OIF files to NT's \incoming.

In the first approach, I created a set of OIF files (OIF00001, OIF00002, etc) in C:\Temp\OIF. Then I wrote a bat file (CopyOIF.bat) to copy from this directory to NT's \incoming directory, started NT; connected to Simulated data feed; switch ATI on.

I copied two OIF files using OIFCOPY.bat to NT's \incoming. Didn't work. I still get read errors, and worse it failed to place the orders. Here are the results:

11/10/2007 11:56:05 AM|1|4|Database reset
11/10/2007 11:56:07 AM|1|4|Shutting down SMTP server
11/10/2007 11:56:46 AM|1|2|Simulated Data Feed: Primary connection=Connecting, Price feed=Connecting
11/10/2007 11:56:46 AM|1|2|Simulated Data Feed: Primary connection=Connected, Price feed=Connected
11/10/2007 11:57:21 AM|1|1|AT, 'PLACE;SIM101;ES 12-07;BUY;1;LIMIT;1355.50;1354.00;DAY;;000001;; ' processing
11/10/2007 11:57:21 AM|3|1|OIF, 'PLACE;SIM101;ES 12-07;BUY;1;LIMIT;1355.50;1354.00;DAY;;000001;; ' strategy id ' ' does not exist
11/10/2007 11:57:21 AM|3|1|OIF read error 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00001.txt' Exception 'Could not find file 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00001.txt'.'
11/10/2007 11:57:51 AM|1|1|AT, 'PLACE;SIM101;ES 12-07;SELL;1;LIMIT;1355.50;1357.00;DAY;;000002;; ' processing
11/10/2007 11:57:51 AM|3|1|OIF, 'PLACE;SIM101;ES 12-07;SELL;1;LIMIT;1355.50;1357.00;DAY;;000002;; ' strategy id ' ' does not exist
11/10/2007 11:57:51 AM|3|1|OIF read error 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00002.txt' Exception 'Could not find file 'C:\Documents and Settings\Compaq_Owner\My Documents\NinjaTrader 6\incoming\OIF00002.txt'.'
11/10/2007 11:59:35 AM|1|2|Simulated Data Feed: Primary connection=Disconnected, Price feed=Disconnected
11/10/2007 11:59:42 AM|1|4|Shutting down data server

The use of MSDOS's copy is not actually my preferred solution. The second approach is, provided it works.

Before I described the second approach to you, it would be good to remember that the app that generates the OIFs is a 3rd party software - I have no control over when it will dispose of the file handles. I can close the files, and I can assure you that the OIF files had been closed. So the problem will have to be something else.

Anyway, to get around the problem, I decided to invoke the Kernel32's MoveFileA routine from within my app, and use that to copy the OIF files to the NT \incoming, as soon as the app creates an OIF file.

The OIF files were moved on the fly as expected, but NT failed to detect them. No messages on the log whatsoever about the presence of the OIF files. Now that's something that I can't understand.

Hope you can help.

TIA

[NinjaTrader_Dierk]

Sorry, this clearly is a problem where your 3rd party app still has an open "pointer" to the file or the file is not formatted properly. You need to address the author.

To verify:
- turn of ATI in NT
- create a valid OIF file using the OIF build (see docs)
- move this file out of the incoming directory
- turn on ATI in NT
- move the file back to the incoming directory

-> it will be processed as expected

[maskiepop]

Sorry, this clearly is a problem where your 3rd party app still has an open "pointer" to the file or the file is not formatted properly. You need to address the author.

To verify:
- turn of ATI in NT
- create a valid OIF file using the OIF build (see docs)
- move this file out of the incoming directory
- turn on ATI in NT
- move the file back to the incoming directory

-> it will be processed as expected

Dierk,

I realize that the 3rd party software has incompatibilities with NT regarding pointers to the file. Or at least the parts of it that I used. But I thought the MSDOS bat copy of OIF files to NT's \incoming, the first test I did, should work. In fact it did work - partially; NT read the file, deleted it, and then came up with an error about the file not existing. Access was not the problem there; seems more like logic.

In my second approach, the use of the Kernel32's MoveFileA routine inside my app, to move OIF files on the fly to NT's \incoming, I believe, proves that there is no access issue. If the Kernel32 can move the file from one directory to another doesn't that mean that the app that created the file has no "open" pointers to it. So why did NT not detect the presence of the OIF files moved to its \incoming with this technique?

Lastly, I'd like to follow your suggestion. But this is hardly an automated test. It is a very manual process. If you can suggest an automated test to prove that it works, I'd gladly consider it.

Thanks.

[NinjaTrader_Dierk]

As pointed out below the issue is (a) related to incorrect OIF file formatting or (b) incorrect file pointer handling.

Please proceed as per my suggestion and copy back the file by any automated procedure you'd prefer (like MS DOS copy or a custom app moving the file).

[maskiepop]

As pointed out below the issue is (a) related to incorrect OIF file formatting or (b) incorrect file pointer handling.

Please proceed as per my suggestion and copy back the file by any automated procedure you'd prefer (like MS DOS copy or a custom app moving the file).

Dierk,

I was able to pass large number of OIF files to NT. Thank you very much for your help. And also to those who replied to my post.

maskiepop