NT8B5-I'm in trouble - out of disk space-NT8 restore - market replay

[sledge]

I can't find the offending area that is consuming space.



I've been doing this - to import NT8 update'd replay data into NT8 on a different machine.

http://ninjatrader.com/support/forum/showthread.php?p=429025

On my 2nd machine where I import the date, I recently ran out of space.

Well that might make sense- I have Oracle database on here and NT7.

Fine - I moved my NT7 replay data and was able to import.

Today - I did the NT8 restore process again. About halfway through - it failed. That was odd since I thought I had enough space.

Export file of replay data from other machine is 15gb, which is what my NT7 files are size of replay only data (maybe .5gb smaller).

Well, I cleaned my my laptop - and had 24GB of free space. Perfect.

Import failed near the last steps -- out of space again!

I cleaned up the failed NT8 DB/replay data folder import.. and now I only have 9GB FREE!!!

I think I'm missing A LOT of drive space, probably at least 50% of my 128gb ssd.

I'm currently running a defrag (29%) needed.

I can not find how on C, anything totals to almost a full drive.

Defrag has a ways to go.

Next step - I'm turning on hidden files - and will poke around.


If anyone else sees they are losing space unexplicably - please chime in.
And any suggestions on where to look - more than welcome!

(Maybe these failed NT8 imports are leaving other files around on my machine?)

[sledge]

Found it - definitely an issue.

Had to turn on hidden files and probably system protected files.

As you can see, my NT8 folder now shows 30gb used, with 0 gb in replay, and 26.2 gb in the TMP folder!!!

There are 3 files in there, 1 folder has 15 gb, the other folder 10 gb and a startup.profile. Both folders are dated TODAY 9/26/2015.

Also of note - I did get a pop up message that I was out of disk space, so I hit OKAY - but the Restore Backup feature was still "running" - the window was there with the GREEN status bar. It didn't go away itself, and I'm pretty sure I hit cancel and exited NT8 normally.

None of my other systems have this issue since I don't have a failed import.

I'm going to cleanup NinjaTrader8/tmp and try again.

[koganam]

Found it - definitely an issue.

Had to turn on hidden files and probably system protected files.

As you can see, my NT8 folder now shows 30gb used, with 0 gb in replay, and 26.2 gb in the TMP folder!!!

There are 3 files in there, 1 folder has 15 gb, the other folder 10 gb and a startup.profile. Both folders are dated TODAY 9/26/2015.

Also of note - I did get a pop up message that I was out of disk space, so I hit OKAY - but the Restore Backup feature was still "running" - the window was there with the GREEN status bar. It didn't go away itself, and I'm pretty sure I hit cancel and exited NT8 normally.

None of my other systems have this issue since I don't have a failed import.

I'm going to cleanup NinjaTrader8/tmp and try again.

If you did not already have those turned on, I am shocked and appalled. For that, we are going to revoke your geek credentials.

You might also want to see how much is hiding in your Recycle Bin.

[sledge]

If you did not already have those turned on, I am shocked and appalled. For that, we are going to revoke your geek credentials.

You might also want to see how much is hiding in your Recycle Bin.

R.B.: 9 items, 35.8KB

[sledge]

Can someone investigate an import fail because of no more disk space? It appears that no clean up in the temp area with a failure causing even more space issues since it remains.

[sledge]

*BUMP*

Anyone?!

[Jim W.]

Thorough disk space cleanup

Two suggestions on giving your computer a thorough disk space cleanup:

1) Start with WinDirStat Directory Statistics Program. It "Shows where all you disk space has gone and help you clean it up." It is available free at http://windirstat.info/ .

"On start up, it reads the whole directory tree once and then presents it in three useful views:

• The directory list, which resembles the tree view of the Windows Explorer but is sorted by file/subtree size,
• The treemap, which shows the whole contents of the directory tree straight away,
• The extension list, which serves as a legend and shows statistics about the file types."

You can delete files and folders directly from WinDirStat directory list view.

2) I have posted a thread in a JigSaw Trading Forum at Solutions For Slow Computer .

"When your desktop/laptop computer seems slow you have two choices, upgrade your computer's hardware or buy a new one. Before you do either, try giving your existing computer a tune-up. It won't cost you anything but your time."

The first two posts include instructions on giving your computer a thorough disk space cleaning.

[sledge]

Thanks Jim, but I think there is really a bug in NT8 not cleaning up a failed install.

On 128gb ssd, so space is a premium.

[Jim W.]

sledge, Understand, but after the bug fixed a tune-up will help. Posted because you made the comment "Maybe these failed NT8 imports are leaving other files around on my machine? ". WinDirStat will make large files obvious and easy to delete.

[NinjaTrader_DaveI]

sledge, do you remember exactly what you found in your Documents/NinjaTrader 8/tmp folder? Are the files still there for you to look at? For reference, I have one file in that directly, and it is taking about 70 kb, so the size of that folder on your machine might be a red flag and/or a clue.

[sledge]

sledge, do you remember exactly what you found in your Documents/NinjaTrader 8/tmp folder? Are the files still there for you to look at? For reference, I have one file in that directly, and it is taking about 70 kb, so the size of that folder on your machine might be a red flag and/or a clue.

It was each and every replay file. They appear to be created in there, then moved to replay folder. Same process in NT7.

It's that upon failure(disk full), the tmp wasn't cleaning up, in my experience with NT8.

[NinjaTrader_DaveI]

Gotcha, thank you. Although it would be tricky to try to replicate a full SSD on my machine to cause the failed import, I believe I have enough information to move forward and report this to our team in the morning. I will keep you updated with any progress or insights along the way.

[sledge]

Gotcha, thank you. Although it would be tricky to try to replicate a full SSD on my machine to cause the failed import, I believe I have enough information to move forward and report this to our team in the morning. I will keep you updated with any progress or insights along the way.

It would require filling your disk up enough with data.

Here, I did it again tonight.

I had 16GB free or so - I imported my 10-26 replay data only. Success.

I copied 7GB of videos to c:\

I imported that same file. FAIL. The import backup file window kept counting remaining time, and my only option was to "CANCEL" and then "CANCEL" again.

And now 7GB were left in the ninjatrader8\tmp

What I did initially was move a lot of files off my drive, not knowing about \tmp leftovers. Made enough space and then it happened again (i had installed other files on c:\). Moved more files off... made some space.. damn this drive keeps shrinking or something.. Where's all my space!? Now I can't even import anything??

[NinjaTrader_DaveI]

Hello,

I've done quite a bit of testing to try to reproduce this issue, but I'm seeing very different results. I created a new logical drive, redirected my Documents folder to it, and filled it up. What I'm finding is that when I import Replay data from NinjaTrader 7 without having enough space on the logical drive, that all of the files are imported at a size of 4 kb, and they do not appear to actually contain any Replay data. I'm also not seeing any hidden or unhidden files populating in the tmp folder as the Replay data is being migrated, and nothing is being left over after the failed migration.

I do have an update on this issue that should prove useful. We're looking into a very similar issue on NTEIGHT-8922, and we plan to implement changes to let users know that any errors from the migration process will be noted in their log files. This should at least mitigate the possibility of another user having to investigate exactly what went wrong in this scenario.

Just to be thorough on this one, have you ever seen an instance where these temporary files are left over after a successful migration, where there was enough free disk space to complete the process?

[sledge]

Hello,

I've done quite a bit of testing to try to reproduce this issue, but I'm seeing very different results. I created a new logical drive, redirected my Documents folder to it, and filled it up. What I'm finding is that when I import Replay data from NinjaTrader 7 without having enough space on the logical drive, that all of the files are imported at a size of 4 kb, and they do not appear to actually contain any Replay data. I'm also not seeing any hidden or unhidden files populating in the tmp folder as the Replay data is being migrated, and nothing is being left over after the failed migration.

I do have an update on this issue that should prove useful. We're looking into a very similar issue on NTEIGHT-8922, and we plan to implement changes to let users know that any errors from the migration process will be noted in their log files. This should at least mitigate the possibility of another user having to investigate exactly what went wrong in this scenario.

Just to be thorough on this one, have you ever seen an instance where these temporary files are left over after a successful migration, where there was enough free disk space to complete the process?

Dave, thanks for looking into this. ..

But I think we are doing 2 things different here. Maybe i wasn't entirely clear.

This is NT8 restore from an NT8 replay backup file from NT8 export.

But on successful completion, the files in tmp are removed.

I hope your are able to reproduce this with this information.