[Feature Request] External Deploy Trigger (For Hot Deploy to running instance)

[JollyWizard]

Feature Request

I am a hobbiest trader / developer currently publishing indicators for Quantower and TradingView, with an emphasis on and enthusiasm for producing public domain code.

I have done some work with NinjaTrader 8, and would like to do more, however the lack of full and coherent integration with Visual Studio has created several headaches that I would like to address.

After extensive research on how NinjaTrader works in tandem with the build system, file system locking, and it's own application lifecycle / integrity concerns, I have come to the conclusion that the best solution, for all parties, would be for NinjaTrader to receive a command from an external source to trigger a rebuild of the Custom folder and relink/reload the relevant dlls, as happens when using the compile button in NinjaScript Editor.

If this feature already exists, please excuse my ignorance, but I believe, from my research, it is not yet available. It is my hope that it will be supported in the future.

I understand that you do not intend to fully support Visual Studio development, and I feel I understand your reasoning, but the only true roadblock I can find to doing it myself is that NinjaTrader only releases control over the relevant dlls when you trigger recompilation inside the NinjaScript editor.

This means that external redeploys can only be managed by rebooting the entire NT application, or "forced finger automation" (i.e. triggering an arbitrary recompile inside the NinjaScript Editor).

As stated above, my request is thus: that NT allows for a request to trigger it's own recompilation / relinking process, as would happen when using the NSE compile button.

I realize that live debugging from Visual Studio would require it to manage the startup of NT or setting up a debugging communication port, and I am not requesting such accommodation with this feature. I am only requesting that NT "play nice" with file system changes by providing a method to engage it's own process, already in use internally, without having to use the graphical interface.​
​

Proposed Method:​

One way to do this would be to watch for a specific file (i.e. ".recompile") and then delete the file when the process is complete. This would probably be the easiest for independent developers to work with, and for NT to implement.

Another acceptable solution would be to listen on a port for well formed requests and respond with a success/fail message. This would be slightly more complicated, but technically sound, and would lay the groundwork for externalizing live debugging without application restart.


Background on my research / experience:

So far I have developed 2 different NT8 indicators, which I have compiled, edited, and published using the embedded tools inside NT8 (NinjaScript Editor and export options).

I found these tools sufficient for a basic startup project, but awkward and insufficient for detailed development or rapid iteration.

I had already developed a series of MSBuild inheritable tasks for Quantower to address similar problems, so I set out to apply these kinds of tools for NinjaTrader as well. I have not provided links to such examples in case it is a forum terms violation (I haven't checked).

When attempting to open the projects in VS2022, in order to research build automation and automatic packaging, I started by using the VS button in NinjaScript Editor, I almost immediately broke compilation back inside NinjaTrader, because of the issue of generating duplicate assembly info files.

I was able to fix the problem, and proceeded to research the root cause and solutions.

Based on my experience with updating other projects to modern standards, and subsequent testing in this cases, I assert this is because the system generates legacy style "DotNet Framework" project files, when the poorly documented, but simpler and more flexible / feature rich "DotNet Core" project files would solve the problem. Cleaner "Framework" solutions are provided with the "Add-On" project template example, but those generated by NSE/VS experience are the least desirable and do not play nice with VS2022.

I spent at least two full "workdays" testing modernized project file output, monitoring the file system access, catching temporary files before deletion (to verify NT internal compilation commands), making changes inside and outside of NinjaTrader, reading dozens of support posts, and comparing the results to my experience with Quantower. I have also reviewed multiple examples of packaging NinjaTrader extensions, from published examples and it's own output, verified the enabling newer CSharp language features, generated proof of concept for automated packaging, and tested several approaches to automated deployment of add-ons. I have broken / fixed / rebooted my NT install countless times.

It is my view that NT support is very respectful, active, and honest, and I understand how organizational and development logistics have placed this issue between customer desire and practical company concerns. I believe the platform is overall technically sound, albeit weighted with technical debt, and exists in a market space where it could benefit from modernizing this process.

I hope that by meeting in the middle with the solutions proposed that the company and the community can work together to provide practical solutions to modernized NT development practices, without placing undo burden on the company's technical staff.

Thank you for your consideration.

[NinjaTrader_ChelseaB]

Hello JollyWizard,

To confirm the only issue is that you are not able to compile using Visual Studio, is this correct?

NinjaTrader does have to compile internally.

In Visual Studio (or any external editor) simply save (Ctrl + s) the .cs file you are editing while the NinjaScript Editor is open, and NinjaTrader will detect the file changes and will automatically trigger a compile. Create and delete files from the NinjaScript Editor, and then edit these in the external editor.

[JollyWizard]

Repectfully,

The issue is deeper than just not being able to compile with Visual Studio.

Among the many issues with NSE, are that the inbuilt compilation system is substandard, and does not allow for adjustments to the build process that would enable newer language features or automatic packaging, or even basic resource packaging (as acknowledged in the official documentation).

NSE also provides a button that opens a VS project in which many default actions and hotkeys in the Visual Studio experience, which are prominent and sometimes automatically triggered by other actions, will break the NSE compilation process in a way that is not well documented and requires manual intervention in the file system and familiarity with the Custom build process.

The NSE is thus artificially limited, drives the user towards breaking the system without proper recourse, and it does so because it does not "play nice" with standard practices and properly accommodate other development patterns.

These issues are further confounded by the akward directory layout requirements, mixing essential algorithms with custom code ("@*.cs") (which makes them unreliable and in-built packaging full of akward notices), add-on dll location requirements, often unnecessary templating injected into basic code, and other esoteric and poorly documented development pattern choices made in NinjaTrader architecture / experience.

So to say the issue is just "You can't compile in Visual Studio" is not what I'm getting at, at all, and the solution provided, to open a file in NSE and then edit it externally, is insufficient as it does not allow one to leave the substandard NSE experience, and develop a better one.

I will acknowledge that this may allow for a hack, where a single dummy file in an open NSE is used to trigger relinking by modifying it's last saved date. I will examine this as an alternative to a proper solution.

However, I want to be clear, that the solution provided has all the qualities of a hack, and not a proper solution. In my original post, I referred to the issue of having to resort to "forced finger automation", and I would include the need to hack around basic development practices by engaging the NSE graphical experience, at all, in this category.

Finally, I would like to add that the phrase "NinjaTrader does have to compile internally" is somewhere between an oversimplification and simply not true.

I have been able to compile, using VS2022 and a "DotNet Core" style project file (with "net48" target), my own "NinjaTrader.Custom.dll", and shown NinjaTrader has no problem accepting this dll, because it does not apply any sort of validation beyond the absolute basics of dll identifiers (provided by the standard compilation options of both "Framework" and "Core"), and the availability of classes referenced and required.

NinjaTrader will run this file without complaint, properly link it with vendor and add-on dlls, and properly run a chart window with external indicators that depend on code files (from an external folder structure) compiled into the "NinjaTrader.Custom.dll" (along with the basic "@*.cs" files). These rolled in files are compiled with the "latest" CS language features, demonstrating that the the language level limitations of the NSE experience are completely arbitrary and dependent on NSE assumptions.

The only issues to completely bypassing the built-in compilation process are that the classloader, when activated, will file-lock the "NinjaTrader.Custom.dll" and not release it until the application is closed or an internal recompile is triggered, nor will it relink any add-on dlls until the application is loaded from scratch or the internal compilation is triggered or add-on management processes are triggered.

These are unnecessary byproducts of the MSBuild toolchain and NinjaTrader UX decisions. The fact that an internal compilation will unload and relink the process means that the NinjaTrader architecture already supports the means to bypass the implicit file locking. The fact that charts will update to support NSE recompiled code changes "on the fly" means internal support is available to reload the dlls.

In reality, I would prefer that NinjaTrader just respect a request to unlink/unload/release all the custom files, wait for a command to relink them, and then do so. This would allow for wholesale replacement of the customization process by external tools.

Despite the lack of advanced security verification, I still anticipate the argument that allowing the arbitrary release of the main dll after loaded could be a security vector issue, and that is why I only requested the allowance of a compilation trigger that is completely file system or port communication based and does not require engaging the NSE graphical experience at all.

Thank you for your time and consideration.

James​

[NinjaTrader_ChelseaB]

Hello JollyWizard,

I'm not clear on what you are trying to do, or what the issue is your are running into.

"Among the many issues with NSE, are that the inbuilt compilation system is substandard, and does not allow for adjustments to the build process that would enable newer language features or automatic packaging, or even basic resource packaging (as acknowledged in the official documentation)."

You are trying to change the NinjaTrader program itself and its build properties?

You are trying to change NinjaTrader's language features?


Are you trying to make an addon that you are custom building as an assembly dll yourself in your own project?