.Net 10 support + Modernization

[galvesribeiro]

Expected Behavior

With .Net 10 being the LTS which includes several performance benefits, I would wonder if it was possible to migration to .Net 10 while modernizing the legacy infrastructure.

Actual Behavior

Current codebase uses .Net 9, old SLN based solution and rely on ancient MEF (which is essentially dead, besides its deep roots in Visual Studio, which is where it was built for). Seems like the engine was ported from the pre-.Net Core era, from .Net Framework, into the modern .Net without any modernization.

(For the record, I'm new to LEAN so forgive me if I lack the proper background on why things are as is Today)

Potential Solution

My proposal is to:

1. Migrate to .Net 10 as is. This is a low hanging fruit since the engine is already on .Net 9.
2. Port over all the legacy infrastructure to use the modern .Net (i.e. Microsoft.Extensions.* packages):

• 2.1. Move away from all the static code and MEF in favor of Microsoft.Extensions.DependencyInjection - this is a relatively easy move since the code is well separated and takes dependencies thru MEF. This will allow way more flexibility on tests and give the option to have scoped dependencies. Right now, everything is a big singleton/static chain. This will also make more space for customizations on the engine by registering types properly, not as plain old static/singleton instances. This would allow multiple instances of the engine to run under a single process, and for example enable scenarios like clusters of engines running different strategies sharing the same compute nodes (yes, there are use cases for that) without requiring any kind of hard isolation like VMs or containers per engine instance.
• 2.2. Move from a custom Logging infrastructure to use Microsoft.Extensions.Logging - If 2.1 is done, this only require changes the call sites since they will all be able to inject ILogger<T>. For all the third party integrations, the the Log() and Log would still be provided but it would be initialized as a forwarded to a singleton ILogger for the sake of back compatibility. Logs can now use source generator, to reduce even further the log penalty on performance.
• 2.3. Move from static, non-refreshable configuration to use Microsoft.Extensions.Options - Right now once the config starts it can't be updated unless you restart the whole process. Yes, some things are not refreshable by definition like assembly names or types which are dynamically loaded, but others could be, like log level, strategy parameters, etc. Once an algorithm starts the only way to change is stop and start again and the configuration only comes from JSON or CLI, so it could come from a remote database or API service and be refreshed. Then whenever configuration is needed, IOptionsMonitor<TOptions> would be used to get the new config. This is specially important for long-running algos or day trading where small tweaks should be made on-the-fly without requiring a restart while also keeping configuration strongly typed. This is another easy to implement feature if 2.1 is implemented.
• 2.4. Telemetry is essentially not supported anywhere as it is. Here I'm talking about technical telemetry like traces, metrics, etc., not market metrics (I.e. P&L, orders, etc). With the OpenTelemetry (OTEL) integration with the modern .Net, we can easily add traces based on Activity along with metrics and have application level performance. Real data required to operate such a system.
• 2.5. Move from custom host to leverage the IHost infrastructure - Since .Net Core was released there is a "Generic Host" set of abstractions that every single .Net server application use it. This provide all the plumbing for hosts and because of the 2.1, it glues together all the services the application will use. This allow you to co-host the engine inside any other services which already use those abstractions. This is also a simple to implement feature.

3. Move in most of places where I/O is required/possible to use Task/Task<T> or ValueTask/ValueTask<T>. Current codebase is all "void-driven" which enforce synchronous code even on places where asynchronous code could be used like for example, inside the broker and data feed integrations. Since the majority of the .Net APIs now are asynchronous, we are forced to call .GetAwaiter().GetResult() or even worse, .Wait(). This is horrible for performance. Making the codebase async first would have significant improvements in performance specially on multithreaded hosts. I understand that some places in the codebase requires synchronisation for the sake of correctness like for example the core engine loop of events. But even there, a custom TaskScheduler would be the correct way of doing it Today with modern .Net and would still allow async code. For the cases where async is optional, or they have most of the cases a sync closure, ValueTask/ValueTask<T> would be used to not even have the cost of allocating Task and prevent the compiler from generating the state machine. This would make the codebase modern and be able to use the existing NuGet packages ecosystem where things that require I/O can properly do it without blocking the CPU.
4. Use the new Math libraries which are SIMD-accelerated. This can be done separately, but would be a nice to have, specially for people who want to use large datasets in memory and do vector based queries.
5. Move away from JSON.Net - .Net now has a way faster and (almost) zero-allocation JSON serializer (irony essentially made by the same creator), System.Text.Json. We should get rid of Newtonsoft.Json, it served its time already. The known internal types would leverage the compile-time source generator for the serializers, avoiding the reflection cost at runtime. For custom types made by the developers, they could inject a JsonSerializerOptions which we could extract their generated serializers.
6. There are many outdated 3rd party packages which had a lot of performance improvements since they kept updating to the latest .Net versions. Some of the used ones even have big security issues which are fixed on newer versions.
7. Commandline parser - .Net have a native CommandLine parser with System.CommandLine (finally!).
8. Make the engine AoT-friendly (when possible, depending on user custom code).
9. Remove RestSharp. I understand the reasons why back on .Net Framework people used RestSharp. I used myself. Since the introduction of Microsoft.Extensions.Http with HttpClientFactory. There is no reason to use RestSharp anymore.

There are bunch of other changes that could be applied but with this one, we can modernize the code base and at same time keep backcompat. For example, configuration adapter to support old format as is, logger that log exact same format of text, etc., the asynchronous methods can be overloads even on interfaces to prevent breakage etc. New features or small changes in behavior like the log format change can be completely opt-in by picking some flags in the configuration, preserving the old behavior for those who don't want to change anything but would like to get the performance improvements those overall changes are bringing to the table.

Those changes would be done in no way as a breaking change and the current behavior and API surface would remain the same.

I currently have done a wrapper around the engine to be able to leverage the modern C# capabilities but it is starting to become too big that I'd rather prefer work on a PR here with those changes.

Would this be acceptable as a PR? I can help. If so, I can create a an issue with a task-list and start doing one by one.

Checklist

• I have completely filled out this template
• I have confirmed that this issue exists on the current master branch
• I have confirmed that this is not a duplicate issue by searching issues