System Information

• OS & Version: Windows 10, Windows 2016
• ML.NET Version: ML.NET > 3.01 (including 5.0 preview)
• .NET Version: .NET 8.0

Describe the bug
We use ML.NET on a large application with approx 20 models using only ML.NET FastTree regression.
For all ML.NET versions after 3.0.1 the time to load a model from a MLContext has increased dramatically

To Reproduce
Create a regression model using FastTree
Load the model from a MLContext with version 3.0.1. Note the loading time
Load the same model with version 4.0.2. The loading time should be much higher
We observed between 7 and 50 times increased in loading times

Expected behavior
The loading time should not be different

Screenshots, Code, Sample Projects
var mlContext = new MLContext(seed: 0);
var loadedModel = mlContext.Model.Load("path_of_model.zip", out _);
-> loading time 111ms with 3.0.1, 1100ms with 4.0.2

Additional contextA
Loading a model produced with version 4.0.2 using ML.NET 3.0.1 is fast
Loading a model produced with version 4.0.2 using ML.NET 4.0.2 is slow
Our model zip file range from 1000ko to 3000ko