Replication on Different Data

[sbaird91]

I have been able to reproduce the results based on the vignette with the Lebron dataset. Everything works great.

However, the Logistic function doesn't seem to work on other datasets. I have attached a simulated data to illustrate. I used the same script as the vignette, with the one change being colnum = 1 ("outcome" column)

data_sim_logit.csv

Logistic(data = test,
colnum = 1,
numresamples = 25,
remove_VIF_greater_than <- 5.00,
save_all_trained_models = "N",
save_all_plots = "N",
set_seed = "N",
how_to_handle_strings = 0,
do_you_have_new_data = "N",
remove_ensemble_correlations_greater_than = 0.80,
use_parallel = "Y",
train_amount = 0.50,
test_amount = 0.25,
validation_amount = 0.25)

I get the following error: Error in [.default(ensemble_tree_train_table, 2, 2) subscript out of bounds

I've never received this error with the Lebron data. The basic structure of the simulated data--all variables being num or int--is similar to Lebron.

I've tried additional datasets, some based on actual observations, without any success. I usually get a similar, although not exact, error.

I ran a simple logit model to make sure there wasn't anything problematic with the data. This ran just fine:
mod <- glm(outcome~., data=test, family = 'binomial')

[InfiniteCuriosity]

Thank you for the note. I was able to get it to run with that data set, no problem. The issue that happened in your results means that the solution matrix was not complete for one of the models. As an example, it should look like this:

$Summary_tables$Ensemble Gradient Boosted
ensemble_y_test
ensemble_gb_test_predictions_binomial 0 1
0 583 1469
1 1023 3175

Instead one of the rows is missing, so it comes out something like this (note the 2nd row is missing in this example):

$Summary_tables$Ensemble Gradient Boosted
ensemble_y_test
ensemble_gb_test_predictions_binomial 0 1
0 583 1469

This happens because the data is very imbalanced. The 2nd chart below shows bar charts of each of the features in the original data, and is automatically produced. Look at the Plots tab after you run LogisticEnsembles, there are a lot of plots automatically produced.

If the data are imbalanced (lots of one, few of the other), that error might happen, because the package randomizes the data. Just rerun the data, that usually solves the issue unless the data is extremely unbalanced. By the way, the Lebron data is not imbalanced, which is why it does not produce errors.

I'm working on a solution to prevent this issue when the data is imbalanced, but it's still in progress.

This has the easiest solution possible - just run it again :)

Good news, I was able to get it to run using this code, no problem. You wrote the same thing happened with other data sets. Please email me, I'm happy to work on those to see what is going on and get those fixed.

Logistic(data = df,
colnum = 1,
numresamples = 5,
remove_VIF_greater_than <- 5.00,
save_all_trained_models = "N",
save_all_plots = "N",
set_seed = "Y",
how_to_handle_strings = 0,
do_you_have_new_data = "N",
remove_ensemble_correlations_greater_than = 0.80,
use_parallel = "Y",
train_amount = 0.50,
test_amount = 0.25,
validation_amount = 0.25)

I used the seed = 12345, and the lowest RMSE was with the RandomForest model, with a score of 0.7376. You should be able to get the same results with the same seed. (by the way, Generalized Linear Models, came in 18th place with a score of 0.5972 for comparison if you look at the full report). Here are some of the results using a seed of 12345 on the same data set:

Thanks for the comment, please let me know if there are any questions.

[InfiniteCuriosity]

By the way, I received the good news this morning the updated version of LogisticEnsembles is on CRAN. Please update LogisticEnsembles to the latest version, which is 0.8.1. It does not solve the imbalanced data issue, but there are a ton of other improvements in the updated package.

[sbaird91]

Thank you for taking a look at this and providing updates. Once I reinstalled the newest CRAN version, everything works! I think at least a part of my issue was related to installing/updating the dependencies. I would sometimes get errors such as installation of package ‘glue’ had non-zero exit status, which indicates there's something weird with my local environment, not the package itself.

Either way, things are up and running! Looking forward to playing around with this package. Pretty awesome stuff! Thanks.

[sbaird91]

I do have one follow-up question now that everything is running smoothly. In the simulation example, as you reported, the RandomForest model seems to be the best performer. Now, I want to extract that particular model and make predictions on new data. How do I do this? I haven't seen anything in the CRAN documentation that points me in the right direction (of course, I may be missing something, but wanted to check).

[InfiniteCuriosity]

Thank you for the note. Yes, there is a way to save that trained model. Actually, this solution saves all the trained models. If you look at the Logistic function again that you’ve been looking at, one of the options is save_all_trained_models. Change that from “N” to “Y” and it will automatically save all of the trained models. For example: ``` Logistic(data = Lebron, colnum = 6, numresamples = 25, remove_VIF_greater_than <- 5.00, save_all_trained_models = "Y", save_all_plots = "N", set_seed = "N", how_to_handle_strings = 0, do_you_have_new_data = "N", remove_ensemble_correlations_greater_than = 0.80, use_parallel = "Y", train_amount = 0.50, test_amount = 0.25, validation_amount = 0.25) ``` All of the trained models are automatically saved in tempdir() and that directory is automatically deleted as soon as you finish your R session. My package will automatically print to screen the location where the temp files are located on your drive. Find that location on your hard drive and move only the trained models you want to keep to a location that will not be deleted. I commonly move it to the desktop and start there. For example: ```

tempdir()

[1] "/var/folders/2c/hvxqjjcn3sq5lcshxhv8wnsw0000gp/T//RtmpbxgGHk" ``` The list of models looks like this:  Simply move the RF (Random Forest) model to a different location (such as the desktop), and you can use it from there. One feature is that this allows you to go deeper into the model. Just load the model into R, then you can get all the features of the model. Load the model, then put a $ at the end of the model name, and it shows you all the features of the model, like this: <img width="387" height="248" alt="Image" src="https://github.com/user-attachments/assets/c3abbad5-244b-40c8-bfa0-1f33fe489867" /> You may also rename it, or do anything you want with it. This works with all the models. Let me know if there are any questions, very happy to help. Thank you very much for the comments, greatly appreciated! Best wishes, Russ Conte PS - I'm trying to get it to save the images so they show here, but I can't get that to save, my apologies about that. On Sep 2, 2025, at 2:04 PM, sbaird91 ***@***.***> wrote:sbaird91 left a comment (InfiniteCuriosity/LogisticEnsembles#9) I do have one follow-up question now that everything is running smoothly. In the simulation example, as you reported, the RandomForest model seems to be the best performer. Now, I want to extract that particular model and make predictions on new data. How do I do this? I haven't seen anything in the CRAN documentation that points me in the right direction (of course, I may be missing something, but wanted to check). —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[InfiniteCuriosity]

Now, I want to extract that particular model and make predictions on new data. How do I do this?

A much easier way that is built into the LogisticEnsembles package is to simply mark "Y" to "do_you_have_new_data" in the LogisticEnsembles function.

The first thing it will do is ask where the new data is located. That question is seen in the Console. (a CSV file works fine, such as 'example.csv'), then LogisticEnsembles will run normally, and you'll be able to see the predictions on the new data (in the Viewer) using all the trained models when everything is finished.

You can ask LogisticEnsembles to do any number of things at once, so you can ask it to predict on new data AND save all trained models AND set a seed AND save all plots (or any other combination of all the features), all in one shot. It's extremely cool to see!

For example:

Logistic(data = Lebron,
         colnum = 6,
         numresamples = 25,
         remove_VIF_greater_than <- 5.00,
         save_all_trained_models = "Y",
         save_all_plots = "Y",
         set_seed = "Y",
         how_to_handle_strings = 0,
         do_you_have_new_data = "Y",
         remove_ensemble_correlations_greater_than = 0.80,
         use_parallel = "Y",
         train_amount = 0.50,
         test_amount = 0.25,
         validation_amount = 0.25)