Jupyter kernel dies with njobs > 1

[oort77]

Describe the bug

Jupyter kernel dies with njobs > 1 in classification task.

To Reproduce

import pandas as pd
from sklearn.datasets import load_breast_cancer
from atom import ATOMClassifier

X, y = load_breast_cancer(return_X_y=True)
atom = ATOMClassifier(X, y, logger="auto", n_jobs=2, verbose=2)
atom.impute(strat_num="knn", strat_cat="most_frequent", max_nan_rows=0.1)  
atom.encode(strategy="LeaveOneOut", max_onehot=8, frac_to_other=0.05)  
atom.feature_selection(strategy="PCA", n_features=12)

atom.run(
    models=["LR","XGB"],#  "lSVM", "LDA"
    metric="f1",
    n_calls=25,
    n_initial_points=10,
    n_bootstrap=4
)

Expected behavior

As described above. Works fine with njobs=1.

Additional context

Ubuntu 20.04. Python 3.9.7. Separate conda environment with packages installed
from atom requirements.txt.

Version

4.9.1

[tvdboom]

I couldn't reproduce the error on my Windows machine with Python 3.8.3. Can you check a couple of things so that I have a better understanding of what's happening:

• How much memory does your machine have and how much of that is being used when running the command?
• Does the same error occur when you don't use hyperparameter tuning (n_calls=0)?

[oort77]

@tvdboom Ubuntu 20.04 remote box, 32GB RAM. Doesn't seem to be a memory problem, works fine with n_jobs=1.

Setting n_calls=0 doesn't change the behaviour.

And here it dies.

[tvdboom]

I did some googling and it could be related to a conda / MKL problem, see scikit-learn/scikit-learn#5046.

Proposed solutions:

• conda install numpy (I don't think this will solve it but can't hurt to try)
• conda install nomkl
• conda install -c intel mkl
• Try another virtual environment (e.g. venv, virtualenv)

If nothing of this works I'll keep digging, but either way, this is probably not an ATOM issue, but more likely a scikit-learn / conda issue.