[python-package] model_to_string() returns empty string intermittently when early_stopping callback is used

[nbx-liz]

Description

When using LGBMRegressor.fit() with eval_set and lgb.early_stopping() callback, model_to_string() intermittently returns an empty model string containing only pandas_categorical metadata (no tree structure). This causes model_from_string() at engine.py:350 to fail with:

LightGBMError: Model file doesn't specify the number of classes

The booster itself is valid (correct num_trees(), num_feature(), current_iteration()), but the C API LGBM_BoosterSaveModelToString produces an incomplete output.

Reproducibility

• Failure rate: ~5-10% per fit() call
• Affected versions: 4.3.0, 4.5.0, 4.6.0 (all tested)
• Platform: Linux (WSL2), Python 3.11
• Non-deterministic: same data/params sometimes succeeds, sometimes fails

Minimal Reproduction

import lightgbm as lgb
import pandas as pd
import numpy as np

url = "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/diamonds.csv"
df = pd.read_csv(url)
y = df["price"].values
X = df.drop(columns=["price"])
for col in X.select_dtypes(include="object").columns:
    X[col] = X[col].astype("category").cat.codes

idx = np.random.RandomState(42).permutation(len(X))
X_train, y_train = X.iloc[idx[:40000]], y[idx[:40000]]
X_valid, y_valid = X.iloc[idx[40000:45000]], y[idx[40000:45000]]

# Repeatedly fit new models — ~5-10% will crash
for i in range(100):
    n_est = np.random.randint(600, 2500)
    obj = np.random.choice(["huber", "mae"])
    model = lgb.LGBMRegressor(
        n_estimators=n_est, objective=obj,
        learning_rate=0.01, max_depth=8, verbose=-1,
    )
    model.fit(
        X_train, y_train,
        eval_set=[(X_valid, y_valid)],
        callbacks=[lgb.early_stopping(150, verbose=False), lgb.log_evaluation(-1)],
    )
    # Crash happens inside fit() at engine.py:350

Condition Isolation

Condition	Failure rate
eval_set + early_stopping callback	~8%
eval_set + log_evaluation only	0%
eval_set only (no callbacks)	0%
No eval_set	0%

The early_stopping callback does not need to actually trigger (models run to num_boost_round). Its mere presence in the callback list is sufficient.

Debug Findings

By patching engine.train() with keep_training_booster=True and manually inspecting model_to_string():

• Empty model string: 167 bytes, containing only pandas_categorical:[...]
• Booster state is valid: num_trees()=1597, num_feature()=9, current_iteration()=1597
• The C API LGBM_BoosterSaveModelToString returns a truncated result despite the booster holding a valid model

Expected Behavior

model_to_string() should always return the complete model string including tree structures when num_trees() > 0.

Environment

LightGBM: 4.3.0 / 4.5.0 / 4.6.0 (all reproduce)
OS: Linux 6.6.87 (WSL2)
Python: 3.11.14

[nbx-liz]

Additional Finding: sklearn wrapper only

Further investigation reveals this bug is specific to the sklearn API (LGBMRegressor/LGBMClassifier). The Booster API (lgb.train()) does not reproduce the issue even with identical data, params, and callbacks.

Comparison (100 iterations each)

API	early_stopping	categorical_feature	Failure rate
lgb.train()	Yes	No	0/100
lgb.train()	Yes	Yes	0/100
lgb.train() + record_evaluation	Yes	No	0/100
LGBMRegressor.fit()	Yes	Yes	6/100
LGBMRegressor.fit()	Yes	No	2/100
LGBMRegressor.fit()	No	Yes	0/100

Reproduction with Booster API (passes)

import lightgbm as lgb
import pandas as pd
import numpy as np

url = "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/diamonds.csv"
df = pd.read_csv(url)
y = df["price"].values
X = df.drop(columns=["price"])
for col in X.select_dtypes(include="object").columns:
    X[col] = X[col].astype("category").cat.codes

idx = np.random.RandomState(42).permutation(len(X))
X_train, y_train = X.iloc[idx[:40000]], y[idx[:40000]]
X_valid, y_valid = X.iloc[idx[40000:45000]], y[idx[40000:45000]]

# This NEVER fails (0/100)
for i in range(100):
    train_data = lgb.Dataset(X_train, label=y_train)
    valid_data = lgb.Dataset(X_valid, label=y_valid, reference=train_data)
    n_est = np.random.randint(600, 2500)
    obj = np.random.choice(["huber", "mae"])
    params = {"objective": obj, "learning_rate": 0.01, "max_depth": 8, "verbose": -1}
    booster = lgb.train(
        params, train_data, num_boost_round=n_est,
        valid_sets=[valid_data],
        callbacks=[lgb.early_stopping(150, verbose=False), lgb.log_evaluation(-1)],
    )

Reproduction with sklearn API (fails ~5-8%)

# This fails ~5-8% of the time
for i in range(100):
    n_est = np.random.randint(600, 2500)
    obj = np.random.choice(["huber", "mae"])
    model = lgb.LGBMRegressor(
        n_estimators=n_est, objective=obj,
        learning_rate=0.01, max_depth=8, verbose=-1,
    )
    model.fit(
        X_train, y_train,
        eval_set=[(X_valid, y_valid)],
        callbacks=[lgb.early_stopping(150, verbose=False), lgb.log_evaluation(-1)],
    )

Debug observation

When patching engine.train() to use keep_training_booster=True and manually calling model_to_string():

• The booster is valid: num_trees()=1597, num_feature()=9, current_iteration()=1597
• But model_to_string() returns only 167 bytes containing pandas_categorical:[...] — no tree structure at all

This suggests something in the sklearn wrapper's Dataset construction or param handling path corrupts the booster's internal state for serialization, but only when the early_stopping callback is present.

[jameslamb]

Thanks for using LightGBM and for reporting this.

I'd be happy to help, but we need some help from you. This report is a bit hard to follow right now. Could you please try to narrow it down further to a reproducible and minimal example?

Some things that will help get to a reproducible example:

• only use LightGBM 4.6.0
• pass parameters to LightGBM to make it deterministic
  • seed = 708
  • deterministic = True
  • n_jobs = 1
  • force_row_wise = True
• set verbose: 1 so we can see if there are relevant log messages in the cases that fail
• in those for loops, when the error you're reporting happens, capture and print the values of n_est and obj

Hopefully those changes will help you narrow down the specific inputs that lead to the error you're reporting.

at engine.py:350

Line numbers change as the library changes. It would have been helpful if you'd included the full stack trace in your report so we could see the part of the code this refers to.

I looked through the versions your reported trying, and I'm guessing you're referring to this code:

LightGBM/python-package/lightgbm/engine.py

Lines 349 to 350 in d02a01a

	if not keep_training_booster:
	booster.model_from_string(booster.model_to_string()).free_dataset()

[nbx-liz]

Thanks for the guidance, @jameslamb. I ran systematic isolation experiments on LightGBM 4.6.0 and narrowed down the root cause significantly.

Key finding: n_jobs=1 alone eliminates the bug

Setting deterministic=True is not required — n_jobs=1 by itself is sufficient to suppress the bug (0/100). This confirms a race condition in multi-threaded mode within the sklearn wrapper's model_to_string() path.

Because this is a threading race condition, deterministic=True makes it unreproducible — the specific inputs (n_est, obj) don't matter; the same parameters succeed or fail depending on thread timing.

Minimal reproduction (LightGBM 4.6.0)

import lightgbm as lgb
import numpy as np
import pandas as pd

print(f"lightgbm={lgb.__version__}  numpy={np.__version__}  pandas={pd.__version__}")

df = pd.read_csv(
    "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/diamonds.csv"
)
y = df["price"].values
X = df[["carat", "depth", "table", "x", "y", "z"]]  # numeric only

idx = np.random.RandomState(42).permutation(len(X))
X_tr, y_tr = X.iloc[idx[:40000]], y[idx[:40000]]
X_va, y_va = X.iloc[idx[40000:45000]], y[idx[40000:45000]]

for i in range(200):
    try:
        lgb.LGBMRegressor(
            n_estimators=1500, objective="huber",
            learning_rate=0.01, max_depth=8, verbose=-1,
        ).fit(
            X_tr, y_tr,
            eval_set=[(X_va, y_va)],
            callbacks=[lgb.early_stopping(150, verbose=False), lgb.log_evaluation(-1)],
        )
    except Exception as e:
        print(f"FAILED at iteration {i}: {e}")
        break
else:
    print("No failure in 200 iterations (re-run to retry; ~2-8% per iteration)")

Notes:

• Categorical features are not required — numeric-only columns reproduce the bug.
• Synthetic data (np.random.randn, sklearn.datasets.make_regression) does not reproduce — the diamonds dataset's distribution appears necessary.
• The failure rate is ~2-8% per fit() call; a single run of 200 iterations may occasionally pass. Re-running typically triggers the failure.

Full stack trace

Traceback (most recent call last):
  File "repro.py", line 18, in <module>
    lgb.LGBMRegressor(...).fit(...)
  File "lightgbm/sklearn.py", line 1398, in fit
    super().fit(
  File "lightgbm/sklearn.py", line 1049, in fit
    self._Booster = train(
  File "lightgbm/engine.py", line 350, in train
    booster.model_from_string(booster.model_to_string()).free_dataset()
  File "lightgbm/basic.py", line 4540, in model_from_string
    _safe_call(
  File "lightgbm/basic.py", line 313, in _safe_call
    raise LightGBMError(_LIB.LGBM_GetLastError().decode("utf-8"))
lightgbm.basic.LightGBMError: Model file doesn't specify the number of classes

Isolation experiments (100 trials each)

Setting	Failures	Notes
Baseline (default params)	1–2/100	Some runs SIGSEGV instead of Python exception
n_jobs=1 only	0/100	Threading is the root cause
deterministic=True only	0/50	Also suppresses (forces single-thread internally)
n_jobs=1, force_row_wise=True	0/100	Redundant; n_jobs=1 alone is sufficient
force_row_wise=True only	2/100	No effect (still multi-threaded)
force_col_wise=True only	14/100	Worse — col-wise threading amplifies the race
Booster API (lgb.train())	0/100	Not affected (skips model_to_string() round-trip)

Per-objective breakdown (100 trials, default params, subprocess-isolated to catch SIGSEGV):

Objective	Result
regression	SIGSEGV
huber	SIGSEGV
mae	SIGSEGV
quantile	11/100 (Python exception)
poisson	0/100

The bug manifests as SIGSEGV for some objectives and as a Python LightGBMError for others.

Summary

• Root cause: Race condition in multi-threaded mode, specific to the sklearn wrapper's model_to_string() → model_from_string() round-trip at engine.py:350.
• Why sklearn only: The Booster API uses keep_training_booster=True by default and skips this code path.
• Why early_stopping required: The callback's presence appears to create the conditions for the race, even when early stopping doesn't actually trigger.
• force_col_wise=True increases the failure rate from ~2% to 14%, suggesting the col-wise multi-threading path is more susceptible.

Environment

LightGBM: 4.6.0
Python: 3.11.14
OS: Linux 6.6.87 (WSL2), 32 cores
NumPy: 2.4.2
Pandas: 3.0.1
scikit-learn: 1.8.0

[jameslamb]

Why sklearn only: The Booster API uses keep_training_booster=True by default and skips this code path.

I don't understand this statement at all. The only part of lightgbm (the Python package) with a parameter named that way is train(), which sets it to False by default

LightGBM/python-package/lightgbm/engine.py

Line 117 in 0412fd8

keep_training_booster: bool = False,

The scikit-learn interface does not override that default.

It is also confusing to not see keep_training_booster=True in any of the example code you gave. I don't understand how that's related.

Can you please clarify that?

_ narrowed down the root cause significantly._

If we're narrowing down, the amount of information in these messages should start getting smaller. For example your most recent post mentioned 5 different loss functions... it looks like regression is sufficient to trigger the bug, let's please stick with just that.

If you could please try to make these messages a bit more concise, we could get closer to a minimal, reproducible example. The for loops and tables with % and all the different variations are making this difficult to follow. Find a single combination of inputs that triggers the error condition once, and try running just that combination 100 times. If it reproduces the error a few times, share just that code with us.

I suspect you might be using some LLM tool to do this... that's totally fine as a research aid, but the verbosity of the output and all the extra stuff being added is making it difficult to narrow this down. I'll ask that if you are using those tools, you please write the messages you post here yourself (even if that is just you hand-editing what the LLM produces).

[nbx-liz]

I apologize for the confusion. After my previous post, I continued investigating and realized there was a bug in my experiment code along with several misunderstandings on my part. The claim that early_stopping is a necessary condition, among other things, was incorrect.

The error can be reproduced with the following code:

import lightgbm as lgb
import pandas as pd

df = pd.read_csv(
    "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/diamonds.csv"
)
y = df["price"].values
X = df[["carat", "depth", "table", "x", "y", "z"]]

for i in range(100):
    lgb.LGBMRegressor(n_estimators=1500).fit(X, y)
    print(f"OK {i}")

In my environment, this typically fails around iteration 20–50 with lightgbm.basic.LightGBMError: Model file doesn't specify the number of classes.

Environment:

LightGBM: 4.6.0, Python: 3.11.14, OS: Linux 6.6.87 (WSL2), 32 cores

I have confirmed that the error persists after reinstalling all packages. Please let me know if any additional information would be helpful.