Saving the trained classifier

[zkliesmete]

Hi, I have an extremely large reference dataset and would therefore like to be able to save it and load later on for classifying different new datasets. How do I best save the object preduced by the singler.train_single function?

Many thanks!

[LTLA]

Not easily, I'm afraid. The object returned by train_single() is mostly a wrapper around a pointer into the singlepp::TrainedSingleIntersect C++ object, which doesn't have any serialization mechanism.

Currently, the best I can suggest is extracting the trained._full_data, trained._features, trained._full_label_codes, trained._labels and trained._markers, pickling them, and then feeding them back into train_single() in a new Python session. This should save some time by skipping marker detection and aggregation (if you performed any).

singler.train_single(
    ref_data=unpickled_full_data,
    ref_labels=[unpickled_labels[i] for i in unpickled_label_codes],
    ref_features=unpickled_features,
    check_missing=False, # not needed anymore
    markers=unpickled_markers,
    aggregate=False, # not needed anymore
    # ... extra arguments as needed ...
)

If you're pulling data out of a "slow" format like HDF5, you could also consider realizing unpickled_full_data into a faster format before pickling (e.g., with delayedarray.to_sparse_array() or delayedarray.to_scipy_sparse_matrix()). This trades some memory usage for faster train_single() calls, especially with multiple threads.

FWIW the training time is rarely an issue in my experience. The classification time is usually an order of magnitude longer. The time required to process each cell in the test dataset is proportional to the number of cells in the reference dataset (assuming no aggregation was performed), and we usually have quite a few cells in the test dataset, so the classification run-time ends up being proportional to the product of the number of cells in both datasets. This means that, if your reference is really so large that training is painful, the training time is likely to be the least of your problems.

[zkliesmete]

I see, the aggregation step indeed helps a lot, and probably also makes the results more stable. Thanks a lot!