Some Questions.

[dibu28]

Can I use it for classification with 500+ categories?
(Like my documents can be from 500 different categories but only one category per document)

Can I use it with hierarchical categories?
(Like there can be 10 root categories 20 sub categories and 30 sub-sub categories. but full name of the category can be 10.20.30 ? )

Can I use it with ModernBert models?

Can I add new documents and categories at runtime? So that they be used to classify new documents?

Can it be used for 10000 or 100000 documents?

[codelion]

Can I use it for classification with 500+ categories?
(Like my documents can be from 500 different categories but only one category per document)

Yes you can, for good results make sure there are a few examples for each category.

Can I use it with hierarchical categories?
(Like there can be 10 root categories 20 sub categories and 30 sub-sub categories. but full name of the category can be 10.20.30 ? )

Yes, the product category classification use case has a similar structure for categories as you can see in the example colab -https://colab.research.google.com/drive/1VyxVubB8LXXES6qElEYJL241emkV_Wxc?usp=sharing

        "tool": "53 in Fiberglass Handle Mortar Hoe",
        "description": "This 53 in. Fiberglass Handle Mortar Hoe is designed for spreading and mixing mortar, cement, and concrete. It features a high-strength, forged metal head with a multi-step hammer-tone finish and integrated holes for improved flow of materials. The forged steel head is attached with dual stainless steel rivets, and the tool includes an integrated hanging hole for easy storage. The fiberglass handle is resistant to environmental damage and has an over-molded end-grip and mid-grip for better leverage and two-handed use.",
        "category": "/Building Materials/Concrete, Cement & Masonry/Concrete Tools"
    },
    {
        "tool": "20V MAX* 14 in. Folding String Trimmer",
        "description": "This 20V MAX* 14 in. String Trimmer includes a folding hinge mechanism to reduce its length by 40%** for easier storage and portability. The QuickLoad™ Spool facilitates straightforward line replacement for the DEWALT 0.080 in. line. The 14 in. cutting swath covers a broad area in one pass. The variable speed trigger provides power control, while the Hi/Lo speed control switch assists with performance and runtime management.",
        "category": "/Outdoors/Outdoor Power Equipment"
    },
    {
        "tool": "8 ft Fiberglass Twin Front Step Ladder",
        "description": "This DEWALT 8 ft. Fiberglass Twin Front Step Ladder features a 300lb load capacity per side, allowing two people to use it simultaneously, one on each side. It includes a top with magnet and tool slots, heavy duty boots, an impact absorption system, and a 25% larger step surface.",
        "category": "/Building Materials/Ladders/Step Ladders"
    },
    {
        "tool": "Rotex™ Protective Eyewear",
        "description": "The DEWALT DPG103 ROTEX™ Safety Glass features a lightweight frame and flexible temples with soft, rubber grips for a comfortable and secure fit.",
        "category": "/Safety Equipment/Protective Eyewear/Safety Glasses"
    }

Can I use it with ModernBert models?

Yes, just use classifier = AdaptiveClassifier("answerdotai/ModernBERT-base")

Can I add new documents and categories at runtime? So that they be used to classify new documents?

Yes, that is the main benefit of using the adaptive-classifier. See the example_usage.py file where a new class is added to an existing classifier here - https://github.com/codelion/adaptive-classifier/blob/main/examples/basic_usage.py#L85

Can it be used for 10000 or 100000 documents?

Yes, the LLM router adaptive-classifier was trained on the routellm dataset - https://huggingface.co/datasets/routellm/gpt4_dataset it has 109k items. You can see the training script in the repo - https://github.com/codelion/adaptive-classifier/blob/main/scripts/eval_llmrouter_classifier.py

[dibu28]

What is the mininum number of documents for each class needed?

[dibu28]

I've tried now your example. and changed model

like this:
classifier = AdaptiveClassifier("answerdotai/ModernBERT-base")

but got this error:
ValueError: The checkpoint you are trying to load has model type modernbert but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

[codelion]

You need to update your transformers version. It was only introduced in version 4.48.0 - https://github.com/huggingface/transformers/releases/tag/v4.48.0

Here is a colab showing that it works with latest transformers - https://colab.research.google.com/drive/15pU1bgz70mK5kTMtIamYVhxbO0VEBFTV?usp=sharing

[dibu28]

Tried link you've sent it is also doesn't work:

[codelion]

I added a cell in the notebook before this line, what do you see when you execute that? can you confirm that transformers version is >= 4.48.1

[dibu28]

Yes it is working now in both notenooks. Thank you!

[dibu28]

But I have strange results in this notebook:
https://colab.research.google.com/drive/1VyxVubB8LXXES6qElEYJL241emkV_Wxc?usp=sharing#scrollTo=hezCEDh9Sa4k

If i use Smaller model I have better Validaion score:
For example for Distilber I have 72% for bert 65% and for ModernBert only 45%. Is it as it should be?

[codelion]

If we change the base model we may need to fiddle around with some of the other config options to get better results. In my limited testing I also didn't find ModernBERT to be better for this task.

ModernBERT is built for long texts (8,192 tokens) but the product descriptions are very short (~21 words).
Larger model + small dataset = higher chance of overfitting. DistilBERT's smaller size naturally prevents this.