SpanCat category limit

[e-e]

Is there a limit to how many different categories can be trained? If there was a need to go beyond this limit, what are some approaches that can be taken?

[polm]

There's no hardcoded limit on the number of labels or anything. I guess if you start rubbing up against INT_MAX or something you'll have trouble, but I suspect you'd have data issues long before that.

How many labels did you have in mind?

[e-e]

500? 1000 maybe? I guess I'm asking more about potential accuracy problems.

[polm]

Ah, OK. It's true that once you get to that many categories things just get hard. The first step is to make sure you have enough coverage of all categories. If you do and thing still don't work, what I've done before is use a hierarchical classifier - so first you classify into broad categories, and then you classify into finer categories, and so on down to the smallest categories. This works well if your categories are already in a tree, like for products on Amazon, for example (though if any of your categories are weird you'll find out really quickly).

I think we have plans to make a hierarchical classification component but they're still in the early stages now.

You might want to look at the literature on "extreme classification", which often has many more categories than 1000.

[e-e]

That's helpful, thank you!