It would be convenient to be able to register your own Chunkers

[aropb]

Currently, the creation of a Chunker instance is strictly embedded in the code (SummarizationHandler, TextPartitioningHandler).
It would be very convenient to be able to register your Chunkers instead of the standard ones and then use them through dependencies.

It is also important to be able to use your own Tokenizer (instead of CL100KTokenizer()).

The CL100KTokenizer is currently being created by default in SummarizationHandler, TextPartitioningHandler for default Chunkers.

Thanks.

[dluc]

The recommended approach is to load and use custom handlers. When loading files, the "steps" parameter allows to choose which handlers to execute, which in turns allows to customize all the ingestion aspects: extraction, chunking, storage, etc.

[aropb]

That's what I'm doing now, but why rewrite the entire handler if you only need to replace the chunker.

[dluc]

Custom handlers can be added and consumed using the docker image and service implementation, withouth the need to rebuild and compile the solution. Replacing chunkers requires new configuration settings with a custom implementation or using DI, which is available only in the serverless version. If you’re working with the serverless code, you can modify the underlying code directly since it’s fully customizable.

[aropb]

Of course, I have found a solution and I will do as I need to. But if you want to improve the product and make it flexible, then add Chunker configuration using DI.