Hybrid chunking using OpenAI Tokenizer

[mohammedfaisal]

The documentation explains how to configure and use OpenAI Tokenizer with hybrid chunker in python.

       import tiktoken

        from docling_core.transforms.chunker.tokenizer.openai import OpenAITokenizer

        tokenizer = OpenAITokenizer(
                   tokenizer=tiktoken.encoding_for_model("gpt-4o"),
                   max_tokens=128 * 1024,  # context window length required for OpenAI tokenizers
        )
        
        chunker = HybridChunker(
                tokenizer=tokenizer,
                merge_peers=True,  # optional, defaults to True
        )
        chunk_iter = chunker.chunk(dl_doc=doc)
        chunks = list(chunk_iter)

How to do the same operation using docling-java ?
The HybridChunkerOptions.builder().tokenizer() seems support only HuggingFace models.

[edeandrea]

I'll have to look at the docling-serve documentation. If it's exposed in the api then we probably just missed it. If it isn't part of the docling-serve API then they will need to add it there first.

[edeandrea]

Just checking quickly, but I don't see anything in https://github.com/docling-project/docling-serve/blob/main/docs/usage.md that describes a way to use an OpenAI transformer.

@dolfim-ibm is that documentation correct?