StackOverflow Exception

[Fantom579]

Raclettes - Recettes Inedites.pdf

This bad file invoke stack overflow exception v0.1.12

[BobLd]

@Fantom579 thanks a lot for opening the issue and sharing the document.

From what I see, there's no straightforward fix to the bug. The solution would involve refactoring of some part of the code.

The stack overflow issue stems from ArrayTokenizer and DictionaryTokenizer TryTokenize() method calling CoreTokenScanner.MoveNext(), which itself calls TryTokenize()

From my point of view, the code should be refactored to account for stack depth, which is not easily doable at the moment

[Fantom579]

Is it possible to do some kind of emulation or pre-check of the document to avoid crashing the application process?

[Fantom579]

At the moment I haven't found any way other than running a separate process, but that's very expensive.

[BobLd]

@Fantom579 I've created a PR to fix the issue. Do you have a way to test the changes on your end? Possibly on a large amount of documents to assess any side effect

[Fantom579]

I install 0.1.13-alpha-20251203-c70b3

And the library still gives a stack overflow

[BobLd]

@Fantom579 the fix is not yet in the main branch, so this is expected.

Can you try the library based on the branch in this PR: #1220

I can create a NuGet package if need be

[Fantom579]

@Fantom579 the fix is not yet in the main branch, so this is expected.

Can you try the library based on the branch in this PR: #1220

I can create a NuGet package if need be

Please create a nuget package, I don't see the new function in the source code in the repository.