maxTokenSize, is there also minTokenSize?

[ErwinAI]

I noticed that sometimes, when combineChunks is set to true, some chunks contain just a few characters/tokens.
Would it be an idea to add minTokenSize and perhaps anything under that, to be added to the next chunk?

Not a high priority but might be a nice setting to make sure chunks have sufficient amounts of text in them.

[jparkerweb]

@ErwinAI would you be able to put together an example to work off of?

• text file (or blob of text to chunk)
• all the chunking settings

You can spin-up the new Web UI to help with this :)
https://github.com/jparkerweb/semantic-chunking/tree/main/webui

I have a demo server setup for the Web UI you could try as well:
https://chunking.dyndns.org/

Let me know if that isn't clear, but would really help me nail down the requirements and have a sanity check to test my changes against.

Additionally, you might get the effect you are looking for if you lower the combineChunksSimilarityThreshold value, as it will combine neighboring chunks (up to the max defined chunk token size).

Thanks :)