Tutorial on how to create a prompt extension?

[AlienRenders]

I want to create an extension that will auto-BREAK my prompt. Too often, I use someone else's prompt or if it goes over 75, it will split your prompt and it won't understand what the part that got split means because. For example, if it splits anywhere between "white background"... say in between the words or even in the middle, the first 75 block will just see "white" while the second block will start with "background" and it won't have a clue what to do with either of those tokens.

So I want to look at commas and periods and insert BREAK's at the proper locations to keep phrases between commas and periods intact.

I saw examples how to get the prompt, but not sure how to pass it on to the "engine" without touching the prompt itself. OTOH, I could have a button that manually fixes it. Either way is fine, but I have no clue how to do either (ie. create the button). But I can do the parsing. Having a button where the stylesheet buttons are would be fine, but having it be automatic in the background would be best.

I also need access to the code that can count the tokens.

I guess the style extension is the best one to look at. I've looked at that and it's a good start, but is overcomplicated for what I want to do. It also doesn't explain how to get access to the token counter code.

Any help would be appreciated.

[Haoming02]

if it splits anywhere between "white background"

This should rarely happens if the prompt contains adequate commas, thanks to this setting

in tokens - for texts shorter than specified, if they don't fit into 75 token limit, move them to the next 75 token chunk

[AlienRenders]

Oh, it already exists. Thank you! But I still get different results. In fact, many people are starting to use BREAK to get better results. Yeah, I just tried it. That setting doesn't work. And the token counter widget certainly doesn't take that setting into account.

[AlienRenders]

I think I'd just like that setting to work when it's delimited by commas and periods. I'm looking through the code where it's applied to see if there's a bug. Comments say it only works on commas. You'd think if I replace a comma just before the 75th token with a BREAK that I would get identical results. I don't. Never have. Is there a reason this would happen?

[AlienRenders]

Ok, I figured out what's going on. The comma at the end of the previous chunk is left in the chunk. When I manually replaced the comma with a BREAK, the comma is gone. I just added a comma before the BREAK and I get identical output. Strange that a comma affects the output like that. Same behavior for embeddings. It leaves the comma in the previous chunk. I suppose it's not wrong per se. Locally, I'm gonna change it to remove the comma and I'm happy. Thanks for pointing me in the right direction!

I suppose I would want to skip commas at the start of a chunk as well. I'm going to add that now.