[HF][streaming][1/n] Text Summarization

[rossdanlm]

[HF][streaming][1/n] Text Summarization

TSIA

Adding streaming functionality to text summarization model parser

Test Plan

Rebase onto and test it with 11ace0a.

Follow the README from AIConfig Editor https://github.com/lastmile-ai/aiconfig/tree/main/python/src/aiconfig/editor#dev, then run these command

aiconfig_path=/Users/rossdancraig/Projects/aiconfig/cookbooks/Gradio/huggingface.aiconfig.json
parsers_path=/Users/rossdancraig/Projects/aiconfig/cookbooks/Gradio/hf_model_parsers.py
alias aiconfig="python3 -m 'aiconfig.scripts.aiconfig_cli'"
aiconfig edit --aiconfig-path=$aiconfig_path --server-port=8080 --server-mode=debug_servers --parsers-module-path=$parsers_path

Then in AIConfig Editor run the prompt (it will be streaming format by default)

Screen.Recording.2024-01-10.at.01.16.45.mov

---

Stack created with Sapling. Best reviewed with ReviewStack.

• [HF][5/n] Image2Text: Allow base64 inputs for images #856
• [HF][streaming][4/n] Image2Text (no streaming, but lots of fixing) #855
• [HF][streaming][3/n] Text2Speech (no streaming, but updating docs on completion params) #854
• [HF][streaming][2/n] Text Translation #853
• -> [HF][streaming][1/n] Text Summarization #851

[saqadri]

Do these just signify the start and end of the stream?

[rossdanlm]

Not quite! I'm really not sure why they show up for a few models while for TextGeneration this isn't needed.

I saw something like </s><s>Streaming Text</s> But yea just removed so it looks clean

[saqadri]

This is a very complicated logic. Ideally would prefer to break this out into if/else or create a helper function for this, but not ship-blocking if it works. I just am struggling to think how this would work.

For e.g. if no options is specified, stream will be false, even if there is stream in completion_data. Is that correct behavior?

[rossdanlm]

if no options is specified, stream will be false, even if there is stream in completion_data. Is that correct behavior?

That is correct! The reasoning for this is that stream isn't really a model setting, it's an inference setting which we would need to pass in a stream callback, so streaming without it doesn't really make sense if it's not included (where would you stream out to?).

Sure I made a task to split this into a helper function: #861