Add Raven-avatar support to SillyTavern

[Technologicat]

It would be particularly interesting to introduce Raven-avatar to replace the discontinued Talkinghead mode of the Character expressions extension in SillyTavern. Basically, Raven-avatar is an upgraded version of that, with many new features.

Since SillyTavern-Extras (which provided Talkinghead) was discontinued, the Talkinghead feature has been removed from the SillyTavern frontend.

In SillyTavern/SillyTavern#4034, it was decided that if the new avatar is added to SillyTavern, this should be done as a new extension.

• Minimally, the ST extension needs:

  • A canvas to render the video stream in, aligned to the bottom edge of the window (since the avatar is a cowboy shot)
    • Get the image size from the video stream, it depends on upscaler settings.
  • Start, pause, resume, stop
    • Like the final version of Talkinghead, pause when the ST window is minimized, and resume when no longer minimized, to save GPU when the avatar is not visible.
  • Character emotion driver.
    • Trigger when more LLM output appears. Call the server's classify API to detect the AI character's emotion from recent LLM output (maybe the few last sentences). Then via the avatar API, set that emotion to the avatar.
    • The old Talkinghead used to auto-update its emotion whenever classify was called, but in Raven-avatar, these have been separated, for feature orthogonality. Also, we now support multiple avatar sessions simultaneously, so it's better to be explicit about which one to update.
  • Support for ST's /emote command (set the character's emotion manually)
  • GUI to adjust lipsync time offset (to allow the user to adjust AV sync as needed)
  • A settings GUI to select the character, the postprocessor settings, and the emotion templates.

• For cel blending and animefx, the extension needs to automatically send also the character's extra cels.

  Since a JS client can't scan for additional files (like the Python client does), maybe we need to pack the character and its extra cels into a single zip file. This needs some support on the server side to receive zipped character packages, but Python has zip libraries available, so this should be easy. Even when the avatar is used with the Python client, a zip file would be a cleaner way to distribute ready-made characters.

  For bonus points, we could allow optionally including also the postprocessor settings and the emotion templates in the character package.

  I think all the rest (actually configuring the postprocessor and customizing the emotion templates) can be done using raven-avatar-settings-editor and raven-avatar-pose-editor, so we don't need to replicate those in JS.

• The TTS feature of the SillyTavern frontend needs to co-operate with the avatar's lipsync mode.

  Lipsync requires timestamped phoneme data. Currently, the avatar can only get that from the Kokoro-82M TTS that is also served by Raven-server.

  The lipsync driver that actually controls the avatar's mouth is implemented on the client side (in raven.client.tts), because that is where the audio playback occurs.

  I'm open to the idea of extending lipsync to support other TTS implementations, such as the newer, smaller and faster Kitten-TTS (but note Timestamps KittenML/KittenTTS#14), or to various RVC models. We could even extend this to support phoneme detection from bare speech audio (so that lipsync could work with arbitrary voice audio sources, not just TTS). But I don't have the development resources to do any of that, either.

  Long story short: when the avatar is enabled, SillyTavern should call the new tts_speak_lipsynced (in Raven's JS bindings, discussed above) instead of calling its TTS service normally. (It doesn't matter if the avatar is paused.) For a Python example, see how raven.avatar.settings_editor_app uses tts_speak_lipsynced.

This is really cool. I've developed ST extensions before, so I'm very interested in helping with this when I have some free time.

I've reviewed the README and the hardware requirements. Unfortunately, my current setup doesn't meet the minimum requirements for running the core features like the avatar server.

Because of this hardware limitation, I won't be able to contribute to the development at this time. I'm really sorry about that, as I was genuinely excited about this project.

I'll keep following the project's progress. Best of luck with the development!

[Technologicat]

Wait - what kind of hardware do you have?

By disabling unnecessary server modules, it's possible to run the server with much less VRAM. I think the avatar should do fine with 1-2 GB allocated to it. I haven't measured it specifically, though - I'm currently busy developing stuff, so the default config is basically what happens to work on my dev laptop. Server config optimization comes later. :)

In case you have an NVIDIA GPU, but not CUDA 12.x, it should be possible to run with older CUDA versions.

The 12.x requirement is mostly just because that's what I happen to have at the moment - so that's what I've tested with. And this way, I only needed to add one CUDA dependency group to pyproject.toml. We could add alternative dependency groups to support other CUDA versions.

If your GPU is AMD, see #1. In that case I'd be interested in helping out.

If no discrete GPU, then yeah, the THA3 posing engine is unfortunately too heavy for realtime use on a CPU.

Thanks for the detailed reply and the suggestions. You're right in your last point; the issue is that I don't have a discrete GPU. My system is running on integrated graphics, so it won't be able to handle the posing engine.

[Technologicat]

If integrated AMD, might be possible to do something.

But if integrated Intel, then the models will probably have to run on CPU.

I have one more idea: would you be interested in an avatar-based static expression/pose renderer, that could render those expressions from a single static image using THA3?

(In contrast to Character Expressions, which requires a separate image for each of the 28 emotions.)

It could use all the features of the avatar, such as the postprocessor and animefx. The end-to-end latency would be under 1 s.

This idea has been at the back of my mind ever since I developed the final version of Talkinghead, but I've never found the time to work on it.

This shouldn't be too hard to add to the server. It needs something like triggering raven.server.modules.avatar.Animator.render_animation_frame on-demand, without running a continuous animation. With appropriate refactoring, the pose editor (which can run completely offline on CPU) might also benefit from this.

Another, completely different approach would be to replace the THA3 engine with a solution based on a 3D engine, such as Panda3D. I happen to have a private, improved toon shader for that* from way back in the 1.8 days, before Panda3D was on GitHub. But that would require a 3D model as input, instead of a single image, so it's more work for the user to provide a custom character.

(* Actually, some of Raven-avatar's postprocessor filters are PyTorch ports of unpublished code I originally wrote for Panda3D's CommonFilters.)

I have one more idea: would you be interested in an avatar-based static expression/pose renderer, that could render those expressions from a single static image using THA3?

Thanks again for thinking through alternative solutions for my hardware limitations. I've been giving the static pose renderer idea a lot of thought.

It's a powerful concept, but I have a question about the workflow. One could argue that a user could achieve a high-fidelity experience by using the Raven-avatar engine offline to pre-generate a very comprehensive set of expressions (say, 50 or 100 images covering various emotions and intensities), and then simply load them into the standard SillyTavern expressions system.

From your perspective, is the main advantage of the real-time static renderer the ability to generate "in-between" or blended emotions (e.g., 70% happy, 30% surprised) on the fly, which is impossible with a finite set of pre-generated files? Or are there other benefits like dynamic application of post-processing effects that I'm not fully appreciating?

But if integrated Intel, then the models will probably have to run on CPU.

Unfortunately, that's exactly the case. It's an integrated Intel GPU. 😢

Perhaps I could take on the frontend development part, but can't test the GPU-specific features. My main focus right now is on accessibility work for SillyTavern, so my availability will be limited.

[Technologicat]

It's a powerful concept, but I have a question about the workflow. One could argue that a user could achieve a high-fidelity experience by using the Raven-avatar engine offline to pre-generate a very comprehensive set of expressions (say, 50 or 100 images covering various emotions and intensities), and then simply load them into the standard SillyTavern expressions system.

From your perspective, is the main advantage of the real-time static renderer the ability to generate "in-between" or blended emotions (e.g., 70% happy, 30% surprised) on the fly, which is impossible with a finite set of pre-generated files? Or are there other benefits like dynamic application of post-processing effects that I'm not fully appreciating?

That's a fair point. There is indeed already a feature in the pose editor (Batch save, Ctrl+Shift+S) that generates the 28 expressions and saves them as PNGs (along with JSON dumps of the parameters, for reference).

So, what could realtime static rendering give us that this doesn't?

Emotion blending is an interesting idea. I've thought about it, too. The classify module output looks like some kind of softmax anyway, so we could blend the emotion templates using those coefficients (perhaps after a normalization step).

Dynamic postprocessing, yes, this is another possibility. This would allow e.g. activating bloom only if the chat window's background image looks dark but high-contrast enough so that the viewer would expect bloom to occur.

We could also slightly randomize the pose (like the live avatar does - see the sway subsystem), so that the same expression doesn't always render as the exact same image.

Also, animefx? E.g. confusion could initially spawn a question mark, which then vanishes after a second or two (while keeping the pose the same).

Unfortunately, that's exactly the case. It's an integrated Intel GPU. 😢

I see.

Perhaps I could take on the frontend development part, but can't test the GPU-specific features.

That would be perfect.

I'm primarily a pythonista, so my development efficiency as well as knowledge of standard patterns in the JS sphere is limited. I also haven't done much development of ST extensions, except a short stint overhauling the UX of Timelines a while back.

So this could go much faster if you could help with the JS frontend, so that I can concentrate on the server side.

If we go ahead on the static rendering mode, I can add the needed features to the server side. Also, the postprocessor settings editor should probably support that - I can do that, too.

I can also help with testing the integration with the GPU-specific parts.

My main focus right now is on accessibility work for SillyTavern, so my availability will be limited.

No rush.

I'm in the middle of coding a complete LLM frontend for scientific use myself. :P

I actively use ST, too - Raven-librarian is for a slightly different use case, where I need 100% control over its RAG and tool-calling mechanisms, and the like. I'm also curious how a chat frontend turns out, when the nonlinear-timelines philosophy is built into its core from day one.

That sounds like a great plan. I'm happy to help with the JS frontend while you focus on the server-side features. Your willingness to handle the GPU-specific testing makes this a very workable collaboration.

On a related note, thinking about how to get more contributors for a project this ambitious, an idea came to mind. The discussion here on GitHub is a bit quiet, and I feel this project could benefit from wider visibility.

I'm aware of a very large and active Chinese SillyTavern community called 类脑. From what I've heard, it has tens of thousands of members, including many who are technically skilled and passionate about new ST features.

Crucially, they apparently have an international channel where English is used, so communication shouldn't be an issue.

While I'm not a member myself for various reasons, I wanted to pass along the suggestion. If you were to introduce Raven-avatar there, you might find a lot of enthusiasm and potentially more people willing to contribute—whether with development, testing on different hardware, or character creation.

Just a thought, but it seems like a good place to find the manpower this project deserves. I believe they would be very interested in what you're building.

@Technologicat

I am incredibly sorry, but I have to back out of helping with the frontend.

Unfortunately, I no longer have the time available to work on SillyTavern plugins. I feel terrible about this, especially since we just agreed on a plan.

This is no reflection on your project, which I think is amazing. I'm really sorry to let you down and hope you can find someone else to help soon. Best of luck with the development.