Skip to content

Commit ba26df9

Browse files
HardCodeDev777HardCodeDev777
committed
Add docs files
1 parent abf1bb8 commit ba26df9

File tree

13 files changed

+1164
-1
lines changed

13 files changed

+1164
-1
lines changed

‎.gitignore‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
.venv/
2-
.idea/
2+
.idea/
3+
/site

‎docs/index.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
<div style="text-align: center;">
2+
<img src="media/logo.png">
3+
</div>
4+
5+
#
6+
7+
> **Make your Unity characters hear, think, and talk — using real voice AI. Locally. No cloud.**
8+
9+
---
10+
11+
UnityNeuroSpeech is a lightweight and open-source framework for creating **fully voice-interactive AI agents** inside Unity.
12+
It connects:
13+
14+
- 🧠 **Whisper** (STT) – converts your speech into text
15+
- 💬 **Ollama** (LLM) – generates smart responses
16+
- 🗣️ **XTTS** (TTS) – speaks back with *custom voice + emotions*
17+
18+
All locally. All offline.
19+
No subscriptions, no accounts, no OpenAI API keys.
20+
21+
---
22+
23+
## 🚀 What can you build with UnityNeuroSpeech?
24+
25+
- 🎮 AI characters that understand your voice and reply in real time
26+
- 🗿 NPCs with personality and memory
27+
- 🧪 Experiments in AI conversation and narrative design
28+
- 🕹️ Voice-driven gameplay mechanics
29+
- 🤖 Interactive bots with humanlike voice responses
30+
31+
---
32+
33+
## ✨ Core Features
34+
35+
| Feature | Description |
36+
|--------|--------------------------------------------------------------------------------------------|
37+
| 🎙️ **Voice Input** | Uses [whisper.unity](https://github.com/Macoron/whisper.unity) for accurate speech-to-text |
38+
| 🧠 **AI Brain (LLM)** | Easily connect to any local model via [Ollama](https://ollama.com) |
39+
| 🗣️ **Custom TTS** | Supports any voice with [Coqui XTTS](https://github.com/coqui-ai/TTS) |
40+
| 😄 **Emotions** | Emotion tags (`<happy>`, `<sad>`, etc.) parsed automatically from LLM |
41+
| 🎛️ **Agent API** | Subscribe to events like `BeforeTTS()` or access `AgentState` directly |
42+
| 🛠️ **Editor Tools** | Create, manage and customize agents inside Unity Editor |
43+
| 🧱 **No cloud** | All models and voice run locally on your machine |
44+
| 🌐 **Multilingual** | Works with over **15+ languages**, including English, Russian, Chinese, etc. |
45+
46+
---
47+
48+
## 🧪 Built with:
49+
50+
- 🧠 [`Microsoft.Extensions.AI`](https://learn.microsoft.com/en-us/dotnet/machine-learning/) (Ollama)
51+
- 🎤 [`whisper.unity`](https://github.com/Macoron/whisper.unity)
52+
- 🐍 [Python Flask server](other/server.md) (for TTS)
53+
- 🧊 [Coqui XTTS model](https://github.com/coqui-ai/TTS)
54+
- 🤖 Unity 6
55+
56+
---
57+
58+
## 📚 Get Started
59+
60+
- 🏁 [Quick Start](quickstart.md)
61+
- ⚙️ [Configure Settings](unity/configure-settings.md)
62+
- 🧠 [Create Agent](unity/creating-agent.md)
63+
- 🔌 [Agent API](unity/agent-api.md)
64+
- 🔊 [Build Your Game](unity/building-game.md)
65+
- 🐍 [About Python Server](other/server.md)
66+
- ❓ [FAQ](other/faq.md)
67+
68+
---
69+
70+
## 😎 Who made this?
71+
72+
UnityNeuroSpeech was created by [HardCodeDev](https://github.com/HardCodeDev777) —
73+
indie dev from Russia who just wanted to make AI talk in Unity.
74+
75+
---
76+
77+
## ⭐ Star it on GitHub
78+
[👉 github.com/HardCodeDev777/UnityNeuroSpeech](https://github.com/HardCodeDev777/UnityNeuroSpeech)

‎docs/media/favicon.png‎

6.68 KB
Loading

‎docs/media/logo.png‎

40.6 KB
Loading

‎docs/other/faq.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# ❓ FAQ
2+
3+
---
4+
5+
## ⚙️ General
6+
7+
---
8+
9+
### Why use Coqui XTTS as the TTS model?
10+
11+
Three main reasons:
12+
13+
- **Custom Voices**.
14+
15+
You can use any voice you want: your own, Cyn, Vito Corleone, Justin Timberlake — whatever.
16+
I believe that’s way more flexible than static, pre-generated voices.
17+
18+
- **License**.
19+
20+
Some TTS models have extremely restrictive licenses. For example, Silero TTS is under CC BY-NC-SA 4.0, which means:
21+
- You can’t use it commercially.
22+
- Even worse: I’d be forced to release this entire framework under the same license.
23+
That’s a no-go.
24+
25+
- **Simplicity**.
26+
27+
The Python server is **just ~100 lines of code**.
28+
It supports 10+ languages out of the box.
29+
You send a UnityWebRequest with text — it sends back .wav.
30+
That’s it. No ONNX, no weird tensors, no converting text into token IDs, etc.
31+
32+
33+
But if you hate Python or Coqui XTTS, you can integrate your favorite model.
34+
Modularity is the point. I won’t stop you 🤠
35+
36+
---
37+
38+
## 🐍 Python
39+
40+
---
41+
42+
### Why use Python for TTS?
43+
44+
I didn’t really have a choice.
45+
46+
**Coqui XTTS only offers a Python API**, and there are no stable C#, C++, C, TypeScript, JavaScript or Go bindings.
47+
If there had been a solid alternative in another language, I would’ve gladly used it — but this is what we’ve got.
48+
49+
---
50+
51+
### Why is Python.exe using so much RAM?
52+
Because it’s running a full neural TTS model — locally — in Python.
53+
That’s the price you pay for realistic voices and multilingual support.
54+
If it bothers you — try switching to smaller models or optimize the server logic yourself.
55+
56+
---
57+
58+
## ❗ Most Important Question
59+
60+
---
61+
62+
### Why is it so hard to configure and get started?
63+
64+
A totally fair question — especially after you’ve gone through all the setup steps and seen how many pieces are involved.
65+
66+
Here’s the truth:
67+
> UnityNeuroSpeech is the first — not just Unity, but game development framework — that allows you to talk to AI directly inside your game.
68+
69+
---
70+
71+
The only out-of-the-box solution for Unity in this whole stack is [whisper.unity](https://github.com/Macoron/whisper.unity).
72+
And even then — you still need to create a separate `SetupWhisperPath.cs` script to make it work properly in builds. Yes, you can try using **StreamingAssets**, and whisper.unity supports that.
73+
But I personally prefer not to scatter files across multiple Unity folders.
74+
If that’s critical for your project — feel free to try it with StreamingAssets.
75+
76+
---
77+
78+
Now let’s talk about Text-to-Speech...
79+
80+
Frankly, there are no really “clean” or “easy” TTS solutions — not just for Unity, **but even for C# as a whole**.
81+
(If you’re curious about the deep rabbit hole of TTS licensing and tech, check out the [Python Server](server.md) page.)
82+
83+
---
84+
85+
As for **Ollama**, I integrated it into Unity myself — using Microsoft’s `Microsoft.Extensions` libraries.
86+
87+
---
88+
89+
Yes — it does feel like a lot at first.
90+
91+
But don’t worry — **UnityNeuroSpeech is actively being improved**, and with every update, the setup process will become simpler and more automated.
92+
93+
---
94+
95+
And maybe one day, this kind of tech will be built into Unity out of the box.
96+
You’ll click a button, and boom — talking AI agents everywhere.
97+
98+
But when that day comes... **it won’t be unique anymore.**
99+
100+
So if you want to stand out with a game that features **real, emotional voice interaction powered by AI** — now’s your chance.
101+
Go for it 😎

0 commit comments

Comments
 (0)