[GSoC 2026] Exploring Hands-Free Multimodal Voice Mode

[Devnil434]

Hi Bryan Sir,

I’m very interested in working on the Hands-Free Multimodal Voice Mode project for GSoC 2026 and have started exploring the Gemini CLI codebase to understand how this could be integrated properly.

So far, I have:

• Reviewed the agent lifecycle and streaming response handling in gemini-cli-core
• Studied how current streaming outputs are managed inside the CLI
• Started prototyping:
  • Real-time microphone capture using PCM streams in Node.js
  • WebSocket-based streaming with AbortController cancellation
  • A basic Voice Activity Detection (VAD) experiment for push-to-talk mode

Before moving further, I’d love to clarify a few architectural questions to ensure alignment with the project vision:

---

🔧 Architecture Questions

1. Should the bidirectional audio streaming be abstracted behind a transport layer in cli-core, or integrated directly into the existing agent streaming pipeline?
2. Would you prefer the voice mode to be implemented as a modular plugin (opt-in feature), or deeply integrated into the main agent loop?
3. Do you envision a state-machine-based design (Idle → Listening → Processing → Speaking) for managing voice interaction states?

---

⚡ Performance & UX Considerations

4. Are there specific latency targets expected for conversational responsiveness?
5. Should streaming speech output overlap with token streaming, or wait for complete responses?
6. Would push-to-talk be the default interaction mode initially, or continuous listening with VAD?

---

🛡️ Scope & Design Clarifications

7. Is wake-word detection ("Hey Gemini") expected within scope, or optional?
8. Should voice mode be experimental behind a feature flag during early rollout?
9. Are there privacy expectations around transcript logging or mic access prompts?

---

I want to ensure the design is scalable, interruptible, and consistent with the existing CLI architecture before drafting a detailed proposal.

Any guidance or high-level direction would be greatly appreciated.

Thank you for your time!

Best regards,
Nilanjan

[Devnil434]

@bdmorgan sir, Update: I started exploring how streaming responses are handled in the CLI.
I’m currently prototyping a mic capture pipeline using PCM streams and
experimenting with a simple VAD-based push-to-talk mechanism.

I’m also drafting a state-machine design for managing voice interaction
states (Idle → Listening → Processing → Speaking).