[New Component] Full-Duplex Intercom for ESPHome — ESP32 two-way audio with AEC, Voice Assistant coexistence, and Home Assistant integration

[n-IA-hane]

[New Component] Full-Duplex Intercom for ESPHome — ESP32 two-way audio with AEC, Voice Assistant coexistence, and Home Assistant integration

Update: This proposal replaces the original "UDP Intercom" post. The go2rtc approach worked well for local use, but I've since developed a new architecture that simplifies setup and adds features that weren't possible with UDP/WebRTC.

---

ESPHome Intercom API — a full-duplex, real-time, bidirectional audio intercom for ESP32 devices. Supports echo cancellation (AEC), runs alongside Voice Assistant + Micro Wake Word on the same device, and uses Home Assistant as a PBX hub for multi-device calling.

Doorbell, baby monitor, room-to-room intercom — one component, zero external dependencies.

Repository: https://github.com/n-IA-hane/intercom-api

Idle

Calling

Ringing

In Call

From UDP Intercom to Intercom API

The original UDP Intercom worked well on local networks with go2rtc. This new version takes a different approach to unlock additional capabilities:

• Remote access out of the box — audio flows through HA's WebSocket, so Nabu Casa / reverse proxy / VPN all work without extra configuration
• Simpler setup — no go2rtc server, no ffmpeg pipelines, no STUN/TURN
• ESP-to-ESP calls — devices can call each other through HA acting as a PBX, which wasn't possible with the go2rtc-centric approach
• Voice Assistant coexistence — intercom and VA share the same mic/speaker on the same device

Architecture

┌─────────────┐         ┌─────────────┐         ┌─────────────┐
│   Browser   │◄──WS───►│     HA      │◄──TCP──►│    ESP32    │
│  (Card)     │         │  (relay +   │  6054   │  (intercom  │
└─────────────┘         │  signaling) │         │   _api)     │
                        └──────┬──────┘         └─────────────┘
                               │
                        ┌──────┴──────┐
                        │   ESP32 #2  │
                        │  (intercom  │
                        │   _api)     │
                        └─────────────┘

Features

• Full-duplex audio — talk and listen simultaneously (16kHz 16-bit mono PCM)
• Two operating modes:
  • Simple: Browser ↔ HA ↔ ESP (doorbell, baby monitor)
  • Full: ESP ↔ HA ↔ ESP (PBX-like intercom between rooms)
• Echo Cancellation (AEC) — built-in ESP-SR AEC with multiple quality modes
• Voice Assistant compatible — coexists with VA + Micro Wake Word on the same device, same microphone
• Auto Answer — configurable automatic call acceptance
• Volume/Gain control — persistent settings saved to flash
• Contact management — auto-discovery of intercom devices via HA
• Caller ID — shows who's calling
• Ringing timeout — auto-decline if not answered
• Event callbacks — on_incoming_call, on_outgoing_call, on_streaming, on_idle, on_hangup, on_call_failed
• Remote access — works through any HA remote access method (Nabu Casa, VPN, reverse proxy)

Voice Assistant Coexistence

The intercom runs alongside Voice Assistant and Micro Wake Word on the same ESP32. You can trigger the wake word and talk to your assistant during an active intercom call. The AEC prevents the TTS audio from feeding back into the intercom stream.

Bundled Components

intercom_api — The Intercom Engine

TCP server with FreeRTOS tasks for real-time audio. Handles call state machine (Idle → Outgoing → Ringing → Streaming → Idle), audio I/O, protocol messaging.

i2s_audio_duplex — Full-Duplex Single-Bus I2S

Standard ESPHome i2s_audio cannot drive mic + speaker on the same I2S bus simultaneously. i2s_audio_duplex solves this for codecs like ES8311, ES8388, WM8960:

• True full-duplex on a single I2S port
• Built-in AEC integration with digital feedback (sample-accurate reference)
• Dual mic paths: raw (for MWW) + AEC-processed (for VA/intercom)
• Reference counting for multi-consumer mic sharing

esp_aec — Acoustic Echo Cancellation

Lightweight AEC using Espressif's ESP-SR library. Multiple modes from voip_low_cost to sr_high_perf. Configurable filter length.

intercom_native — Home Assistant Integration

Custom component that registers WebSocket API, creates device sensors, and manages the auto-bridge for ESP↔ESP calls.

Lovelace Card

Custom card with:

• AudioWorklet for low-latency browser audio
• Full call controls (call, answer, decline, hangup, volume)
• Contact selector for Full mode
• Real-time state display

Example Configuration

Minimal (Simple Mode — doorbell)

external_components:
  - source:
      type: git
      url: https://github.com/n-IA-hane/intercom-api
      ref: main
      path: esphome_components
    components: [intercom_api, esp_aec]

intercom_api:
  id: intercom
  mode: simple
  microphone: my_mic
  speaker: my_spk

Full Mode (ESP ↔ ESP intercom)

intercom_api:
  id: intercom
  mode: full
  microphone: mic_aec
  speaker: intercom_speaker
  ringing_timeout: 30s

  on_incoming_call:
    - light.turn_on:
        id: led
        effect: "Ringing"
  on_streaming:
    - light.turn_on:
        id: led
        effect: none
  on_idle:
    - light.turn_off: led

Single-Bus Codec (ES8311 with i2s_audio_duplex)

external_components:
  - source:
      type: git
      url: https://github.com/n-IA-hane/intercom-api
      ref: main
      path: esphome_components
    components: [intercom_api, i2s_audio_duplex, esp_aec]

i2s_audio_duplex:
  id: i2s_duplex
  i2s_lrclk_pin: GPIO45
  i2s_bclk_pin: GPIO9
  i2s_mclk_pin: GPIO16
  i2s_din_pin: GPIO10
  i2s_dout_pin: GPIO8
  sample_rate: 16000

microphone:
  - platform: i2s_audio_duplex
    id: mic_raw
    i2s_audio_duplex_id: i2s_duplex
    channel: right
  - platform: i2s_audio_duplex
    id: mic_aec
    i2s_audio_duplex_id: i2s_duplex
    aec: true

speaker:
  - platform: i2s_audio_duplex
    id: hw_speaker
    i2s_audio_duplex_id: i2s_duplex

Tested Hardware

Device	Microphone	Speaker	I2S Mode	VA/MWW
ESP32-S3 Mini	SPH0645	MAX98357A	Dual bus (i2s_audio)	Yes
Xiaozhi Ball V3	ES8311	ES8311	Single bus (i2s_audio_duplex)	Yes

Audio Format

Parameter	Value
Sample Rate	16000 Hz
Bit Depth	16-bit signed PCM
Channels	Mono
Round-trip Latency	< 500ms

TCP Protocol (Port 6054)

Binary protocol with 4-byte header: [Type, Flags, Length_Lo, Length_Hi]

Code	Name	Description
0x01	AUDIO	PCM audio data
0x02	START	Start call (includes caller name, no_ring flag)
0x03	STOP	End call
0x04	PING	Keep-alive
0x05	PONG	Keep-alive response
0x06	ERROR	Error notification

Status

This is actively developed and used daily. Current version: v2.0.3.

Full documentation, installation guide, and example YAML configurations available at:
https://github.com/n-IA-hane/intercom-api

Questions for Maintainers

@rmeissn raised a great point about decoupling audio processing from the intercom logic. Currently the repo includes:

1. intercom_api — the intercom engine (TCP + call state machine)
2. i2s_audio_duplex — full-duplex single-bus I2S driver
3. esp_aec — AEC wrapper

These are already separate ESPHome components and can be used independently. i2s_audio_duplex in particular fills a real gap in ESPHome — many codecs (ES8311, ES8388, WM8960) use a single I2S bus, and there's no native ESPHome solution for full-duplex on a single bus.

Questions:

1. Is there interest in upstreaming any of these components? (i2s_audio_duplex especially seems broadly useful)
2. Should esp_aec be extended to support the full ESP-AFE pipeline (NS + AGC + BSS)? Or keep it lightweight?
3. What architectural feedback do you have?

Happy to refactor as needed.

[nagyrobi]

Any option to also have direct intercom between two (or more) ESPs? Even if there's no HA or HA is down?

Simply with static IPs, being able to "shout in" from a microphone-based ESP to one or more speakers. Like when in a mall when they make an announcement, without having to "answer" a "call" (or auto-answer it).

[n-IA-hane]

It could be a good idea. I’ll try to work on it over the next few days; I don’t think it should be that hard to implement.

[n-IA-hane]

Hi, I'm actively working on it these days. I managed to get two-way audio between ESPs. The component isn't easy to implement. I'm using my Claude Pro subscription with Claude Code, which speeds up development dramatically. However, in the latest versions, I've unfortunately noticed that it has implemented a lot of logic within the add-on, things that could have been easily implemented in Lambda, or in the native logic of ESPHome. In the next few days, I want to separate all these additions and have a standard component compatible with all configurations. We've also created a component that queries MDNS to discover other ESPs that have the same component. This MDNS discovery could also easily be another platform and add-on instead of being hardcoded in the udp_intercom. With your ESP, you can see the list of other ESPs with the same firmware, dynamically select one, press it, and open two-way streaming with the other ESP, like a walkie-talkie. I also need to implement one-way audio properly. I think it might be helpful not to have to use a separate speaker from an ESP, perhaps you just want to do ambient listening. And at this point, I'd also like to do some multicast tests. If you speak from one, audio will be played back to many.

[nagyrobi]

Wonderful thoughts!

[flubshi]

@n-IA-hane I like your idea. In the home assistant community forum, multiple people already requested this and were waiting for a solution. :)

From a user perspective, it would be great if it could somehow be combined with voice assistants. Like: "Please call room xy". Not sure whether it is possible to run your component on the same microphone the voice assistant is listening (could be stopped while intercom is running).

Maybe @kahrendt or @jesserockz can provide feedback, how to integrate your existing implementation.

[rmeissn]

Just to let you know, there is also the following code available, which enables microphone post-processing on the ESP itself as a component, creating a seperate (post-processed) microphone:

https://github.com/craigmillard86/waveshare-s2-audio_esphome_voice/tree/rework-branch/components/esp_afe

It might be nice to decouple your approach from audio post-processing like implemented as in the code above.

I'm mainly interested in parts of your work, namely the single I2S bus, driving a speaker and microphone at the same time, but also on microphone post-processing (like Noise Cancelling and Auto Gain).

[n-IA-hane]

Thanks @rmeissn for pointing that out! I've had a look at Craig's ESP-AFE component — interesting approach using the full esp-sr AFE pipeline (NS + AGC + BSS) rather than just AEC standalone.

I've just updated the proposal above to reflect the current state of the project, which has evolved significantly from the original UDP Intercom. The new Intercom API architecture is TCP-based with HA as a PBX hub, and the components are already decoupled:

1. intercom_api — the intercom engine (TCP server, call state machine, audio I/O)
2. i2s_audio_duplex — full-duplex single-bus I2S driver for codecs like ES8311, ES8388, WM8960
3. esp_aec — lightweight AEC wrapper around ESP-SR

Each one is a separate ESPHome component and can be used independently. i2s_audio_duplex is the one that solves the single-bus problem you mentioned — it handles simultaneous mic + speaker on one I2S port, with built-in AEC integration using digital feedback (sample-accurate speaker reference), dual mic paths (raw for MWW + AEC-processed for VA/intercom), and reference counting for multi-consumer mic sharing.

Regarding NS and AGC — our esp_aec is intentionally lightweight (just echo cancellation). Extending it to wrap the full ESP-AFE pipeline is something we've considered but haven't prioritized yet, since the AEC alone handles the main problem (speaker feedback into mic during calls and TTS). That said, if there's interest, it could make sense as a separate esp_afe component alongside the existing esp_aec.

Would love your feedback on the updated proposal — especially on i2s_audio_duplex since that seems to be the part most relevant to your use case.