通过webrtc连接https://webrtc.espressif.com/，报错ESP_PEER_ERR_NO_MEM

[ClearrSky]

Checklist

• Checked the issue tracker for similar issues to ensure this is not a duplicate
• Read the documentation to confirm the issue is not addressed there and your configuration is set correctly
• Tested with the latest version to ensure the issue hasn't been fixed

How often does this bug occurs?

always

Expected behavior

我想通过在本地的ESP32开启一个webrtc连接，连接到https://webrtc.espressif.com/ 中的房间。
我希望的效果是：我的ESP32设备可以正确推送音视频流到房间中，而我可以通过电脑访问该网址的对应房间看到对应的音视频画面。但是在我开启连接的时候遇到了一些问题

int webrtcHelper::start_webrtc()
{
    uint32_t rand_num = esp_random();  
    // 生成 9 位数 (100000000 ~ 999999999)
    uint32_t nine_digit = 100000000 + (rand_num % 900000000);
    char signal_url_buffer[64];  // 预留足够大缓冲区
    sprintf(signal_url_buffer, "https://webrtc.espressif.com/join/%lu", nine_digit);
    ESP_LOGI(TAG, "room id is %lu", nine_digit);

    if (webrtc) {
        esp_webrtc_close(webrtc);
        webrtc = NULL;
    }

    esp_peer_default_cfg_t peer_default_cfg = {
        .agent_recv_timeout = 100,   // Enlarge this value if network is poor
        .data_ch_cfg = {
            .send_cache_size = 1536, // Should big than one MTU size
            .recv_cache_size = 1536, // Should big than one MTU size
        },
        .rtp_cfg = {
            .audio_recv_jitter = {
                .cache_size = 1024,
            },
            .send_pool_size = 1024,
            .send_queue_num = 10,
        },
        // .data_ch_cfg = {
        //     .send_cache_size = WEBRTC_DATA_CH_SEND_CACHE_SIZE,
        //     .recv_cache_size = WEBRTC_DATA_CH_RECV_CACHE_SIZE,
        // }
    };

    esp_webrtc_cfg_t cfg = {
        .signaling_impl = esp_signaling_get_apprtc_impl(),
        .signaling_cfg = {
            .signal_url = signal_url_buffer,
            .ctx = &webrtc,
        },
        .peer_impl = esp_peer_get_default_impl(),
        .peer_cfg = {
            .audio_info = {
                .codec = ESP_PEER_AUDIO_CODEC_G711A,
            },
            .video_info = {
                .codec = ESP_PEER_VIDEO_CODEC_MJPEG,
                .width = 320,
                .height = 192,
                .fps = 1,
            },
            .audio_dir = ESP_PEER_MEDIA_DIR_SEND_ONLY,
            .video_dir = ESP_PEER_MEDIA_DIR_SEND_ONLY,
            .enable_data_channel = DATA_CHANNEL_ENABLED,
            .video_over_data_channel = false, //  video transfer over data channel
            .no_auto_reconnect = false,
            
            .extra_cfg = &peer_default_cfg,
            .extra_size = sizeof(peer_default_cfg),
            .on_custom_data = nullptr,
        },
    };
    esp_webrtc_media_provider_t media_provider = {
        .capture = capture_handle_,
        .player = NULL,
    };

    int ret = esp_webrtc_open(&cfg, &webrtc);
    if (ret != 0) {
        ESP_LOGE(TAG, "Fail to open webrtc");
        return ret;
    }

    esp_webrtc_set_media_provider(webrtc, &media_provider);
    // Set event handler
    esp_webrtc_set_event_handler(webrtc, webrtc_event_handler, webrtc);

    // Default disable auto connect of peer connection
    esp_webrtc_enable_peer_connection(webrtc, false);

    ret = esp_webrtc_start(webrtc);
    if (ret != 0) {
        ESP_LOGE(TAG, "Fail to start webrtc");
    }

    ESP_LOGI(TAG,"Open webrtc success!");
    return ret;

这是我开启webrtc的代码，为了方便debug，我在等待大约5秒之后(也就是确保获取到了ICE信息之后)设置了esp_webrtc_enable_peer_connection(webrtc, true);

Actual behavior (suspected bug)

实际上在进行连接时，也就是调用esp_webrtc_enable_peer_connection(webrtc, true);之后，发生了报错，返回值为-2，也就是ESP_PEER_ERR_NO_MEM，似乎是内存不够，我在调用该函数之前打印了我的设备内存，看起来还比较充足。请问是我的实现有问题？比如没有正确处理该信令服务器的信息，还是真的内存不足，如果内存不足的话，以我的设置大概需要多少内存？

以及我定位了一下，具体应该是在调用esp_peer_open()函数中的ops->open()函数时出现的错误返回，但是似乎是由于具体实现封装在静态库里了，我无法查看具体的open()是如何实现的，也无法定位内存不足的位置，请您指教

Error logs or terminal output

>> Free Internal RAM: 138859 bytes
>> Largest Free Internal Block: 63488 bytes
>> Free PSRAM: 8256708 bytes
I (8533) MODEL_LOADER: Successfully load srmodels
I (8543) Assets: Applying new assets to display
I (8543) webrtc_helper: room id is 162633973
I (9423) esp-x509-crt-bundle: Certificate validated
I (10613) APPRTC_SIG: result SUCCESS
Initials set to 1
I (10613) HTTPS_CLIENT: HTTP Status = 200, content_length = ld
I (11463) esp-x509-crt-bundle: Certificate validated
Got url:stun:webrtc.espressif.com:3478 user_name: 1758867390:ninefingers psw:KKcqfTm0BwfwcslTkxMYeEyGYMM=
I (12453) HTTPS_CLIENT: HTTP Status = 200, content_length = ld
I (12463) webrtc: Pending connection until user enable
I (12463) APPRTC_SIG: Registering signaling channel.
I (12463) APPRTC_SIG: Connecting to wss://webrtc.espressif.com:8089/ws...
I (12473) websocket_client: Started
I (12473) webrtc_helper: Open webrtc success!
I (12483) Application: STATE: activating
I (12483) Ota: Current version: 2.0.0
I (13103) esp-x509-crt-bundle: Certificate validated
I (14313) APPRTC_SIG: WEBSOCKET_EVENT_CONNECTED
I (14313) APPRTC_SIG: send to remote : {"cmd":"register","roomid":"162633973","clientid":"38724972"}
I (82533) webrtc: ICE info loaded
>> Free Internal RAM: 129619 bytes
>> Largest Free Internal Block: 63488 bytes
>> Free PSRAM: 8188316 bytes
I (82533) esp_peer: ops->open return -2
E (82543) webrtc: Fail to open peer ret -2

我的电脑浏览器在房间内的控制台输出为

1.643: Initializing; server= undefined.
util.js:212 1.643: Initializing; room=undefined.
util.js:212 65.579: Opening signaling channel.
util.js:212 66.066: Retrieved ICE server information.
util.js:212 66.087: Signaling channel opened.
util.js:212 66.158: Joined the room.
util.js:212 66.158: Registering signaling channel.
util.js:212 66.158: Signaling channel registered.
util.js:212 66.579: Got access to local media with mediaConstraints:
  '{"audio":true,"video":{"optional":[{"minWidth":"1280"},{"minHeight":"720"}],"mandatory":{}}}'
util.js:212 66.579: User has granted access to local media.
util.js:212 66.579: Attaching local stream.
util.js:212 66.580: Starting signaling.
util.js:212 66.581: ECDSA certificate generated successfully.
util.js:212 66.581: Creating RTCPeerConnnection with:
  config: '{"bundlePolicy":"max-bundle","rtcpMuxPolicy":"require","iceServers":[{"urls":["stun:webrtc.espressif.com:3478","turn:webrtc.espressif.com:3478"],"username":"1758868168:ninefingers","credential":"p3K1sva1iQvBgo8fny8i9UgNorY="}],"certificates":[{}]}';
  constraints: '{"optional":[]}'.
util.js:212 66.582: Created PeerConnectionClient
util.js:212 66.582: Adding local stream.

Steps to reproduce the behavior

无

Project release version

latest

System architecture

Intel/AMD 64-bit (modern PC, older Mac)

Operating system

Windows

Operating system version

Windows 11

Shell

PowerShell

Additional context

开发板为立创实战派ESP32-S3

[TempoTian]

大概率是忘了注册os adapter了: https://github.com/espressif/esp-webrtc-solution/blob/main/solutions/doorbell_demo/main/main.c#L278
另外确认下 CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL=256 这个配置是否开了，要不可能RAM不够了没分到PSRAM中去。

#include "media_lib_adapter.h"
media_lib_add_default_adapter();

[ClearrSky]

谢谢您，问题解决了，另外我想问目前webrtc支持只发送视频帧而不发送音频吗？也就是说在peer_cfg中设置.audio_dir为None，.video_dir为SEND_ONLY，同时在media_provider的capture_handle中设置

        esp_capture_cfg_t cfg = {
            .sync_mode = ESP_CAPTURE_SYNC_MODE_NONE,
            .audio_src = NULL,
            .video_src = webrtc_helper.video_source_,
        };

我这么尝试之后在建立连接的最后一步发生了这样的报错：

I (83465) VID_PIPE_NEGO: Success to negotiate 0 format:yuv420p 320x240 3fps
I (83475) ESP_GMF_TASK: Waiting to run... [tsk:venc_0-0x3fcf13c0, wk:0x3c382944, run:0]
I (83485) webrtc_helper: webrtc connected
I (83485) s3 ll_cam: DMA Channel=3
I (83485) cam_hal: cam init ok
I (83505) ov3660: Mismatch PID=0x0
I (83505) ov5640: Mismatch PID=0x0
I (83505) camera: Camera PID=0x2145 VER=0x00 MIDL=0x00 MIDH=0x00
I (83505) camera: Detected GC2145 camera
I (83505) camera: Detected camera at address=0x3c
I (83915) cam_hal: PSRAM DMA mode enabled
I (83915) s3 ll_cam: node_size: 3840, nodes_per_line: 1, lines_per_node: 6
I (83915) s3 ll_cam: dma_half_buffer_min:  3840, dma_half_buffer:  3840, lines_per_half_buffer:  6, dma_buffer_size: 153600
I (83925) cam_hal: buffer_size: 153600, half_buffer_size: 3840, node_buffer_size: 3840, node_cnt: 40, total_cnt: 41
I (83935) cam_hal: Allocating 157456 Byte frame buffer in PSRAM
I (83935) cam_hal: Frame[0]: Offset: 16, Addr: 0x3C3C06B0
I (83945) cam_hal: cam config ok
I (83945) gc2145: subsample win:640x480, ratio:0.500000
I (83955) VIDEO_COMM: Video info for vid_overlay-0x3c3845d8 format:yuv420p 320x240 3fps
I (83965) ESP_GMF_TASK: One times job is complete, del[wk:0x3c382944, ctx:0x3c36db7c, label:vid_src_open]
I (83965) ESP_GMF_PORT: ACQ IN, new self payload:0x3c382944, port:0x3c36dc24, el:0x3c36db7c-vid_src
I (84025) ESP_GMF_TASK: One times job is complete, del[wk:0x3c382a24, ctx:0x3c3845d8, label��٪;�k�����pU�Ol]
Guru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.

Core  0 register dump:
PC      : 0x4038f994  PS      : 0x00060733  A0      : 0x8038f831  A1      : 0x3c3be3f0
--- 0x4038f994: block_set_prev_free at D:/Espressif/frameworks/esp-idf-v5.5/components/heap/tlsf/tlsf_block_functions.h:123
--- (inlined by) block_mark_as_free at D:/Espressif/frameworks/esp-idf-v5.5/components/heap/tlsf/tlsf_block_functions.h:177
--- (inlined by) tlsf_free at D:/Espressif/frameworks/esp-idf-v5.5/components/heap/tlsf/tlsf.c:631
A2      : 0x3c350014  A3      : 0x3c3c0684  A4      : 0x3fcb5be8  A5      : 0x3c3be3b3
A6      : 0x0000000a  A7      : 0x3c3c067c  A8      : 0xdec7b610  A9      : 0xa28baf92
A10     : 0xfffffffc  A11     : 0xa28baf90  A12     : 0x3c3c0684  A13     : 0x00060723
A14     : 0x3fcbc8c4  A15     : 0x0000cdcd  SAR     : 0x00000004  EXCCAUSE: 0x0000001c
EXCVADDR: 0xdec7b614  LBEG    : 0x400555cd  LEND    : 0x400555e1  LCOUNT  : 0xfffffffb
--- 0x400555cd: strcpy in ROM
--- 0x400555e1: strcpy in ROM


Backtrace: 0x4038f991:0x3c3be3f0 0x4038f82e:0x3c3be410 0x4037a7ab:0x3c3be430 0x40390c49:0x3c3be450 0x4215d575:0x3c3be470 0x4215c59a:0x3c3be490 0x4215c7b1:0x3c3be4b0 0x4215cc14:0x3c3be4e0
--- 0x4038f991: block_next at D:/Espressif/frameworks/esp-idf-v5.5/components/heap/tlsf/tlsf_block_functions.h:161
--- (inlined by) block_link_next at D:/Espressif/frameworks/esp-idf-v5.5/components/heap/tlsf/tlsf_block_functions.h:168
--- (inlined by) block_mark_as_free at D:/Espressif/frameworks/esp-idf-v5.5/components/heap/tlsf/tlsf_block_functions.h:176
--- (inlined by) tlsf_free at D:/Espressif/frameworks/esp-idf-v5.5/components/heap/tlsf/tlsf.c



ELF file SHA256: 158f2ceb4
:631
--- 0x4038f82e: multi_heap_free_impl at D:/Espressif/frameworks/esp-idf-v5.5/components/heap/multi_heap.c:240
--- (inlined by) multi_heap_free_impl at D:/Espressif/frameworks/esp-idf-v5.5/components/heap/multi_heap.c:229
--- 0x4037a7ab: heap_caps_free at D:/Espressif/frameworks/esp-idf-v5.5/components/heap/heap_caps_base.c:86
--- 0x40390c49: free at D:/Espressif/frameworks/esp-idf-v5.5/components/newlib/src/heap.c:34
--- 0x4215d575: esp_gmf_oal_free at E:/xiaozhi-board/xiaozhi-esp32/managed_components/espressif__gmf_core/oal/esp_gmf_oal_mem.c:69
--- 0x4215c59a: esp_gmf_job_item_free at E:/xiaozhi-board/xiaozhi-esp32/managed_components/espressif__gmf_core/src/esp_gmf_task.c:141
--- 0x4215c7b1: _esp_gmf_get_next_job at E:/xiaozhi-board/xiaozhi-esp32/managed_components/espressif__gmf_core/src/esp_gmf_task.c:158
--- 0x4215cc14: process_func at E:/xiaozhi-board/xiaozhi-esp32/managed_components/espressif__gmf_core/src/esp_gmf_task.c:250
--- (inlined by) esp_gmf_thread_fun at E:/xiaozhi-board/xiaozhi-esp32/managed_components/espressif__gmf_core/src/esp_gmf_task.c:298

添加上audio_src之后就没有出现这样的报错了，请问您是否知道原因呢

[ClearrSky]

还有一个问题，就是在idf.py menuconfig的Camera Configuration中开启了enable psram dma mode by default之后，我的摄像头输出的画面出现了很多色块，如果不开启则输出的画面是正常的，该怎么解决呢，如图所示，想向您请教一下。

[TempoTian]

@ClearrSky 上面的log看应该是内存非法访问了，请提供用的dependency.lock 文件我来用相应的版本复制下看。
下面的问题试着更新下esp32_camera的版本看，我看 espressif/esp32-camera#781 这个issue是相关的，不过我看capture拉的是最新的应该。

[ClearrSky]

dependencies.zip，感谢！这是我的dependency.lock文件。顺带问一下，现在的esp_webrtc库如果设置视频编码格式为H.264的话，是否有相应的接口可以设置编码的QP或者bitrate等参数？因为我看到esp_h264库是存在set_bitrate()这个方法的

我在esp_capture库中找到了这么一段代码，似乎可以用来设置码率：

esp_capture_err_t esp_capture_sink_set_bitrate(esp_capture_sink_handle_t h, esp_capture_stream_type_t stream_type, uint32_t bitrate)
{
    capture_path_t *path = (capture_path_t *)h;
    if (path == NULL || path->parent == NULL) {
        return ESP_CAPTURE_ERR_INVALID_ARG;
    }
    capture_t *capture = path->parent;
    capture_mutex_lock(capture->api_lock, CAPTURE_MAX_LOCK_TIME);
    esp_capture_path_set_type_t type = ESP_CAPTURE_PATH_SET_TYPE_NONE;
    int ret = ESP_CAPTURE_ERR_NOT_SUPPORTED;
    if (stream_type == ESP_CAPTURE_STREAM_TYPE_VIDEO) {
        type = ESP_CAPTURE_PATH_SET_TYPE_VIDEO_BITRATE;
        esp_capture_path_mngr_if_t *audio_path = &capture->cfg.audio_path->base;
        if (audio_path) {
            audio_path->set(audio_path, path->path_type, type, &bitrate, sizeof(uint32_t));
        }
    } else if (stream_type == ESP_CAPTURE_STREAM_TYPE_AUDIO) {
        type = ESP_CAPTURE_PATH_SET_TYPE_AUDIO_BITRATE;
        esp_capture_path_mngr_if_t *video_path = &capture->cfg.video_path->base;
        if (video_path) {
            video_path->set(video_path, path->path_type, type, &bitrate, sizeof(uint32_t));
        }
    }
    capture_mutex_unlock(capture->api_lock);
    return ret;
}

但是我对这段代码感觉到很困惑，因为它好像无论如何都会返回ESP_CAPTURE_ERR_NOT_SUPPORTED，而且当stream_type为video的时候，它设置了audio_path，当stream_type为video的时候，它设置了video_path

[josefeliuf]

@ClearrSky I created a PR to fix this problem in esp_gmf/esp_capture component. Check this issue: espressif/esp-gmf#29. Aditionally I will create a PR in this repo to be able to set the bitrate at runtime

Edit: here's the PR: #109

[ClearrSky]

That's fantastic! Thank you so much for picking this up and creating the PR, @josefeliuf . I'll check it out.