src/literals.h is stored in .rodata section (RAM) on ESP8266

[bdraco]

Platform

ESP8266

IDE / Tooling

Arduino (IDE/CLI)

What happened?

Hi! First, I want to thank you for creating and maintaining ESPAsyncWebServer - it's an excellent library that has been invaluable for countless ESP8266 and ESP32 projects. The async architecture and feature set are fantastic.

I've been doing some memory profiling on ESP8266 projects and noticed that the string literals defined in src/literals.h (HTTP status messages, MIME types, and header names) are currently stored in RAM via the .rodata section, consuming approximately 2KB.

Would you be open to considering moving these string constants to PROGMEM on ESP8266 builds? This could free up nearly 2KB of RAM on a platform where every byte counts, while having minimal performance impact since these strings are accessed relatively infrequently.

Perhaps a hybrid approach could work well - keeping frequently-used strings (like "OK", "text/html", common headers) in RAM for performance, while moving the less common HTTP status messages and MIME types to PROGMEM. Of course, you know the library's usage patterns best, so any approach you think makes sense would be wonderful.

I understand this would require some refactoring to handle the different string access methods between platforms, but the RAM savings on ESP8266 could be quite beneficial for memory-constrained applications.

Thanks again for all your work on this library - it really is appreciated by the community!

Stack Trace

n/a

Minimal Reproductible Example (MRE)

python3 analyze_ram_strings.py .esphome/build/log8266/.pioenvs/log8266/firmware.elf --min-length 10

#!/usr/bin/env python3
"""
Analyze ESP8266 firmware ELF file to find strings stored in RAM.

This script identifies strings that are in RAM sections (.data, .bss, .rodata)
rather than in flash sections (.irom0.text, .irom.text).
"""

import subprocess
import sys
import re
from pathlib import Path
from typing import List, Tuple, Dict, Set
import argparse


def run_command(cmd: List[str]) -> str:
    """Run a command and return its output."""
    try:
        result = subprocess.run(cmd, capture_output=True, text=True, check=True)
        return result.stdout
    except subprocess.CalledProcessError as e:
        print(f"Error running command: {' '.join(cmd)}", file=sys.stderr)
        print(f"Error: {e.stderr}", file=sys.stderr)
        sys.exit(1)
    except FileNotFoundError:
        print(f"Command not found: {cmd[0]}", file=sys.stderr)
        print("Please ensure you have the xtensa toolchain installed", file=sys.stderr)
        sys.exit(1)


def get_sections(elf_file: str) -> Dict[str, Tuple[int, int]]:
    """Get section addresses and sizes from ELF file."""
    sections = {}
    
    # Try to find the right objdump command
    objdump_cmd = None
    for cmd in ['xtensa-lx106-elf-objdump', 'xtensa-esp32-elf-objdump', 'objdump']:
        try:
            subprocess.run([cmd, '--version'], capture_output=True, check=True)
            objdump_cmd = cmd
            break
        except (subprocess.CalledProcessError, FileNotFoundError):
            continue
    
    if not objdump_cmd:
        print("Error: Could not find objdump command", file=sys.stderr)
        sys.exit(1)
    
    output = run_command([objdump_cmd, '-h', elf_file])
    
    # Parse section headers
    # Format: Idx Name          Size      VMA       LMA       File off  Algn
    section_pattern = r'^\s*\d+\s+(\S+)\s+([0-9a-fA-F]+)\s+([0-9a-fA-F]+)'
    
    for line in output.split('\n'):
        match = re.match(section_pattern, line)
        if match:
            name = match.group(1)
            size = int(match.group(2), 16)
            vma = int(match.group(3), 16)
            sections[name] = (vma, size)
    
    return sections


def get_strings_from_section(elf_file: str, section: str, min_length: int = 4) -> List[Tuple[int, str]]:
    """Extract strings from a specific section."""
    strings = []
    
    # Try to find the right objdump command
    objdump_cmd = None
    for cmd in ['xtensa-lx106-elf-objdump', 'xtensa-esp32-elf-objdump', 'objdump']:
        try:
            subprocess.run([cmd, '--version'], capture_output=True, check=True)
            objdump_cmd = cmd
            break
        except (subprocess.CalledProcessError, FileNotFoundError):
            continue
    
    if not objdump_cmd:
        return strings
    
    try:
        output = run_command([objdump_cmd, '-s', '-j', section, elf_file])
    except:
        return strings
    
    # Parse hex dump output
    current_string = bytearray()
    current_addr = 0
    string_start_addr = 0
    
    for line in output.split('\n'):
        # Lines look like: " 3ffef8a0 00000000 00000000 00000000 00000000  ................"
        match = re.match(r'^\s+([0-9a-fA-F]+)\s+((?:[0-9a-fA-F]{2,8}\s*)+)', line)
        if match:
            addr = int(match.group(1), 16)
            hex_data = match.group(2).strip()
            
            # Convert hex to bytes
            hex_bytes = hex_data.split()
            for hex_chunk in hex_bytes:
                # Handle both byte-by-byte and word formats
                if len(hex_chunk) == 2:
                    byte_val = int(hex_chunk, 16)
                    if 0x20 <= byte_val <= 0x7e:  # Printable ASCII
                        if not current_string:
                            string_start_addr = addr
                        current_string.append(byte_val)
                    else:
                        if byte_val == 0 and len(current_string) >= min_length:
                            # Found null terminator
                            strings.append((string_start_addr, current_string.decode('ascii', errors='ignore')))
                        current_string = bytearray()
                else:
                    # Handle multi-byte chunks (little-endian)
                    for i in range(0, len(hex_chunk), 2):
                        byte_val = int(hex_chunk[i:i+2], 16)
                        if 0x20 <= byte_val <= 0x7e:  # Printable ASCII
                            if not current_string:
                                string_start_addr = addr + i // 2
                            current_string.append(byte_val)
                        else:
                            if byte_val == 0 and len(current_string) >= min_length:
                                strings.append((string_start_addr, current_string.decode('ascii', errors='ignore')))
                            current_string = bytearray()
                
                addr += 1
    
    return strings


def analyze_symbols(elf_file: str) -> Dict[str, Tuple[str, int, int]]:
    """Get symbol information from ELF file."""
    symbols = {}
    
    # Try to find the right nm command
    nm_cmd = None
    for cmd in ['xtensa-lx106-elf-nm', 'xtensa-esp32-elf-nm', 'nm']:
        try:
            subprocess.run([cmd, '--version'], capture_output=True, check=True)
            nm_cmd = cmd
            break
        except (subprocess.CalledProcessError, FileNotFoundError):
            continue
    
    if not nm_cmd:
        return symbols
    
    output = run_command([nm_cmd, '-S', '--size-sort', elf_file])
    
    for line in output.split('\n'):
        parts = line.split()
        if len(parts) >= 4:
            addr = int(parts[0], 16)
            size = int(parts[1], 16) if parts[1] != '?' else 0
            sym_type = parts[2]
            name = ' '.join(parts[3:])
            
            # Filter for data symbols
            if sym_type in ['D', 'd', 'R', 'r', 'B', 'b']:
                symbols[name] = (sym_type, addr, size)
    
    return symbols


def main():
    parser = argparse.ArgumentParser(description='Analyze ESP8266 firmware for RAM-stored strings')
    parser.add_argument('elf_file', help='Path to firmware.elf file')
    parser.add_argument('--min-length', type=int, default=8, help='Minimum string length to report (default: 8)')
    parser.add_argument('--show-all', action='store_true', help='Show all sections, not just RAM')
    args = parser.parse_args()
    
    if not Path(args.elf_file).exists():
        print(f"Error: File not found: {args.elf_file}", file=sys.stderr)
        sys.exit(1)
    
    print(f"Analyzing: {args.elf_file}\n")
    
    # Get sections
    sections = get_sections(args.elf_file)
    
    # Define RAM sections for ESP8266
    ram_sections = {'.data', '.rodata', '.bss'}
    flash_sections = {'.irom0.text', '.irom.text', '.text'}
    
    print("=" * 80)
    print("SECTION ANALYSIS")
    print("=" * 80)
    print(f"{'Section':<20} {'Address':<12} {'Size':<12} {'Location'}")
    print("-" * 80)
    
    total_ram_usage = 0
    total_flash_usage = 0
    
    for name, (addr, size) in sorted(sections.items()):
        if name in ram_sections:
            location = "RAM"
            total_ram_usage += size
        elif name in flash_sections:
            location = "FLASH"
            total_flash_usage += size
        else:
            location = "OTHER"
        
        if args.show_all or name in ram_sections:
            print(f"{name:<20} 0x{addr:08x}   {size:>8} B   {location}")
    
    print("-" * 80)
    print(f"Total RAM sections size: {total_ram_usage:,} bytes")
    print(f"Total Flash sections size: {total_flash_usage:,} bytes")
    
    # Analyze strings in RAM sections
    print("\n" + "=" * 80)
    print("STRINGS IN RAM SECTIONS")
    print("=" * 80)
    
    all_ram_strings = []
    
    for section in ram_sections:
        if section in sections:
            strings = get_strings_from_section(args.elf_file, section, args.min_length)
            if strings:
                print(f"\nSection: {section}")
                print("-" * 40)
                for addr, string in sorted(strings):
                    if len(string) >= args.min_length:
                        # Clean up string for display
                        clean_string = string[:100] + ('...' if len(string) > 100 else '')
                        print(f"  0x{addr:08x}: \"{clean_string}\" (len={len(string)})")
                        all_ram_strings.append((section, addr, string))
    
    # Analyze symbols
    print("\n" + "=" * 80)
    print("LARGE DATA SYMBOLS IN RAM")
    print("=" * 80)
    
    symbols = analyze_symbols(args.elf_file)
    
    # Filter and sort by size
    ram_symbols = []
    for name, (sym_type, addr, size) in symbols.items():
        # Check if symbol is in a RAM range
        for section_name in ram_sections:
            if section_name in sections:
                section_start, section_size = sections[section_name]
                if section_start <= addr < section_start + section_size:
                    ram_symbols.append((name, sym_type, addr, size, section_name))
                    break
    
    # Sort by size (largest first)
    ram_symbols.sort(key=lambda x: x[3], reverse=True)
    
    print(f"\n{'Symbol':<40} {'Type':<6} {'Size':<10} {'Section'}")
    print("-" * 80)
    
    # Show top 20 largest symbols
    for name, sym_type, addr, size, section in ram_symbols[:20]:
        if size > 0:  # Only show symbols with known size
            print(f"{name[:39]:<40} {sym_type:<6} {size:>8} B  {section}")
    
    # Summary
    print("\n" + "=" * 80)
    print("SUMMARY")
    print("=" * 80)
    print(f"Total strings found in RAM: {len(all_ram_strings)}")
    total_string_bytes = sum(len(s) + 1 for _, _, s in all_ram_strings)  # +1 for null terminator
    print(f"Total bytes used by strings: {total_string_bytes:,}")
    
    # Identify potential optimization targets
    print("\n" + "=" * 80)
    print("POTENTIAL OPTIMIZATION TARGETS")
    print("=" * 80)
    
    # Find commonly repeated strings
    string_counts = {}
    for _, _, string in all_ram_strings:
        string_counts[string] = string_counts.get(string, 0) + 1
    
    repeated_strings = [(s, c) for s, c in string_counts.items() if c > 1]
    if repeated_strings:
        print("\nRepeated strings (could be deduplicated):")
        for string, count in sorted(repeated_strings, key=lambda x: x[1] * len(x[0]), reverse=True)[:10]:
            savings = (count - 1) * (len(string) + 1)
            clean_string = string[:50] + ('...' if len(string) > 50 else '')
            print(f"  \"{clean_string}\" - appears {count} times (potential savings: {savings} bytes)")
    
    # Find long strings that could be moved to PROGMEM
    long_strings = [(s, a, st) for s, a, st in all_ram_strings if len(st) >= 20]
    if long_strings:
        print(f"\nLong strings that could be moved to PROGMEM (>= 20 chars):")
        for section, addr, string in sorted(long_strings, key=lambda x: len(x[2]), reverse=True)[:10]:
            clean_string = string[:60] + ('...' if len(string) > 60 else '')
            print(f"  {section} @ 0x{addr:08x}: \"{clean_string}\" ({len(string)} bytes)")


if __name__ == '__main__':
    main()

I confirm that:

• I have read the documentation.
• I have searched for similar discussions.
• I have searched for similar issues.
• I have looked at the examples.
• I have upgraded to the lasted version of ESPAsyncWebServer (and AsyncTCP for ESP32).

[mathieucarbou]

Hello,

The bench you saw is indeed bad for ESP8266 because all the recent changes and improvements on the lib so far are done and focused on ESP32 only. We keep the lib compatible (meaning... compilable) with ESP8266 but there is no more work neither perf improvements on ESP8266.

The constraints set by ESP8266 when dealing with constant string is so messy that it bloaters the code with macros, that are only specific to ESP8266.

That being said, if someone sends a clean PR to improve current perf in a way that is isolated from the rest, I think we will be OK to merge. I insists on the words clean and isolated, because there is a maintenance cost associated to that.

FYI, we do not support ESP8266 anymore, ESP8266 hardware will go EOL in a few years and next major version of this lib will not support ESP8266. Also, ESP8266 is single-core, which limits the benefit of an asynchronous lib.

[bdraco]

Thank you for taking the time to respond and for maintaining this excellent library! I really appreciate your openness to a clean PR despite the challenges.

I completely understand and share your sentiment about ESP8266 reaching EOL - we'd all love to see the ecosystem move to more capable chips. However, the reality is that ESP8266 will likely persist in the wild for quite some time. Many users have dozens of these devices deployed and aren't eager to replace functioning hardware, and unfortunately, vendors are still shipping new products with ESP8266 chips (presumably due to cost and existing inventory).

A clean solution for ESP8266 PROGMEM support is challenging. The platform's quirks around flash string handling would indeed require careful macro work to avoid cluttering the codebase. After looking at the code, I can see why wrapping everything in platform-specific macros would be a maintenance burden.

[bdraco]

Yes, this is indeed for ESPHome. Thank you very much for the offer to collaborate.

I'll bring this up with my ESPHome team colleagues to discuss the best approach. Since it's the weekend, it will likely be early next week before we can have that discussion.

[sivar2311]

The ESP8266 will go EOL in 2029. See https://www.espressif.com/en/products/longevity-commitment

Espressif recommends updating to ESP32-C2. See https://www.espressif.com/en/products/socs

Since ESP32-C2 has no Arduino support, I would recommend updating to C3, C5 (which has dualband 2.4 & 5 Ghz WiFi), C6 or S3

[mathieucarbou]

Yes, this is indeed for ESPHome. Thank you very much for the offer to collaborate.

I'll bring this up with my ESPHome team colleagues to discuss the best approach. Since it's the weekend, it will likely be early next week before we can have that discussion.

Of course!
I've also brought the subject to the team but in any case any help is appreciated, especially for the parts we cannot do.
You guys also have a pretty large user base so I am sure you'll be able to bring a lot of cases for improvements ;-)

[bdraco]

I did a deep dive into this and came up with a minimal, safe approach: esphome@c8264af

This change only moves HTTP response code strings to PROGMEM, which is safe because:

• They're only used in responseCodeToString() and never in string comparisons
• The Arduino String::concat() method has overloads for both const char* and const __FlashStringHelper*, handling both transparently

I investigated converting everything else in literals.h but found it would be risky and invasive. Header names and other constants are used in string comparisons and passed to functions throughout the codebase, making conversion complex and error-prone.

While this only captures 436 bytes of the ~1740 bytes potential savings, it's still meaningful - that's over 1% of the ~40KB usable RAM on ESP8266. For memory-constrained projects, every bit helps.

The implementation is clean and isolated using platform-specific macros that don't affect ESP32 builds. Would you consider this targeted optimization?

Thanks again

[willmmiles]

Hi, I'm "maintaining" another ESPAsyncWebserver fork for WLED. We also have a reasonably large userbase of existing ESP8266 installs.

To be frank, I don't think it's possible to meaningfully support the ESP8266 with the development path taken in this repository -- at least not for a project of our size.

On the one hand, on the specific subject here: I do believe it's possible to orchestrate putting the strings in PROGMEM without putting an undue burden on other platforms: through a combination of existing ESP8266 platform configuration (-D NON32XFER_HANDLER turns crashes into performance problems), and making standard library (str*, mem*, etc) calls 32-bit access safe wherever possible. If every strcpy instead called strcpy_P, I think the performance cost would be managable. At the limit, I have entertained the idea of patching the compiler to place string literals in a unique section that can be explicitly placed in PROGMEM by the linker.

The change here that is unworkable on ESP8266 (at least for us) is adopting std container structures. The ESP8266's biggest weakness is its limited RAM, so practical out-of-memory handling is a must have. Unfortunately, enabling exceptions on ESP8266 increases the ROM size by about 25%, pushing our project well outside the 1MB limit. In pretty much every other context, I'd be the first to champion using std: it promotes clean and correct code. For us, though, we just don't have the space. :(

The upshot is that - at least for the forseeable future - I expect to be maintaining a fork purely for the purposes of managing memory without C++ exceptions.