[BUG] MQTT auto discovery (HA) seems not to work

[networxnet]

Describe the bug
MQTT documentation tells:
_

MQTT - Auto Discovery¶
(AutoDiscovery related settings added 2025/06/28)

Enable Auto Discovery: When checked, the controller will send discovery data for all connected and enabled tasks to the MQTT server on first startup, and when receiving an online message in the Discovery Trigger Topic, when configured.

_

But where is the mentioned "Enable Auto Discovery" checkbox? I can't find on april release nor on the november nightly i tried yesterday.
As the documentation has been updated in june i'm a bit confused.

In addition, the config message that is sent and that Home Assistant needs for discovery and entity creation appears to be empty or incorrectly filled.
If I understand correctly, these functions should have been updated or implemented with #5199 .

What you see here is what I managed to achieve with a lot of tinkering and adding "/config" to the "Controller Subscribe" and "Controller Publish" topics. However, no config message with the necessary information is sent here on its own. Therefore, the MQTT integration in HA does not create any entities.

To Reproduce
Steps to reproduce the behavior:

1. Go to 'controllers/MQTT'
2. check for Auto Discovery checkbox or feature
3. check mqtt messages send, i. e. in MQTT Explorer integration

Expected behavior
Aktivierung der MQTT Autodiscovery Funktion möglich, damit die nötigen Messages gesendet werden.

Used platform (please complete the following information):

• ESP type: ESP8266
• Build version: [ESP_Easy_mega_20251105_custom_312_ESP8266_4M2M_LittleFS Nov 5 2025]

Platform Specifics (when applicable, please complete the following information):

• Flash size: 4M
• Brand/Model: NodeMCU
• Browser + OS Win10 + Brave

[tonhuisman]

By default, and on purpose for not properly fitting in the .bin files for ESP8266 (too much time has been spent already to try and fit the code for ESP8266), new features are only available in ESP32 builds.
You can create a self-defined Custom ESP8266 build yourself (the default Custom ESP8266 builds don't include this either), following the guidance available here.

And of course you can switch to an ESP32 (many models available, ESP32 Classic, ESP32-C3, ESP32-C6, ESP32-S3), as those ESPEasy builds do include the HA Auto discovery feature.

[networxnet]

Ah, okay, thanks for the helpful tip. I must have overlooked that information. It might be a good idea to share it again in a more prominent way.

I still have a few ESP8266s lying around from a project for fine dust sensors that I ran in my city a few years ago. They need to be used up first. :D The ESP32-S2 that I still have here doesn't want to work; I can't get any firmware onto it, only the bootloader comes up.

[tonhuisman]

The ESP32-S2 that I still have here doesn't want to work; I can't get any firmware onto it, only the bootloader comes up.

That's seen quite often with the ESP32-S2, it's a bit of a PITA 😱

To get it into flash-mode, connect the GPIO-0 pin to GND, then reset or power-up the device. Next flash it using the Espressif Flash Download tool.
For ESPEasy you need to flash with a .factory.bin file at address 0. That bin file is available in the all.zip file you can download from https://td-er.nl/ESPEasy/latest/
You can of course try to flash it from that page, but I'm not sure if the flashing will work when the device is already in flash-mode, as the web-flasher does some smart probing to check what ESP is connected.

[networxnet]

By default, and on purpose for not properly fitting in the .bin files for ESP8266 (too much time has been spent already to try and fit the code for ESP8266), new features are only available in ESP32 builds. You can create a self-defined Custom ESP8266 build yourself (the default Custom ESP8266 builds don't include this either), following the guidance available here.

And of course you can switch to an ESP32 (many models available, ESP32 Classic, ESP32-C3, ESP32-C6, ESP32-S3), as those ESPEasy builds do include the HA Auto discovery feature.

Ich bin nach der Anleitung vergegangen und habe VScode, Platformio etc. installiert. Kannst du sagen was ich in der pre_custom_esp82xx.py hinzufügen muss um MQTT Auto Discovery zu ergänzen und ein Build mit LittleFS Dateisystem zu erzeugen. Oder kann ich auch das vorhandene 8266LittleFS Enviroment editieren?

[TD-er]

Better not using the ESP8266 LittleFS build as LittleFS on ESP8266 (at least the way how we use it in ESPEasy) is terribly slow when saving settings.

You need at least FEATURE_MQTT_DISCOVER (define it to be 1 )
Maybe you also need FEATURE_MQTT_DEVICECLASS , FEATURE_MQTT_STATE_CLASS and FEATURE_CUSTOM_TASKVAR_VTYPE (each set to 1 )

[networxnet]

Thanks for the tips.

I wanted to use LittleFS because I wanted to write the read values persistently to the flash and it supports true real leveling.

Does speed play such a big role when a value is written every 2 seconds at most?

[tonhuisman]

When writing to Flash every 2 seconds, the flash chips will wear out in a few months, if not sooner, so that should better be avoided.

There is an unfinished PR open that adds support for FRAM (and EEPROM), that can be written much more often, but that'll take some time to complete.

[networxnet]

I have read that this should not be a problem with reasonable wear leveling, such as that provided by littleFS. According to calculations, this should work for at least 10 years, as all cells are written to evenly.

How do other ESP solutions do this? Tasmota, for example, also writes consumption data persistently so that it survives a restart. I have been using some Tasmota plugs for over 5 years now.

[tonhuisman]

Most likely, the data surviving a restart is stored in so called RTC. That's a mechanism also used in ESPEasy, to restore task values after a warm boot or wake-up from sleep mode.

[TD-er]

SPIFFS also has some wear levelling algorithm... or at least it uses some neat tricks which take into account how this type of flash memory is being used.

NAND flash can only change a 1 into a 0. If you need to change a 0 into a 1, then it needs to read the entire block, erase the block and write back the changed data.
This block erase is what wears out the flash as it is a rather 'violent' action and it does change the levels of all bits in that block. (or resets them to 1, which of those 2 options is done differs per brand/type)

So if you only need to change a 1 to a 0, then this should not be counted as a write.
Only the erase calls should count as a write.
This is exactly what SPIFFS uses to keep track of used sectors, free blocks, etc.
So in the 'administration block' these bits are changed to 0 when being marked as 'ready to be freed', or 'used' (and some other administration)

Problem with SPIFFS is that these administration blocks are always the same. So if you are writing often to an almost full file system, you will wear out these blocks quite fast, causing all kinds of issues.
Since SPIFFS does all this administration in memory, you will only notice file system corruption after a reboot and likely even later when writing again as you may overwrite data which was still in use.

LittleFS does do things a bit more intelligently, so it can recover from these kind of corruptions and also when a crash or reboot happened while writing.

However, when using LittleFS, changing a block somewhere in the middle (or worse, at the beginning) of a file, requires the file to be completely rewritten from the start of the change. This makes it so slow as ESPEasy settings are stored in a single large file and changes are made mostly in the first half of the file.

When using HA + MQTT there are other ways to 'store' values.
You can publish a message to some topic using the retained flag and then use for example MQTT import to subscribe to those topics.
As soon as you subscribe to a topic, the last published retained message will be sent to that client that is subscribing to that topic.
N.B. Publishing an empty message (with retained flag) to that topic will remove the retained message.

This way the data will be kept by the MQTT broker and it depends on the settings of the broker if retained messages will be stored on the storage of that broker and how frequent. (e.g. once per half hour, immediately, etc)

As Ton mentioned, ESPEasy also stores task value data in RTC memory of the ESP.
This memory will be kept as long as the device is powered. (so will survive a reboot, though may not always survive an update of the firmware, on ESP32-xx due to how RTC memory is being assigned)
This way you can store a value in a dummy task (upto 4 values per dummy task), which will survive a reboot.
Using rules you can act on received retained messages from the broker when you subscribe to it.
N.B. When ESPEasy lost WiFi connection and reconnects, the MQTT client also reconnects, which will cause the retained messages to be received again.

[networxnet]

Most likely, the data surviving a restart is stored in so called RTC. That's a mechanism also used in ESPEasy, to restore task values after a warm boot or wake-up from sleep mode.

No, I'm not talking about a warm start, but about completely disconnecting the device from the power supply. I use a number of switchable plugs with Tasmota, some of which I use on different devices or only on a session-by-session basis, e.g., to control Christmas lights. I can put these away in a cupboard for six months, then reconnect them and the old consumption values are still there. That should rule out the use of the RTC, right?

[TD-er]

Yep, that rules out RTC memory use.
Still might be that they do use some hybrid form, where the last values are saved every X minutes.
I know Theo (the man behind Tasmota) does often use some clever tricks and I know they have been using some round-Robin writing to flash sectors.
So it might be that they are writing these kind of values taking into account the low level writing principle of flash.
For example keeping some kind of counter at the start (or end?) of a block and mark off where no longer valid values are stored.
Then you can write N values until your block is full, which only counts like 1 write.
For a 4k block, you can write roughly 1000x a 32-bit value like this.

We don't have this in ESPEasy. (yet)

[networxnet]

Better not using the ESP8266 LittleFS build as LittleFS on ESP8266 (at least the way how we use it in ESPEasy) is terribly slow when saving settings.

You need at least FEATURE_MQTT_DISCOVER (define it to be 1 ) Maybe you also need FEATURE_MQTT_DEVICECLASS , FEATURE_MQTT_STATE_CLASS and FEATURE_CUSTOM_TASKVAR_VTYPE (each set to 1 )

Die Einträge sind ind er per_custom_esp82xx.py nicht enthalten. Reicht es sie einfach unten im "D-FEATURE_ .. Bloch hinzuzufügen? Also

"-DFEATURE_MQTT_DISCOVER=1",
"-DFEATURE_MQTT_DEVICECLASS=1",
"-DFEATURE_MQTT_STATE_CLASS=1",
"-DFEATURE_CUSTOM_TASKVAR_VTYPE=1"

`Import("env")
import os

# access to global construction environment
#print env

# Dump construction environment (for debug purpose)
#print(env.Dump())

# append extra flags to global build environment
# which later will be used to build:
# - project source code
# - frameworks
# - dependent libraries

custom_defines = []
#env.Append(CPPDEFINES=[
  # ,"NO_HTTP_UPDATER"
  # ,("WEBSERVER_RULES_DEBUG", "0")
#])
if os.path.isfile('src/Custom.h'):
  custom_defines=["-DUSE_CUSTOM_H"]
else:
  custom_defines=[
    "-DCONTROLLER_SET_ALL",
    "-DNOTIFIER_SET_NONE",
    "-DPLUGIN_SET_NONE",
    "-DUSES_P001",  # Switch
    "-DUSES_P002",  # ADC
    "-DUSES_P003",  # Generic Pulse Counter
    "-DUSES_P004",  # Dallas DS18b20
    "-DUSES_P026",  # System info
    "-DUSES_P027",  # INA219
    "-DUSES_P028",  # BME280
    "-DUSES_P033",  # Dummy
    "-DUSES_P036",  # FrameOLED
    "-DUSES_P037",  # MQTT Import
    "-DUSES_P045",  # MPU6050
    "-DUSES_P049",  # MHZ19
    "-DUSES_P052",  # SenseAir
    "-DUSES_P056",  # SDS011-Dust
#    "-DUSES_P059",  # Encoder
#    "-DUSES_P080",  # Dallas iButton
    "-DUSES_P081",  # Cron
    "-DUSES_P082",  # GPS
#   "-DUSES_P085",  # AcuDC24x
#    "-DUSES_P098",  # PWM motor
#   "-DUSES_P100",  # Pulse Counter - DS2423
#   "-DUSES_P087",  # Serial Proxy
#   "-DUSES_P094",  # CUL Reader
#   "-DUSES_P095",  # TFT ILI9341
#    "-DUSES_P106",  # BME680
#    "-DUSES_P107",  # SI1145 UV index

    "-DUSES_P146",  # Cache Reader
    "-DUSES_P169",  # AS3935 Lightning Detector 

    "-DUSES_C016",  # Cache Controller
#    "-DUSES_C018",  # TTN/RN2483
#   "-DUSES_C015",  # Blynk

    "-DFEATURE_MQTT_TLS=0",
    "-DFEATURE_EMAIL_TLS=0",
#    "-DFEATURE_MDNS=1",
#    "-DFEATURE_SD=1",
    "-DFEATURE_EXT_RTC=1",
    "-DFEATURE_I2CMULTIPLEXER=1",
    "-DFEATURE_TRIGONOMETRIC_FUNCTIONS_RULES=1",
    "-DFEATURE_CUSTOM_PROVISIONING=1",
    "-DFEATURE_DEFINE_SERIAL_CONSOLE_PORT=0",

    "-DFEATURE_ESPEASY_P2P=1",
    "-DEMBED_ESPEASY_DEFAULT_MIN_CSS",

#    "-DPLUGIN_STATS_NR_ELEMENTS=30",
    "-DFEATURE_PLUGIN_STATS=1",
    "-DFEATURE_CHART_JS=1",

#    "-DFEATURE_ADC_VCC=1",

    "-DFEATURE_SETTINGS_ARCHIVE=1",
    "-DDISABLE_SC16IS752_SPI"
  ]

my_flags = env.ParseFlags(env['BUILD_FLAGS'])
my_defines = my_flags.get("CPPDEFINES")
env.Append(CXXFLAGS=custom_defines)
#defines = {k: v for (k, v) in my_defines}

print("\u001b[32m Custom PIO configuration check \u001b[0m")
# print the defines
print("\u001b[33m CPPDEFINES: \u001b[0m  {}".format(my_defines))
print("\u001b[33m Custom CPPDEFINES: \u001b[0m  {}".format(custom_defines))
print("\u001b[32m ------------------------------- \u001b[0m")


if (len(my_defines) == 0):
  print("\u001b[31m No defines are set, probably configuration error. \u001b[0m")
  raise ValueError


`

[TD-er]

Yep, something like this:

...
    "-DFEATURE_SETTINGS_ARCHIVE=1",
    "-DFEATURE_MQTT_DISCOVER=1",
    "-DFEATURE_MQTT_DEVICECLASS=1",
    "-DFEATURE_MQTT_STATE_CLASS=1",
    "-DFEATURE_CUSTOM_TASKVAR_VTYPE=1",
    "-DDISABLE_SC16IS752_SPI"
  ]

You probably have to remove some other parts of this to make it fit as the build size of an ESP8266 build should not exceed 1 MByte (minus 4k)

Or you copy the Custom-sample.h to Custom.h (mind the capital C and the rest lower-case) and define these like this:

#define FEATURE_MQTT_DISCOVER 1
#define FEATURE_MQTT_DEVICECLASS 1
#define FEATURE_MQTT_STATE_CLASS 1
#define FEATURE_CUSTOM_TASKVAR_VTYPE 1

N.B. I don't know if these features all have been thoroughly tested on ESP8266. It should work....

[networxnet]

The ESP32-S2 that I still have here doesn't want to work; I can't get any firmware onto it, only the bootloader comes up.

That's seen quite often with the ESP32-S2, it's a bit of a PITA 😱

To get it into flash-mode, connect the GPIO-0 pin to GND, then reset or power-up the device. Next flash it using the Espressif Flash Download tool. For ESPEasy you need to flash with a .factory.bin file at address 0. That bin file is available in the all.zip file you can download from https://td-er.nl/ESPEasy/latest/ You can of course try to flash it from that page, but I'm not sure if the flashing will work when the device is already in flash-mode, as the web-flasher does some smart probing to check what ESP is connected.

Thank you very much! With the factory image, I was able to revive the LILYGO TTGO T8 ESP32-S2. However, it now appears that the ST77789 display is not supported. :(