Hi, I was looking on how to enable log deduplication. What I've found is:
In the OTEL collector id field is set to a random ksuid. This prevents any deduplication from happening, because the duplicated record would get a new random id on ingestion. An alternative would be to set id to log.record.uid if it is present. github link
Even if the id was non random, schema itself doesn't support deduplication. logs_v2 table that holds the logs uses MergeTree engine, to support deduplication it needs to use ReplacingMergeTree. github link
Is adding deduplication on your roadmap? Or maybe it's already implemented in some other way?
Would you be open to accepting a change that would fix it - and how that change needs to be structured? For example both changes can be hidden behind env flags. Like SIGNOZ_USE_OTEL_ID=true to make id be set from log.record.uid and SIGNOZ_LOG_TABLE_ENGINE=ReplacingMergeTree to specify the engine during migration.
I've found one very old related issue: SigNoz/signoz#3689
Thank you!
Hi, I was looking on how to enable log deduplication. What I've found is:
In the OTEL collector
idfield is set to a random ksuid. This prevents any deduplication from happening, because the duplicated record would get a new randomidon ingestion. An alternative would be to set id tolog.record.uidif it is present. github linkEven if the id was non random, schema itself doesn't support deduplication.
logs_v2table that holds the logs uses MergeTree engine, to support deduplication it needs to use ReplacingMergeTree. github linkIs adding deduplication on your roadmap? Or maybe it's already implemented in some other way?
Would you be open to accepting a change that would fix it - and how that change needs to be structured? For example both changes can be hidden behind env flags. Like
SIGNOZ_USE_OTEL_ID=trueto make id be set fromlog.record.uidandSIGNOZ_LOG_TABLE_ENGINE=ReplacingMergeTreeto specify the engine during migration.I've found one very old related issue: SigNoz/signoz#3689
Thank you!