Uploading logs retrospectively

[kartikprakash1]

Feature description

Hi I was wondering if it's possible to support a feature like this or if there is already a way to achieve this:

We usually set our application's log verbosity to WARN level. However when there is an issue with the application we would like to have the capability to look at the logs from X days ago.

Is it possible to add a configuration to control what log levels are uploaded and if we change that configuration to something else all logs from the new level get uploaded from past X days and going forward?

Proposed solution

I would assume this can be achieved with some file based rolling storage, that keep all logs from past X days, and purge the oldest if we go beyond X days. We may want to add a storage limit as well so the logs don't eat a lot of storage from VERBOSE level.

Other relevant information

No response

[aleksandr-gringauz]

Hello @kartikprakash1 !

Thank you for your question.

Is it possible to add a configuration to control what log levels are uploaded

You can drop log events on the android app side using an EventMapper (link). In order to drop the event, you need to return null from the map method.

and if we change that configuration to something else all logs from the new level get uploaded from past X days and going forward

Currently our SDK doesn't support it. The SDK uploads the events as soon as it can.

You might also want to look at the options to exclude the logs from indexing https://docs.datadoghq.com/logs/log_configuration/indexes/#exclusion-filters

I would assume this can be achieved with some file based rolling storage, that keep all logs from past X days, and purge the oldest if we go beyond X days. We may want to add a storage limit as well so the logs don't eat a lot of storage from VERBOSE level.

Do you propose to introduce some kind of archive for logs that were dropped by the EventMapper that holds these logs for X days before deleting?

[kartikprakash1]

Hi @aleksandr-gringauz ,

Thank you for sharing the information. I have looked into using Datadog.setVerbosity() as described here which drops the logs as needed: https://docs.datadoghq.com/logs/log_collection/android/?tab=kotlin

But to your last question, yes someway to archive not enabled level logs for X days and a way to upload them later if needed.

This is a very generic use-case I can think of that can help understand why it will be a good feature to have:

• A user is using an android application out in the field, they encounter some issues with the application which is not easy to reproduce, they report back to development team by providing timeline details on when the issue happened.

• Now the development team, can enable a remotely a configuration that will allow uploading all level logs for that timeline for that particular user, and will have the capability to see back in the past on what really happened with the application.

[aleksandr-gringauz]

@kartikprakash1

Could you please elaborate on how setVerbosity solves your problem?

It only affects which internal logs of the SDK appear in logcat. Not the ones that you can log using Logger, but rather the ones that are logged using InternalLogger which is internal to the SDK and can't be used by you.

[aleksandr-gringauz]

Also please have a look at RUM retention filters https://docs.datadoghq.com/real_user_monitoring/guide/retention_filter_best_practices/#suggested-retention-filters-and-use-cases. They allow you for example to retain all RUM sessions that have a crash (or any other condition that you might want to specify in the filter).

IIUC your question is about the Logs product. But maybe you'll still find this useful.

[kartikprakash1]

@kartikprakash1

Could you please elaborate on how setVerbosity solves your problem?

It only affects which internal logs of the SDK appear in logcat. Not the ones that you can log using Logger, but rather the ones that are logged using InternalLogger which is internal to the SDK and can't be used by you.

Hi @aleksandr-gringauz You are correct, it only affects internal logging. I was mistaken, the only way for us to control what gets logged is manually not logging if certain level of logs is disabled.

[kartikprakash1]

Also please have a look at RUM retention filters https://docs.datadoghq.com/real_user_monitoring/guide/retention_filter_best_practices/#suggested-retention-filters-and-use-cases. They allow you for example to retain all RUM sessions that have a crash (or any other condition that you might want to specify in the filter).

IIUC your question is about the Logs product. But maybe you'll still find this useful.

Thanks for sharing this. I can look into this, however I still feel a feature to store logs over a period of time and upload only some log levels, can be a good idea for debugging purposes.

[aleksandr-gringauz]

however I still feel a feature to store logs over a period of time and upload only some log levels, can be a good idea for debugging purposes.

@kartikprakash1 For now we don't have this in our plans. If this feature is of high priority to you, please connect with Datadog through support channels and make a feature request.