Cloudwatch dashboard for indexer logs and docdb status #86
Description
Is your feature request related to a problem? Please describe.
The indexer flags various types of warnings and errors based on the state of the data assets in S3 and docdb. For example, data assets are skipped (not pushed to docdb) if the s3 prefix is invalid, if the location or name in the metadata.nd.json file are invalid, etc. These logs are currently queried manually in Cloudwatch. It would be nice to have a better way to surface these errors.
cc: @dyf
Describe the solution you'd like
- A saved query in Cloudwatch (within an appropriately named folder) to parse useful info about warnings and logs.
- A saved dashboard in Cloudwatch to surface the query results as a table. Optionally add a chart for count of each log type, or other visualization.
- Optional: refactor warnings and error messages to include error type/code based on the msgs
Describe alternatives you've considered
Leaving it as is and having maintainers manually query logs.
Additional context
The query below parses the log severity and error type from the log message.
fields @timestamp, @message, @logStream, @log
| parse @message "*:root:" as severity, short_message
| filter ispresent(severity) and severity not in ["INFO", "DEBUG"]
| parse short_message "Location field * or name field * does not match actual location of record *!" as json_location, json_name, actual_location
| parse short_message "Prefix * not valid in bucket *! Skipping." as invalid_prefix, actual_bucket
| parse short_message "Error processing *: WriteError(\"Name is not valid for storage, full error: {'index': 0, 'code': 163, 'errmsg': 'Name is not valid for storage'}\")" as corrupt_location
| fields if(ispresent(json_location), "Location/name", if(ispresent(invalid_prefix), "Prefix", if(ispresent(corrupt_location), "Corrupt", "Other"))) as errorType
| display @timestamp, severity, errorType, @message, @logStream
| sort errorType, @timestamp asc
Cloudwatch can also detect patterns in logs automatically. This eliminates need for complex queries, but cannot be added to a dashboard.
