smoke tests and cloudwatch alarms

[abarciauskas-bgse]

Since titiler-cmr is now being used "in production" by Worldview, FIRMs and soon to be other VEDA instances, I would like to add some automated smoke tests to test a deployed instance with supported datasets and also some cdk for a cloudwatch alarm to email us given a certain number of 500 errors.

@hrodmn let me know if you have other thoughts and suggestions on this topic

Update: Based on below feedback I think we should:

• add logging that includes concept id
• add a dashboard that shows request by concept id
• add a metric for 500 errors
• connect that metric to slack
• I think smoke/integration tests could also be worthwhile

[hrodmn]

It would be cool to add a dashboard in CloudWatch that shows request counts by concept_id. Maybe we could connect the alarm to a slack channel?

[abarciauskas-bgse]

As Henry noted in slack, we can use https://github.com/developmentseed/titiler/blob/main/src/titiler/core/titiler/core/middleware.py#L97. We already include that middleware so can either just remove this conditional OR add the environment variable TITILER_CMR_DEBUG

[abarciauskas-bgse]

trying out removing the conditional in #66

[abarciauskas-bgse]

@hrodmn I added a todo list above. I have not connected cloudwatch to slack before, but came across https://community.aws/content/2cFE52QJ3vlPyDGzxeCJcj9jUXN/integrate-slack-channel-and-cloudwatch-alarms-using-aws-chatbot - is that what you would use?

[hrodmn]

I have not connected CloudWatch to Slack but that guide looks like a great starting point. For the dashboards it might be useful to work off this example from maap-eoapi. I used the CloudWatch query UI to figure out how to properly query the logging attributes so it might take some fiddling to update it for titiler-cmr.

[abarciauskas-bgse]

Very helpful thank you!