Overview

investigation is a repository of sre troubleshooting artifacts created for use with Datadog as the observability platform.

Operational Flowchart

Mermaid Diagram of steps in the investigation process

 flowchart LR;
     J((("Start of Investigation")));
     A(["Problem Statement"]);
     B["Observation"];
     D{"Validate Problem Statement"};
     C["Analysis"];
     E["Troubleshooting"];
     F{"Adjust Problem Statement"};
     H(["Solution Statement"]);
     I((("End of Investigation")));
     G{"Problem Solved"};

     J-->A;
     A-->B;
     B-->C;
     C-->D;
     D-->|Accept| E;
     E-->G;
     G-->|Yes| H;
     H-->I;
     D-.->|Reject| F;
     F-.->|Yes| A;
     F-.->|No| I;
     G-.->|No| C;

Loading

Folders and Contents

Status

The root of this repo contains the following folders by status with one subfolder per investigation

Active

Ongoing investigation artifacts

Archived

Contains completed or abandoned investigation folders with artifacts retained for future reference

Investigation

Each investigation folder contains artifacts by process step

Problem Statement

This is a Markdown document in the investigation folder. It contains an initial short statement requiring investigation.

Note

Each Problem Statement arrives to SRE with inherent assumptions and bias

Observations

Objective data that can be used to perform Analysis

• screenshots
• logs
• metrics
• traces
• events

Analysis

Opinionated correlation of Observations to perform Validation of the Problem Statement

• written documentation
• advanced notebooks

Troubleshooting

Active steps to mitigate Validated Observations

• scripts
• configs

Solution Statement

This is an update to the Problem Statement document once an investigation has been completed. Solution Statements should only appear in archived investigation folders.