[datadog_dashboard] Add support for sorting toplist widget.

[ric-sigma-computing]

Fixes #2651

Add support for sorting toplist widgets in dashboards

[ric-sigma-computing]

@estherk15 Thank you for reviewing - I see I also need a code review from a few other people before merging, is there anything else I need to do from my end? If it's in the review queue that's fine I understand people are busy with the holidays.

[estherk15]

@estherk15 Thank you for reviewing - I see I also need a code review from a few other people before merging, is there anything else I need to do from my end? If it's in the review queue that's fine I understand people are busy with the holidays.

Thanks for the follow up @ric-sigma-computing! Yes, the PR should be on each of the respective teams' queue (I reviewed for documentation). If you don't hear back after the holidays, I'm happy to help nudge the teams!

[ric-sigma-computing]

@estherk15 Happy new year! Would you be able to help bump this review with the team? Thanks in advance

[estherk15]

@ric-sigma-computing I reached out to the product team and they're reviewing this now!

[annagchang]

ℹ️ sort schema is optional, should be only additional non-breaking change

[bonnierhee]

Reviewed this with @annagchang and while it works we have a few questions.

We created a dashboard with a top list widget using this PR (dashboard)

We specified we wanted the sort to have a count of 5. The JSON looks correct - it has the sort { count: 5 } which is what we specified.

{
    "title": "",
    "type": "toplist",
    "requests": [
        {
            "queries": [
                {
                    "compute": {
                        "aggregation": "count"
                    },
                    "data_source": "logs",
                    "group_by": [
                        {
                            "facet": "service",
                            "limit": 10,
                            "sort": {
                                "aggregation": "count",
                                "metric": "count",
                                "order": "desc"
                            }
                        }
                    ],
                    "indexes": [
                        "*"
                    ],
                    "name": "query1",
                    "search": {
                        "query": "env:prod"
                    }
                }
            ],
            "response_format": "scalar",
            "formulas": [
                {
                    "formula": "query1"
                }
            ],
            "sort": {
                "count": 5
            }
        }
    ],
    "style": {}
}

The widget visualization also looks correct - it only shows the top 5

However, the query editor is off:

The "display only" says "Top 10" and "count," but in the image, we are specifying the top 5, and it is using the group by service field to sort, not the count.

Is this expected? Also, do we intentionally allow for a different sort/count feature on the widget when the query editor already has one built in on the query?

[ric-sigma-computing]

Reviewed this with @annagchang and while it works we have a few questions.

We created a dashboard with a top list widget using this PR (dashboard)

We specified we wanted the sort to have a count of 5. The JSON looks correct - it has the sort { count: 5 } which is what we specified.

{
    "title": "",
    "type": "toplist",
    "requests": [
        {
            "queries": [
                {
                    "compute": {
                        "aggregation": "count"
                    },
                    "data_source": "logs",
                    "group_by": [
                        {
                            "facet": "service",
                            "limit": 10,
                            "sort": {
                                "aggregation": "count",
                                "metric": "count",
                                "order": "desc"
                            }
                        }
                    ],
                    "indexes": [
                        "*"
                    ],
                    "name": "query1",
                    "search": {
                        "query": "env:prod"
                    }
                }
            ],
            "response_format": "scalar",
            "formulas": [
                {
                    "formula": "query1"
                }
            ],
            "sort": {
                "count": 5
            }
        }
    ],
    "style": {}
}

The widget visualization also looks correct - it only shows the top 5

However, the query editor is off: The "display only" says "Top 10" and "count," but in the image, we are specifying the top 5, and it is using the `group by service` field to sort, not the count.

Is this expected? Also, do we intentionally allow for a different sort/count feature on the widget when the query editor already has one built in on the query?

@bonnierhee @annagchang Thank you for the detailed review and comments. I am able to reproduce the issue described using the following terraform code

resource "datadog_dashboard" "dashboard1" {
  title       = "testing"
  description = "testing"
  layout_type = "ordered"
  reflow_type = "auto"

  widget {
    toplist_definition {
      title       = ""
      title_size  = "16"
      title_align = "left"

      request {
        query {
          event_query {
            name        = "query1"
            data_source = "logs"
            indexes     = ["*"]
            search {
              query = "*"
            }
            compute {
              aggregation = "count"
            }
            group_by {
              facet = "service"
              limit = 10
              sort {
                aggregation = "count"
                metric      = "count"
                order       = "desc"
              }
            }
            storage = "hot"
          }
        }

        formula {
          formula_expression = "query1"
        }

        sort {
          count = 5
        }
      }

      style {
        display {
          type = "stacked"
        }
      }
    }
  }
}

If I change the sort block to include the order_by block, then the query editor shows the correct count and sort by values

        sort {
          count = 5
          order_by {
            type  = "formula"
            index = 0
            order = "desc"
          }
        }

When the order_by block is omitted, then the query editor shows the count=5 in the "json" tab, but not in the "edit" tab. However, the dashboard does respect the count and indeed only shows the top 5.

I believe this may be due to a divergence in the data processing layer within datadog, the dashboard rendering pipeline respects the count attribute submitted by terraform, but the query editor tool does not unless both the count and order_by attributes are present.

Here are some options I propose:

• I can add a default order_by within terraform when none is specified by the user (anti-pattern in terraform, adding unexpected state)
• I can make terraform throw an error if no order_by is specified by the user (not datadog api spec compliant)
• Leave as-is (my preference)

The reason for the last option is that this seems to be a larger issue than just the terraform code. Even in the datadog widget query editor, if the order_by attribute is removed the same behavior is seen. So the UI would need a fix to default or require the attribute in any case regardless of what is done here.

As for the last question, that may best be answered by datadog product team, but I would guess the query processing layer is separate from the visualization layer and it's a post-processing filter applied on top of the query results hence they are unaware of what sorting/filtering has already taken place.