Support match lists exceeding 10,000 items

[petergs]

Terraform & Provider Version

$> terraform version
Terraform v1.12.2
on darwin_arm64
+ provider registry.terraform.io/hashicorp/external v2.3.5
+ provider registry.terraform.io/sumologic/sumologic v3.1.6

Your version of Terraform is out of date! The latest version
is 1.13.4. You can update by downloading from https://developer.hashicorp.com/terraform/install

Affected Resource(s)

sumologic_cse_match_list

Debug Output

│ Error: [ERROR] Error waiting for match list with id 234 to be updated: {"data": null, "errors": [{"message": "The sum of the offset and limit parameters must not be greater than 10000", "code": "BAD_REQUEST", "error_id": "7c2459e3-419e-415c-8ee9-d1b884f26194"}]}

Expected Behavior

I should be able to add use match lists with over 10,000 items with the Terraform provider. The documentation explicitly calls out a limit of 100,000 for matchlists.

Actual Behavior

Creating or modifying a match list containing over 10,000 items via the terraform provider results in an error.

Steps to Reproduce

• Use the below example main.tf or similar to attempt creation of a match list with over 10,000 items.

# main.tf
terraform {
  backend "local" {}

  required_version = "~> 1.12.0"

  required_providers {
    sumologic = {
      source  = "SumoLogic/sumologic"
      version = ">= 3.0.12"
    }
  }
}

provider "sumologic" {
  environment = "us2"
}

data "external" "random_strings" {
  program = ["python3", "-c", <<-EOT
    import json
    import secrets
    strings = [secrets.token_hex(8) for _ in range(10001)]
    print(json.dumps({"strings": ",".join(strings)}))
  EOT
  ]
}

locals {
  random_strings = split(",", data.external.random_strings.result.strings)
}

resource "sumologic_cse_match_list" "example_10001_items_matchlist" {
  description   = "example_10001_items_matchlist"
  name          = "example_10001_items_matchlist"
  target_column = "Username"

  dynamic "items" {
    for_each = local.random_strings
    content {
      description = items.value
      value       = items.value
    }
  }
}

The root problem is an underlying limitation of the GetMatchListItems endpoint which only supports a sum of 10,000 for the offset and limit functions. This endpoint is called here. The error can be reproduced by running the following curl command:

$> curl -X GET "https://api.us2.sumologic.com/api/sec/v1/match-list-items?limit=100&offset=10000" \
  -H "Authorization: Basic $(echo -n ''$SUMOLOGIC_ACCESS_ID':'$SUMOLOGIC_ACCESS_KEY'' | base64)" \
  -H "Content-Type: application/json"
{"data": null, "errors": [{"message": "The sum of the offset and limit parameters must not be greater than 10000", "code": "BAD_REQUEST", "error_id": "a0e90fad-7cde-4765-8cc3-b65dbe1aed7d"}]}%