Filtering ECS services/tasks based tag is not working as expected

[teajsshekokar]

Command

docker run --rm \
    -e DiscoveryOptions__EcsClusters="my_cluster" \
    -e AWS_REGION="us-west-1" \
    -e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \
    -e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \
    -e AWS_SESSION_TOKEN="$AWS_SESSION_TOKEN" \
   -e  DiscoveryOptions__EcsServiceSelectorTags="PROMETHEUS_TARGET=true" \
    # or  -e DiscoveryOptions__EcsTaskTags="PROMETHEUS_TARGET=true;"
    -e AWS_REGION="$AWS_REGION" \
    -p 9001:9001 \
    apptality/aws-ecs-cloudmap-prometheus-discovery:latest

Current Behavior:
I am attempting to filter services or task definitions based on the tag PROMETHEUS_TARGET=true, but it is not functioning as expected. Instead of fetching only the services with this specific tag, it retrieves all services, including those with the METRICS_PORT tag.

Expected behavior:
The filtering should work based on the DiscoveryOptions__EcsServiceSelectorTags option and only include the targets that match PROMETHEUS_TARGET=true.

And thanks a lot for this project it is really helpful. 🎉

[alex-tsbk]

@teajsshekokar thank you for raising the question.

First off, I would like to clarify a bit on the line above -e DiscoveryOptions__EcsTaskTags="PROMETHEUS_TARGET=true;". The usage is not quite correct, as this parameter is designed to filter tags that should be included into metadata. Consider your ECS task has 2 tags:

• "TagKey_One" with value "Value_One"
• "TagKey_Two" with value "Value_Two"

Then, if you to specify -e DiscoveryOptions__EcsTaskTags="TagKey_One;" response for the target would look like:

[
  {
    "targets": [
      "10.200.10.200:8080"
    ],
    "labels": {
      "__metrics_path__": "/metrics",
      "instance": "10.200.10.200",
      "scrape_target_name": "app",
      "__meta_ecs_cluster": "my-ecs-cluster",
      "__meta_ecs_service": "my-fargate-application",
      "__meta_ecs_task": "arn:aws:ecs:us-west-2:123456789012:task/my-ecs-cluster/c88nc14799fa46d794c1899612061h3s",
      "__meta_ecs_task_definition": "arn:aws:ecs:us-west-2:123456789012:task-definition/my-fargate-application:2",
     ...
      "TagKey_One": "Value_One"
    }
  },
  ...
]

Now, back to the original issue you described. Considering your ECS service has tag with key "PROMETHEUS_TARGET" set to value "true", then -e DiscoveryOptions__EcsServiceSelectorTags="PROMETHEUS_TARGET=true;" should include this service. Now, ECS service is expect to always have 1:1 match to a task definition (thus, service running 2 different task definitions version is not something that is supported, neither I think this is a typical use-case anyways), so there is no way to filter task definitions themselves. You need to either tag your ECS services, or ensure that ECS service enforces tag propagation from task definition.

The idea of the plugin is to find all services across all clusters specified, and return each running task instance as the prometheus scrape target. Imagine having a single ECS service with 10 running tasks. If such ECS service is configured to be discovered (via -e DiscoveryOptions__EcsServiceSelectorTags) then expected output of 9001:/prometheus-targets is 10 entries in the array. Now, if your ECS task is composed of multiple containers, and you only would like to scrape metrics from some of them - then you would have to control such discovery via supplying appropriate triplets of "METRICS_" tags on each ECS service (or ECS task definition, ensuring tags are propagated)

I would like you to provide a bit more information, as there might be few things involved. Would you mind sharing JSON definition of your ECS services and their corresponding ECS tasks? If you could provide at least 2 ECS services definitions that would really help to get to the bottom of it.

In the meantime, I suggest checking few things:

1. IAM role assigned to the discovery service is missing permissions and can't access tagging APIs, thus tags are not fetched properly. I would suggest enabling debug mode and confirm responses from AWS apis do contain tags in responses. There is another open issue where I explain to how enable debug.
2. Tags keys and values are case sensitive. So "True" and "true" are different values. Double-check case, for both tag key and tag value.
3. Do your services use "Service Connect"? If yes - then you might need to harden those by defining value for "CloudMapServiceSelectorTags". Please refer to the documentation