ECS cluster with Vector agent running to ingest API data and S3 object

[tayontech]

I'm currently building a data pipeline that ingest data with the usage of a vector agent running in an ecs cluster that's being managed by Terraform. Also, the docker image read its' configuration file from a s3 bucket.

So, right now the vector agent runs perfect in the ecs service that's pulling the data from the sns topic into the s3 bucket.

Here's the terraform code for it:

resource "aws_ecs_task_definition" "s3_task_def" {
  family                = "vector-s3-task"
  network_mode          = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                   = "256"
  memory                = "512"
  execution_role_arn    = aws_iam_role.logging_execution_role.arn
  task_role_arn         = aws_iam_role.logging_role.arn

  container_definitions = jsonencode([
    {
      "name":      "infosec-vector-container",
      "image":     "${aws_ecr_repository.repository.repository_url}:latest",
      "essential": true,
      "portMappings": [
        {
          "containerPort": 8686,
          "hostPort":      8686
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group":        aws_cloudwatch_log_group.log_group.name,
          "awslogs-region":       "us-west-2",
          "awslogs-stream-prefix": aws_cloudwatch_log_stream.log_stream.name
        }
      },
      "environment": [
        {
          "name":  "VECTOR_FILE",
          "value": var.vector
        }
      ]
    }
  ])
}


# creates an ECS service within the ECS cluster for s3 bucket
resource "aws_ecs_service" "service" {
  name            = "vector-s3-service"
  cluster         = aws_ecs_cluster.cluster.id
  task_definition = aws_ecs_task_definition.s3_task_def.arn
  desired_count   = var.desired_count
  launch_type     = "FARGATE"

  network_configuration {
    security_groups = [aws_security_group.sg.id]
    subnets         = [aws_subnet.private_subnet.id]
  }
}

However, when i add the ecs service that ingest data via api, i start to receive weird logs into the ecs logs and the logs that's being pulled from the api is not being sent to the s3 bucket.

Here's the api ecs service terraform code:

resource "aws_ecs_task_definition" "api_task_def" {
  family                = "vector-api-task"
  network_mode          = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                   = "512"
  memory                = "1024"
  execution_role_arn    = aws_iam_role.logging_execution_role.arn
  task_role_arn         = aws_iam_role.logging_role.arn

  container_definitions = jsonencode([
    {
      "name":      "infosec-vector-container",
      "image":     "${aws_ecr_repository.repository.repository_url}:latest",
      "essential": true,
      "portMappings": [
        {
          "containerPort": 8686,
          "hostPort":      8686
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group":        aws_cloudwatch_log_group.log_group.name,
          "awslogs-region":       "us-west-2",
          "awslogs-stream-prefix": aws_cloudwatch_log_stream.log_stream.name
        }
      },
      "environment": [
        {
          "name":  "VECTOR_FILE",
          "value": var.vector_api
        },
        {
          "name":  "SLACK_KEY",
          "value": data.aws_secretsmanager_secret_version.slack_secret.secret_string
        }
      ]
    }
  ])
}


# creates an ECS service within the ECS cluster for s3api polling
resource "aws_ecs_service" "api_service" {
  name            = "vector-api-service"
  cluster         = aws_ecs_cluster.cluster.id
  task_definition = aws_ecs_task_definition.api_task_def.arn
  desired_count   = 1
  launch_type     = "FARGATE"

  network_configuration {
    security_groups = [aws_security_group.sg.id]
    subnets         = [aws_subnet.private_subnet.id]
  }
}

Also here's my vector.toml file for the api polling config

###################################################################################################################
### SLACK AUDIT ###
###################################################################################################################

[sources.slack_audit]
type = "http_client"
endpoint = "https://api.slack.com/audit/v1/logs"
method = "GET"
scrape_interval_secs = 900
auth.strategy = "bearer"
auth.token = "${SLACK_KEY}"

    [sources.slack_audit.headers]
    Accept = ["application/json"]
   


[transforms.slack_audit_output]
type = "remap"
inputs = ["slack_audit"]
source = '''
    .source = "slack_audit"
    .vtime = now()
    .data = parse_json!(.message)
    del(.message)
'''

###################################################################################################################
### SLACK DATA SOURCE ###
###################################################################################################################
[sinks.infosec_log_prod]
type = "aws_s3"
inputs = ["*_output"]
bucket = "EXAMPLE"
key_prefix = "application={{ source }}/env=prod/year=%Y/month=%m/day=%d/"
region = "us-west-2"
compression = "gzip"
filename_extension = "json"
encoding.codec = "json"
encoding.timestamp_format = "rfc3339"

These are the logs that I'm receiving in both ecs service/task logs and I'm not understanding why.

{"appname":"shaneIxD","facility":"local2","hostname":"random.org","message":"A bug was encountered but not in Vector, which doesn't have bugs","msgid":"ID258","procid":240,"severity":"warning","timestamp":"2023-07-20T21:39:06.581Z","version":1}

{"appname":"meln1ks","facility":"syslog","hostname":"make.com","message":"We're gonna need a bigger boat","msgid":"ID452","procid":4132,"severity":"info","timestamp":"2023-07-20T21:39:07.580Z","version":1}

{"appname":"devankoshal","facility":"local5","hostname":"we.de","message":"Great Scott! We're never gonna reach 88 mph with the flux capacitor in its current state!","msgid":"ID517","procid":8273,"severity":"notice","timestamp":"2023-07-20T21:39:08.580Z","version":1}

{"appname":"ahmadajmi","facility":"user","hostname":"names.com","message":"A bug was encountered but not in Vector, which doesn't have bugs","msgid":"ID236","procid":6192,"severity":"notice","timestamp":"2023-07-20T21:39:09.580Z","version":2}

{"appname":"devankoshal","facility":"local7","hostname":"make.us","message":"There's a breach in the warp core, captain","msgid":"ID172","procid":5465,"severity":"info","timestamp":"2023-07-20T21:39:10.580Z","version":1}

{"appname":"devankoshal","facility":"news","hostname":"make.de","message":"A bug was encountered but not in Vector, which doesn't have bugs","msgid":"ID916","procid":5888,"severity":"emerg","timestamp":"2023-07-20T21:39:11.580Z","version":1}

Every time I restart the service, I get the exact same logs. When I remove the API ECS service, my S3 ECS service works fine. Has anyone encountered this before ?

I've tested the configuration file without sending the data to the s3 bucket and into my terminal instead, it worked as it's supposed to.

I've tried removing the s3 ecs service to see if that was the problem, I still receive the same error.

I've tried removing the API s3 ecs service and everything works fine.

[jszwedko]

Hi @tayontech !

I didn't dig into this deeply, but those logs look like the logs from the default configuration that the Vector image executes so it seems like maybe it isn't loading your configuration but instead using the default one. Hopefully that helps you get pointed in the right direction.

[tayontech]

Hi @jszwedko ,

I assumed it was native to the vector logging when I was receiving the exact same messages.

Another thing to note is that I am using an environmental variables in my docker file “VECTOR_FILE” so that I am able to assign different configuration files to each ECS service.

[jszwedko]

Hi @tayontech !

I think VECTOR_FILE isn't the correct environment variable. It should be VECTOR_CONFIG. See https://vector.dev/docs/reference/cli/#vector_config

[tayontech]

VECTOR_FILE is the environment variable that I'm using in the docker field to allow it to use multiple config files. It worked perfectly fine when I'm pulling data from the sqs queue @jszwedko

[jszwedko]

Ah, gotcha, I didn't realize that was an env var that you were handling. My suspicious is still that Vector is being run without any configuration file specified and so is falling back to the default one.

[tayontech]

That's what I'm thinking as well. I am currently passing Secrets from AWS secret manager to the file that's being held in the s3 bucket and the ECS cluster is reading it from the S3 bucket. So, that could be the problem, but even when I test it to use the actual secret versus passing the secret via a variable I receive the same error