Splunk Input from Docker Driver

[zentourist]

I'm trying to setup a test environment with a Docker container configured with the Splunk driver that outputs to a Fluent-bit configured Splunk input and it does not seem to work. I was wondering if anyone had any advice.

Here is the input configuration:

[INPUT]
    Name           splunk
    Listen         0.0.0.0
    port           24227
    splunk_token   my-token

Here is how I'm running the container:

docker run \
    --rm \
    --log-driver=splunk \
    --log-opt splunk-token=${SPLUNK_TOKEN} \
    --log-opt splunk-url=http://${LOGGER_HOST} \
    --log-opt tag="{{.Name}}/{{.FullID}}" \
    --log-opt splunk-verify-connection=true \
    --log-opt splunk-format=json \
    --entrypoint "echo" \
    ubuntu:latest test

If I have a Splunk token set, I always get a 401 with the following logged to stdout: [input:splunk:splunk.3] missing credentials in request headers

I've tested auth with a curl and it works fine.

If I remove the token requirement from the INPUT and if the splunk-verify-connection is set, I get the following error:

• failed to initialize logging driver: splunk: failed to verify connection - 400 Forbidden - error: invalid HTTP method.

If I remove the splunk-verify-connect, I get no errors, but also no logs.

Any insight would be greatly appreciated. I want to use this same pattern to get AWS Batch logs into Fluent-bit.

🖖 Thx!

[zentourist]

I was able to sniff the traffic and it looks like the Docker driver tries to write to /services/collector/event/1.0. Is there any way to add a custom endpoint to the splunk input? Here is the HTTP capture I had.

POST /services/collector/event/1.0 HTTP/1.1
Host: localhost:24227
User-Agent: Go-http-client/1.1
Content-Length: 183
Authorization: Splunk <mytoken>
Accept-Encoding: gzip

{"event":{"line":"test","source":"stdout","tag":"docker.test.app01"},"time":"1702343351.055591","host":"docker-desktop"}
HTTP/1.1 400 Forbidden
Server: Fluent Bit v2.2.0
Content-Length: 29

error: invalid HTTP endpoint

[agup006]

For Docker I woudl recommend using the Fluentd logging driver which would allow you to use the forward input plugin.

[zentourist]

That would be ideal, however, we want to send logs from an AWS Batch job (Fargate) to a central logger and that only supports AWS or Splunk outputs.