OpenTelemetry sink missing events

[jy-cimcloud]

I'm relatively new to both Vector and OpenTelemetry but I feel like I'm close to getting what I need. I've been fighting this for a few days now and figured it's time to ask for help.

We are trying to collect IIS logs from our web servers and take two actions: ship them to a different location using raw_message (this works) and also ship them upstream to an OpenTelemetry collector using OTLP. The communication seems to work overall and running debug on both sides shows events are sent and received, but on the OTEL side, I only see 1 event and it looks like the first from the file. If I let it run for a while, it seems to log an event periodically but I can't figure out what the pattern is or what settings to adjust.

Here is my basic configuration with the log file shipping removed to focus only on the OTLP side. The console sink actually displays all the records as expected so I think it's parsing everything correctly, just doesn't seem to work with OpenTelemetry.

sources:
  w3svc_logs:
    type: file
    include:
      - "${MY_BASE_LOG_PATH}\\W3SVC*\\*.log"
    data_dir: "${MY_BASE_LOG_PATH}\\vector_state"
    read_from: beginning
    line_delimiter: "\r\n"
    ignore_checkpoints: true
    ignore_not_found: true
    oldest_first: true
    fingerprint:
      strategy: device_and_inode

transforms:
  remap_simple_to_otlp:
    type: remap
    inputs: [w3svc_logs]
    source: |
      collector_type = "vector"
      scope_name = "myapp/website"

      resourceLogs = [{
          "resource": {
              "attributes": [
                  { "key": "collector_type", "value": { "stringValue": collector_type } },
                  { "key": "source_type", "value": { "stringValue": .source_type } },
                  { "key": "source_path", "value": { "stringValue": .file } },
                  { "key": "host.hostname", "value": { "stringValue": to_string(get_hostname() ?? "") } }
              ]
          },
          "scopeLogs": [{
              "scope": {
                  "name": scope_name
              },
              "logRecords": [{
                  "observedTimeUnixNano": to_unix_timestamp(now(), unit: "nanoseconds"),
                  "body": { "stringValue": .message },
                  "attributes": []
              }]
          }]
      }]
      . = {}
      .resourceLogs = resourceLogs

sinks:
  emit_console_basic:
    type: console
    target: stdout
    inputs:
      - remap_simple_to_otlp
    encoding:
      codec: json

  emit_logs_to_otlp:
    inputs:
      - remap_simple_to_otlp
    type: opentelemetry
    protocol:
      type: http
      uri: http://${OTLP_RECEIVER_HTTP_HOST}:${OTLP_RECEIVER_HTTP_PORT}/v1/logs
      method: post
      encoding:
        codec: json
      framing:
        method: newline_delimited
      headers:
        content-type: application/json

Here is my basic OTel collector config:

receivers:
  otlp/general-in:
    protocols:
      grpc:
        endpoint: ${OTLP_RECEIVER_GRPC_HOST}:${OTLP_RECEIVER_GRPC_PORT}
      http:
        endpoint: ${OTLP_RECEIVER_HTTP_HOST}:${OTLP_RECEIVER_HTTP_PORT}

exporters:
  debug/detailed:
    verbosity: detailed

service:
  pipelines:
    logs/cimcloud_website_w3svc_logs_otlp:
      receivers: [otlp/general-in]
      processors:
      exporters: [debug/detailed]

[jy-cimcloud]

Quick update with more details. I implemented a reduce transform to group the logs and am seeing the first set go through with 100 entries, but then it seems to skip all the others except maybe the final group. I'm not completely sure if that's accurate but looking at the date/time values in the body messages for the first group of 100, I see the entries in the group start at 04:02:45 and end at 08:47:46, the the next group starts at 13:50:46. So I'm missing the entries between 08:47:46 and 13:50:46. I've confirmed in my logs that there are values in that time range.

Is is possibly doing sampling or something by default in the opentelemetry sink? It just seems odd that I would see the output as expected on the console, but the opentelemetry sink using the same input doesn't give consistent results.

By the way, I'm using vector 0.46.1 (x86_64-pc-windows-msvc 9a19e8a 2025-04-14 18:36:30.707862743)

[pront]

Hi @jy-cimcloud, sharing this thread in case it helps #22188 (comment).

Is is possibly doing sampling or something by default in the opentelemetry sink?

No, there is no sampling in the sink. I would double check the reduce group settings.

[jy-cimcloud]

Hey @pront, thanks for the reply. I don't think the reduce group is it because I see the same behavior without reduce. It appears as if the first log (individual or grouped set) makes it to the otel collector but the rest of the lines do not. Then at some point, it sends some others. Possibly something with the line delimiter since this is Windows? I'll try to take a look at the thread you sent a little later today.

[jy-cimcloud]

Another quick update. I decided to try it as an HTTP sink and confirmed I'm seeing the same result so it doesn't seem to be OpenTelemetry specific and maybe more the underlying HTTP sink.

[vigviswa]

Any solution to this? I have a similar issue, where my app is sending a large block of logs, and I see only about 5-10 land on the OTEL/HTTP sink out of the ~400 logs.
I definitely feel it's a way the sink is sampling the block being sent. Because, when I pass the transforms as part of a file, the OTEL/HTTP sink only receives the first one

[pront]

You are probably hitting this: #22696