What does "Source send cancelled" mean?

[tronboto]

We started seeing this error in the opentelemetry source and I don't really know how to interpret it:

{
  "timestamp": "2025-06-04T16:05:16.841021Z",
  "level": "ERROR",
  "message": "Events dropped",
  "intentional": false,
  "count": 1,
  "reason": "Source send cancelled.",
  "internal_log_rate_limit": true,
  "target": "vector_common::internal_event::component_events_dropped",
  "span": {
    "grpc_method": "Export",
    "grpc_service": "opentelemetry.proto.collector.logs.v1.LogsService",
    "name": "grpc-request"
  },
  "spans": [
    {
      "component_id": "source_opentelemetry",
      "component_kind": "source",
      "component_type": "opentelemetry",
      "name": "source"
    },
    {
      "grpc_method": "Export",
      "grpc_service": "opentelemetry.proto.collector.logs.v1.LogsService",
      "name": "grpc-request"
    }
  ]
}

[pront]

Hi @tronboto, from the source docs:

// It's possible that the caller stops polling this future while it is blocked waiting
// on `self.send()`. When that happens, we use `UnsentEventCount` to correctly emit
// `ComponentEventsDropped` events.

/// UnsentEvents tracks the number of events yet to be sent in the buffer. This is used to
/// increment the appropriate counters when a future is not polled to completion. Particularly,
/// this is known to happen in a Warp server when a client sends a new HTTP request on a TCP
/// connection that already has a pending request.
///
/// If its internal count is greater than 0 when dropped, the appropriate [ComponentEventsDropped]
/// event is emitted.

This can indicate (among other things) a backpressure issue.

[tronboto]

Thanks @pront . Do you mean backpressure coming from the sinks? Because that's not what we're seeing. Any tips on investigating this further?

vector_component_received_events_total

vector_component_discarded_events_total

vector_adaptive_concurrency_back_pressure_bucket

vector_buffer_events (buffer.max_events is 100000)

[pront]

Hi @tronboto, sorry for the late response, this thread felt through the cracks.

Did you also inspect the utilization gauge?

A ratio from 0 to 1 of the load on a component. A value of 0 would indicate a completely idle component that is simply waiting for input. A value of 1 would indicate a that is never idle. This value is updated every 5 seconds.

It is clear from the vector_component_discarded_events_total that your source is dropping events (and they are never forwarded to the next component).

Any other error logs from Vector? In your original message you shared a sample otel event but not the actual error.

[tronboto]

Hey @pront . Sorry for the delayed response - I've been away.

So there doesn't appear to be a utilization metric for sources unless you want me to check utilization for another component?

Any other error logs from Vector? In your original message you shared a sample otel event but not the actual error.

That was the error from vector, not an otel event. We have logs in json format and I prettified it to make it easier to read.

I checked the logs again and apart from messages about logs being suppressed, this is the only log we see related to the component with id source_opentelemetry.

[pront]

Hey @tronboto, I realized you never shared your config.

• Are you trying to ingest logs or traces?
• Did you previously ingest successfully?
• If you control the sender process you should be able to get errors there. I suspect the invalid payload. Note that the opentelemetry source currently support only application/x-protobuf.

[tronboto]

Are you trying to ingest logs or traces?

Logs

Did you previously ingest successfully?

We are currently ingesting successfully through this source, but apparently not everything.

If you control the sender process you should be able to get errors there. I suspect the invalid payload. Note that the opentelemetry source currently support only application/x-protobuf.

These are logs from an internal library that are being sent using otlp over grpc (protobuf) - I'll see what I can find out. One thing I have seen though is that on the sink end of things, we're seeing 400 errors sending some of these logs to elasticsearch - this shouldn't propagate back to the source in any way though should it?

[tronboto]

Hey @pront . So this an example error message we were seeing on the sender side:

2025-08-18T16:20:53.521Z        error   internal/queue_sender.go:57     Exporting failed. Dropping data.        {"resource": {}, "otelcol.component.id": "otlp/vector", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: rpc error: code = DeadlineExceeded desc = context deadline exceeded", "dropped_items": 220}

After beefing up our vector nodes, the issue has gone away. This is great, but it would have been nice if there was a clearer indicator that this was the cause. Am I right in assuming that any issue with discarded messages in sources is likely going to be resource related? Actually, this was just a coincidence - the overall traffic had dropped. However, now that it's picked up again, the discarded events are back.

[tronboto]

Just to note that we believe this is related to the issue I raised here: #23626