How to solve rate-limiting issue for http sink?

[ep4sh]

Dear community,
I faced an issue with logging via vector:

│ 2024-10-16T15:51:55.535532Z ERROR sink{component_kind="sink" component_id=ingress- component_type=http}:request{request_id=29}: vector_common::internal_event::component_event │
│ s_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.

│ 2024-10-16T15:51:54.441787Z ERROR sink{component_kind="sink" component_id=ingress component_type=http}:request{request_id=28}: vector::sinks::util::retries: Internal log [No │
│ t retriable; dropping the request.] has been suppressed 8 times.

Recently, I used http sink (openobserve) with the following config:

  sinks:
    ingress:
      inputs:
        - ingress-common
      type: http
      uri: "https://<REDACTED>/api/default/ingress-common/_json"
      method: "post"
      auth:
        strategy: "basic"
        user: "<REDACTED>"
        password: "<REDACTED>"
      compression: "gzip"
      encoding:
        codec: "json"
        timestamp_format: "rfc3339"
      healthcheck:
        enabled: false

then I tried to modify requests params (for http sink):

      request:
        concurrency: 300
        rate_limit_num: 1000
        rate_limit_duration_secs: 1

But still catching the error with rate limiting. I assume I should be able to post log - but eventually it can hang/kill my log collector (openobserve).

Could you please advise what the concrete steps I can accomplish to accomplish the issue?

Best regards,P.

[jszwedko]

What is the upstream responding with? Are there error logs before those ones that have a different error message? Usually there will be an error message and then the message you are seeing about the non-retriable request being dropped.

[ep4sh]

@jszwedko
Hi Jesse, thank you for prompt reply!

I assume these logs:

2024-10-16T15:51:21.476344Z  INFO source{component_kind="source" component_id=ingress-common component_type=kubernetes_logs}:file_server: vector::internal_events::file::source: Resuming to watch file. file=/var/log/pods/kube-system_common-ingress-common-7688d557d5-p8qbg_8381229b-fa45-4ed6-a1bf-63e64348dcd6/cron/0.log file_position=7821970



2024-10-16T15:51:21.476383Z  INFO source{component_kind="source" component_id=ingress-common component_type=kubernetes_logs}:file_server: vector::internal_events::file::source: Resuming to watch file. file=/var/log/pods/kube-system_-common-ingress--common-7688d557d5-p8qbg_8381229b-fa45-4ed6-a1bf-63e64348dcd6/cron/0.log.20241015-050019 file_position=10486004
2024-10-16T15:51:23.336143Z ERROR sink{component_kind="sink" component_id=ingress-common component_type=http}:request{request_id=1}: vector::sinks::util::retries: Not retriable; dropping the request. reason="Http status: 302 Found" internal_log_rate_limit=true

2024-10-16T15:51:23.336190Z ERROR sink{component_kind="sink" component_id=ingress-common component_type=http}:request{request_id=1}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=None request_id=1 error_type="request_failed" stage="sending" internal_log_rate_limit=true

2024-10-16T15:51:23.336203Z ERROR sink{component_kind="sink" component_id=ingress-common component_type=http}:request{request_id=1}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=504 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true

2024-10-16T15:51:23.927089Z ERROR sink{component_kind="sink" component_id=ingress-common component_type=http}:request{request_id=2}: vector::sinks::util::retries: Internal log [Not retriable; dropping the request.] is being suppressed to avoid flooding.

[jszwedko]

Thanks! That's interesting. It seems like the HTTP sink is not handling 302 redirects. Would you want to open an issue? I think that would be a potentially reasonable change to make although ideally you would avoid redirects since that would turn every request Vector makes into two and slow down processing. Are you sure that your uri is correct? Do you know why it might be redirecting?

[ep4sh]

Hehe! Thanks for pointing me, that was a OAuth proxy in front of our server. By mistake, I used a wrong URL (external instead of internal).