feat(new sink): add Apache Doris sink support #23117

bingquanzhao · 2025-05-28T16:18:19Z

Summary

This PR introduces a new Apache Doris sink for Vector, enabling users to send log data directly to Apache Doris databases using the Stream Load API. The implementation includes:

Complete Doris sink implementation with Stream Load API integration
Comprehensive configuration options (endpoints, authentication, batching, custom headers)
Full documentation generation using CUE
Health check functionality with proper error handling
Support for Doris-specific Stream Load parameters via custom HTTP headers

Apache Doris is a modern MPP analytical database that provides sub-second query response times on large datasets, making it ideal for real-time data warehouses and log analysis scenarios.

Change Type

New feature
Bug fix
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

How did you test this PR?

Local Testing

Unit Tests: All unit tests pass with cargo test
Configuration Validation: Verified config parsing with vector validate
Documentation Generation: Successfully generated docs with make generate-component-docs
CUE Validation: All CUE files pass format and validation checks
Changelog Validation: Changelog fragment passes validation with ./scripts/check_changelog_fragments.sh

Test Configuration Used

sources:
  demo:
    type: demo_logs
    format: json
    interval: 1

sinks:
  doris:
    type: doris
    inputs: ["demo"]
    
    # Target configuration
    endpoints: 
      - "http://doris-fe1:8030"
      - "http://doris-fe2:8030"
    database: "analytics_db"
    table: "user_events"
    
    # Authentication configuration
    auth:
      strategy: basic
      user: "admin"
      password: "admin123"
    
    # Batch configuration
    batch:
      max_events: 100000        # Maximum events per batch
      timeout_secs: 30          # Batch timeout in seconds
      max_bytes: 1073741824     # Maximum bytes per batch (1GB)
    
    # Custom HTTP headers for Doris Stream Load
    headers:
      format: "json"
      strip_outer_array: "false"
      read_json_by_line: "true"
    
    # Additional configuration
    label_prefix: "vector"
    log_request: true
    log_progress_interval: 10
    buffer_bound: 1

Environment Setup

Tested configuration validation against Vector's validation system
Verified health check functionality (attempts connection to configured endpoints)
All documentation generation and validation checks pass
CUE v0.7.0 used for documentation generation

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the "no-changelog" label to this PR.

Notes

Implementation Details

Stream Load API: Uses Doris's native Stream Load API for optimal performance and compatibility
Authentication: Supports basic authentication with username/password
Batching: Configurable batching with event count, byte size, and timeout limits
Custom Headers: Support for Doris-specific Stream Load parameters via HTTP headers including:
- format: Data format specification (json, csv, etc.)
- read_json_by_line: JSON line-by-line reading mode
- strip_outer_array: Array handling configuration
- columns: Column mapping specification
Error Handling: Comprehensive error handling with configurable retry logic
Health Checks: Validates connectivity and basic authentication
Rate Limiting: Built-in rate limiting and adaptive concurrency control

Documentation

Added complete CUE documentation for the sink configuration
Generated reference documentation automatically using Vector's documentation system
Updated service definitions and URL references
All documentation validation checks pass (CI=true make check-docs)

Dependencies

No new external dependencies added
Uses existing Vector HTTP client infrastructure
Leverages standard Vector authentication, batching, and request frameworks
Follows Vector's established patterns for sink implementation

Code Quality

All code formatted with cargo fmt
Follows Vector's coding standards and patterns
Proper error handling and logging throughout
Comprehensive configuration validation

Testing Strategy

Configuration validation ensures all options are properly parsed
Health check functionality verified through connection attempts
Documentation generation confirms all metadata is correctly defined
Follows Vector's established testing patterns for sinks

References

closes: feat(new sink): add Apache Doris sink support #23117
Apache Doris Stream Load Documentation: https://doris.apache.org/docs/data-operate/import/import-way/stream-load-manual
Apache Doris Official Website: https://doris.apache.org
Vector Sink Development Guidelines: https://vector.dev/docs/reference/configuration/sinks/

bits-bot · 2025-05-28T16:18:24Z

All committers have signed the CLA.

drichards-87 · 2025-05-28T19:49:07Z

Created Jira card for Docs Team review.

maycmlee

Some small suggestions

website/cue/reference/components/sinks/base/doris.cue

website/cue/reference/components/sinks/doris.cue

maycmlee

👍 for docs

pront · 2025-06-25T17:41:23Z

Hi @bingquanzhao, thank you for this PR. Please rebase on master and fix merge conflicts. There are 12k affected lines right now.

- Update socket2 from 0.5.8 to 0.5.10 - Update sqlx from 0.8.3 to 0.8.6 - Keep mysql support in sqlx features for Doris sink

mozhu1024 · 2025-12-03T06:58:41Z

Is this PR process still ongoing?

freejool · 2025-12-05T06:26:11Z

@bingquanzhao Hi, Thanks for your work!

However, I got some warnings when consuming messages from kafka and loading into doris with your version (bingquanzhao@be278bb). Although messages are successfully written(unknown integrity).

2025-12-05T05:52:57.535661Z WARN sink{component_kind="sink" component_id=doris_sink component_type=doris}: vector_buffers::buffer_usage_data: Buffer counter underflowed. Clamping value to 0. current=2467 delta=2476

And here is my config:

data_dir: /tmp/vector/data
sources:
  kafka_source:
    type: kafka
    bootstrap_servers: 100.88.1.4:9092
    group_id: vector_consumer111111111
    topics:
      - test_topic_1
    auto_offset_reset: earliest
    # 解析JSON消息
    decoding:
      codec: bytes
sinks:
  doris_sink:
    type: doris
    inputs:
      - kafka_source
    endpoints:
      - http://100.88.1.4:8030
    database: test
    table: "{{ message_key }}"
    auth:
      strategy: basic
      user: root
      password: "root@1234"
    encoding:
      codec: "text"
      only_fields: ["message"]
    # 启用批处理以提高性能
    batch:
      max_events: 1000
      timeout_secs: 1
    headers:
      format: "json"
      strip_outer_array: "false"
      read_json_by_line: "true"
    # 配置请求
    request:
      concurrency: 1
      rate_limit_duration_secs: 1
      rate_limit_num: 100
    # 配置重试
    acknowledgements:
      enabled: false
    log_request: true

Please ask me for any other information you need!

bingquanzhao · 2025-12-05T07:12:43Z

@bingquanzhao Hi, Thanks for your work!

However, I got some warnings when consuming messages from kafka and loading into doris with your version (bingquanzhao@be278bb). Although messages are successfully written(unknown integrity).

2025-12-05T05:52:57.535661Z WARN sink{component_kind="sink" component_id=doris_sink component_type=doris}: vector_buffers::buffer_usage_data: Buffer counter underflowed. Clamping value to 0. current=2467 delta=2476

And here is my config:

data_dir: /tmp/vector/data
sources:
  kafka_source:
    type: kafka
    bootstrap_servers: 100.88.1.4:9092
    group_id: vector_consumer111111111
    topics:
      - test_topic_1
    auto_offset_reset: earliest
    # 解析JSON消息
    decoding:
      codec: bytes
sinks:
  doris_sink:
    type: doris
    inputs:
      - kafka_source
    endpoints:
      - http://100.88.1.4:8030
    database: test
    table: "{{ message_key }}"
    auth:
      strategy: basic
      user: root
      password: "root@1234"
    encoding:
      codec: "text"
      only_fields: ["message"]
    # 启用批处理以提高性能
    batch:
      max_events: 1000
      timeout_secs: 1
    headers:
      format: "json"
      strip_outer_array: "false"
      read_json_by_line: "true"
    # 配置请求
    request:
      concurrency: 1
      rate_limit_duration_secs: 1
      rate_limit_num: 100
    # 配置重试
    acknowledgements:
      enabled: false
    log_request: true

Please ask me for any other information you need!

This warning is benign and does not affect data integrity.
It is caused by interior mutability in Vector's LogEvent structure, where the internal size cache updates during processing. This results in a slight mismatch between the event size recorded when entering versus leaving the buffer, triggering the underflow.
This is a generic Vector
accounting artifact and is unrelated to the Doris sink. It is safe to ignore.

thomasqueirozb

Sorry for the delay! It was a lot of code 😁

src/internal_events/doris.rs

Cargo.toml

src/sinks/doris/retry.rs

src/sinks/doris/health.rs

src/sinks/doris/common.rs

src/sinks/doris/config.rs

bingquanzhao · 2025-12-11T02:33:13Z

Sorry for the delay! It was a lot of code 😁

Thank you so much for your review. I will adjust the code according to your review suggestions as quickly as possible.

bingquanzhao · 2025-12-22T13:18:01Z

Hi @thomasqueirozb ，thanks for the review! I've addressed your comments. Let me know if there's anything else I should update.

thomasqueirozb

Thanks! Should be good to go realistically. Would only like to talk about the string uri and I'll commit the other required changes myself

src/sinks/doris/service.rs

src/sinks/doris/common.rs

src/sinks/doris/client.rs

src/sinks/doris/service.rs

thomasqueirozb · 2025-12-22T20:38:59Z

It actually looks like integration tests are failing when I run ./scripts/run-integration-tests int doris due to a DB auth issue. I partially fixed the integration tests but didn't debug much further after I hit this issue

bingquanzhao · 2025-12-26T09:13:32Z

It actually looks like integration tests are failing when I run ./scripts/run-integration-tests int doris due to a DB auth issue. I partially fixed the integration tests but didn't debug much further after I hit this issue

Hi @thomasqueirozb ,I’ve addressed the type issue with base_url and fixed the integration test issues.

freejool · 2025-12-31T00:50:12Z

@bingquanzhao Hi, I added header group_commit: async_mode to conf, and got an error from doris "label and group_commit can't be set at the same time". Can you provide a switch to disable label generation? (more efficiency and less integrity)

bingquanzhao · 2025-12-31T02:36:21Z

@bingquanzhao Hi, I added header group_commit: async_mode to conf, and got an error from doris "label and group_commit can't be set at the same time". Can you provide a switch to disable label generation? (more efficiency and less integrity)

I will add a check. When group_commit is set, do not set the label.

bingquanzhao requested review from a team as code owners May 28, 2025 16:18

github-actions bot added domain: sinks Anything related to the Vector's sinks domain: ci Anything related to Vector's CI environment domain: external docs Anything related to Vector's external, public documentation labels May 28, 2025

drichards-87 added the editorial review label May 28, 2025

maycmlee reviewed May 30, 2025

View reviewed changes

maycmlee approved these changes Jun 16, 2025

View reviewed changes

pront added the meta: awaiting author Pull requests that are awaiting their author. label Jun 25, 2025

github-actions bot removed the meta: awaiting author Pull requests that are awaiting their author. label Jul 9, 2025

bingquanzhao added 17 commits July 9, 2025 18:12

unsolve 307 Temporary Redirect

ff17cb8

fix redirect

534f19b

u

9b3cdba

Complete the main code

975c677

fmt

cf5ff46

fix log fmt and annotation

f603887

Merge master into doris_sink and update dependencies

23491c1

- Update socket2 from 0.5.8 to 0.5.10 - Update sqlx from 0.8.3 to 0.8.6 - Keep mysql support in sqlx features for Doris sink

Resolve rebase conflicts with master branch

027ce76

fix client build request

1cfdbd8

optimize

d899ae7

add buffer bound config

ab7db3a

fix

8bf5dc6

modify integration test

bbe6326

fix

0192c15

feat(doris sink): add comprehensive documentation

ee9c302

chore: auto-format CUE files after documentation generation

c0e2c50

fix

c375db8

pront mentioned this pull request Dec 3, 2025

New sink: Apache Doris #20843

Open

thomasqueirozb added 2 commits December 5, 2025 16:50

Merge branch 'master' into doris_sink

9adfcd9

Add NamedInternalEvent

f83555a

thomasqueirozb reviewed Dec 10, 2025

View reviewed changes

Merge branch 'master' into doris_sink

4ed4b94

thomasqueirozb added the meta: awaiting author Pull requests that are awaiting their author. label Dec 10, 2025

Modify the code according to the review comments

127ab55

github-actions bot removed the meta: awaiting author Pull requests that are awaiting their author. label Dec 11, 2025

bingquanzhao added 2 commits December 12, 2025 01:01

internal_metrics doc

e3ee379

Use UriSerde instead of string

1b9b08a

thomasqueirozb added the sink: new Request or implementation of a new sink label Dec 22, 2025

thomasqueirozb added 3 commits December 22, 2025 13:59

Merge branch 'master' into doris_sink

dc2e4e7

Fix integration test in ci and locally

52cdd96

Fix doris integration tests paths

b2977a6

github-actions bot added the domain: ci Anything related to Vector's CI environment label Dec 22, 2025

thomasqueirozb approved these changes Dec 22, 2025

View reviewed changes

thomasqueirozb and others added 2 commits December 22, 2025 14:22

Apply suggestions from code review

90cdc4a

Further fix doris int tests but not completely

62d4dd4

thomasqueirozb added the meta: awaiting author Pull requests that are awaiting their author. label Dec 22, 2025

fix base url and integreation test

aeec78a

github-actions bot removed the meta: awaiting author Pull requests that are awaiting their author. label Dec 25, 2025

feat(new sink): add Apache Doris sink support #23117

Are you sure you want to change the base?

feat(new sink): add Apache Doris sink support #23117

Conversation

bingquanzhao commented May 28, 2025 • edited by pront Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Change Type

Is this a breaking change?

How did you test this PR?

Local Testing

Test Configuration Used

Environment Setup

Does this PR include user facing changes?

Notes

Implementation Details

Documentation

Dependencies

Code Quality

Testing Strategy

References

Uh oh!

bits-bot commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drichards-87 commented May 28, 2025

Uh oh!

maycmlee left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maycmlee left a comment

Choose a reason for hiding this comment

Uh oh!

pront commented Jun 25, 2025

Uh oh!

mozhu1024 commented Dec 3, 2025

Uh oh!

freejool commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bingquanzhao commented Dec 5, 2025

Uh oh!

thomasqueirozb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bingquanzhao commented Dec 11, 2025

Uh oh!

bingquanzhao commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thomasqueirozb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thomasqueirozb commented Dec 22, 2025

bingquanzhao commented May 28, 2025 •

edited by pront

Loading

bits-bot commented May 28, 2025 •

edited

Loading

freejool commented Dec 5, 2025 •

edited

Loading

bingquanzhao commented Dec 22, 2025 •

edited

Loading