Skip to content

how to implement a two-step data load #6625

@freesinger

Description

@freesinger

Discussed in #6624

Originally posted by freesinger December 27, 2022
Hi all, I am using fluent-bit to collect log and import an OLAP DB. Unfortunatly, this DB supports only a two-step data load, which upload to s3 first and pull from s3 bucket via HTTP.
After reading fluent-bit document, it seems no pre-built output plufin can realize my purpose, here are some methods may be word:

  • use Lua Filter and perform upload and pull in it, if it is legal to have no Output stage? Or I just put a dummy output to fulfill?
  • upload via s3 output and pull via http output, if it is legal to have 2 Ouptut stages?
    Or any other workround? Any help will be appreciated.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions