out_arvancloud_cloudlogs: add new output plugin for ArvanCloud CloudLogs #11095
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
javadnasrolahi
requested review from
celalettin1286,
edsiper,
fujimotos,
koleini,
niedbalski and
patrick-stephens
as code owners
|
Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. WalkthroughAdds a new ArvanCloud CloudLogs output plugin: build toggles and options, plugin registration and build files, public header, configuration lifecycle, and full plugin implementation including init, flush (HTTP post, optional gzip), timestamp parsing/formatting, and cleanup. Changes
Sequence Diagram(s)sequenceDiagram
participant FB as Fluent Bit Core
participant Init as cb_arvancloud_init
participant Cfg as flb_arvancloud_conf_create
participant Ctx as Plugin Context
participant Flush as cb_arvancloud_flush
participant Format as arvancloud_format
participant HTTP as Fluent Bit HTTP Client
participant AC as ArvanCloud API
FB->>Init: plugin init
Init->>Cfg: create & validate config (apikey, proxy, keys)
Cfg->>Ctx: allocate context, setup upstream (https, host, port, uri)
Ctx-->>FB: return context
loop For each event chunk
FB->>Flush: flush events
Flush->>Format: decode events, build JSON logs array
Format->>Format: resolve timestamps & logType per record
Format-->>Flush: formatted payload
alt gzip enabled
Flush->>Flush: gzip compress payload
end
Flush->>HTTP: POST payload (headers + optional Authorization)
HTTP->>AC: send request
AC-->>HTTP: respond (status)
HTTP-->>Flush: deliver status
alt 2xx
Flush-->>FB: FLB_OK
else retryable (5xx / network)
Flush-->>FB: FLB_RETRY
else client error (4xx)
Flush-->>FB: FLB_ERROR
end
end
FB->>Init: shutdown
Init->>Ctx: flb_arvancloud_conf_destroy -> free resources
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (7)
🚧 Files skipped from review as they are similar to previous changes (6)
🧰 Additional context used🧬 Code graph analysis (1)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c (9)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Comment on lines
+418
to
+421
| c = flb_http_client(u_conn, FLB_HTTP_POST, ctx->uri, | ||
| final_payload, final_payload_size, | ||
| NULL, 0, // host=NULL, port=0 -> use from u_conn | ||
| NULL, 0); // proxy=NULL |
There was a problem hiding this comment.
The flush path always creates the HTTP client with host=NULL and proxy=NULL, even when proxy was parsed during configuration. Because the upstream was created with TLS against the proxy host, the code now attempts to start a TLS handshake with the proxy and never performs an HTTP CONNECT or sets the real Host header. Standard HTTP proxies require a plain CONNECT request before TLS, so any configuration that specifies a proxy will fail to deliver logs. Pass the proxy information (and target host) to flb_http_client or drop the proxy option.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
CMakeLists.txt(1 hunks)cmake/plugins_options.cmake(1 hunks)plugins/CMakeLists.txt(1 hunks)plugins/out_arvancloud_cloudlogs/CMakeLists.txt(1 hunks)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c(1 hunks)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.h(1 hunks)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c(1 hunks)
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
cmake/plugins_options.cmakeCMakeLists.txt
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.
Applied to files:
cmake/plugins_options.cmakeCMakeLists.txt
🧬 Code graph analysis (3)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.h (1)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (2)
flb_arvancloud_conf_create(27-156)flb_arvancloud_conf_destroy(158-189)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (7)
include/fluent-bit/flb_mem.h (2)
flb_calloc(84-96)flb_free(126-128)include/fluent-bit/flb_output.h (1)
flb_output_config_map_set(1298-1321)src/flb_output.c (2)
flb_output_get_property(1096-1099)flb_output_upstream_set(1556-1647)src/flb_sds.c (2)
flb_sds_create(78-90)flb_sds_destroy(389-399)src/flb_utils.c (1)
flb_utils_url_split(1441-1534)src/flb_record_accessor.c (2)
flb_ra_create(271-358)flb_ra_destroy(232-248)src/flb_upstream.c (2)
flb_upstream_create(290-363)flb_upstream_destroy(656-698)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c (11)
include/fluent-bit/flb_compat.h (1)
gmtime_r(75-81)src/flb_strptime.c (1)
flb_strptime(247-251)src/flb_log_event_decoder.c (3)
flb_log_event_decoder_init(103-120)flb_log_event_decoder_next(309-456)flb_log_event_decoder_destroy(151-178)src/flb_record_accessor.c (1)
flb_ra_translate(628-633)src/flb_sds.c (3)
flb_sds_destroy(389-399)flb_sds_create_size(92-95)flb_sds_cat(120-141)src/flb_pack.c (1)
flb_msgpack_raw_to_json_sds(1026-1085)src/flb_upstream.c (2)
flb_upstream_conn_get(711-844)flb_upstream_conn_release(862-947)src/flb_gzip.c (1)
flb_gzip_compress(157-252)src/flb_http_client.c (5)
flb_http_client(814-859)flb_http_add_header(963-995)flb_http_set_content_encoding_gzip(1123-1132)flb_http_do(1572-1632)flb_http_client_destroy(1688-1695)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (2)
flb_arvancloud_conf_create(27-156)flb_arvancloud_conf_destroy(158-189)
Comment on lines
+53
to
+56
| if (flb_output_config_map_set(ins, (void *) ctx) == -1) { | ||
| flb_plg_error(ins, "flb_output_config_map_set failed"); | ||
| flb_free(ctx); | ||
| return NULL; |
There was a problem hiding this comment.
Call the destructor on config-map failures
If flb_output_config_map_set() populates any of the SDS fields and then errors out, the early flb_free(ctx) leaks those allocations. Reuse the destructor so the partially-initialized fields get cleaned up.
if (flb_output_config_map_set(ins, (void *) ctx) == -1) {
flb_plg_error(ins, "flb_output_config_map_set failed");
- flb_free(ctx);
+ flb_arvancloud_conf_destroy(ctx);
return NULL;
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if (flb_output_config_map_set(ins, (void *) ctx) == -1) { | |
| flb_plg_error(ins, "flb_output_config_map_set failed"); | |
| flb_free(ctx); | |
| return NULL; | |
| if (flb_output_config_map_set(ins, (void *) ctx) == -1) { | |
| flb_plg_error(ins, "flb_output_config_map_set failed"); | |
| flb_arvancloud_conf_destroy(ctx); | |
| return NULL; |
🤖 Prompt for AI Agents
In plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c around lines 53
to 56, the error path currently calls flb_free(ctx) after
flb_output_config_map_set() fails, which leaks any partially-initialized SDS
fields; replace the flb_free(ctx) with a call to the plugin's destructor to
clean up partially-initialized fields (e.g., arvancloud_cloudlogs_conf_free(ctx)
or the actual destructor defined in this file) so the config-map populated
allocations are released before returning NULL.
Comment on lines
+418
to
+422
| c = flb_http_client(u_conn, FLB_HTTP_POST, ctx->uri, | ||
| final_payload, final_payload_size, | ||
| NULL, 0, // host=NULL, port=0 -> use from u_conn | ||
| NULL, 0); // proxy=NULL | ||
| if (!c) { |
There was a problem hiding this comment.
Propagate host/proxy info to the HTTP client
When a proxy is configured, the current call still leaves host/proxy NULL, so the client talks to the proxy with a wrong Host header and never negotiates the CONNECT tunnel—requests won’t reach ArvanCloud. Pass the upstream host/port and the optional proxy string to flb_http_client() so both direct and proxied connections work.
- c = flb_http_client(u_conn, FLB_HTTP_POST, ctx->uri,
- final_payload, final_payload_size,
- NULL, 0, // host=NULL, port=0 -> use from u_conn
- NULL, 0); // proxy=NULL
+ c = flb_http_client(u_conn, FLB_HTTP_POST, ctx->uri,
+ final_payload, final_payload_size,
+ ctx->host, ctx->port,
+ ctx->proxy ? ctx->proxy : NULL, 0);🤖 Prompt for AI Agents
In plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c around lines 418-422,
the flb_http_client() call leaves host and proxy NULL which causes wrong Host
headers and prevents CONNECT proxy tunneling; update the call to pass the
upstream connection's host and port (e.g. u_conn->host and u_conn->port) into
the host/port parameters and pass the configured proxy string (and its length)
into the proxy parameters (e.g. ctx->proxy and proxy_len or equivalent) so the
HTTP client can set the correct Host header and establish CONNECT when a proxy
is used.
This patch adds a new output plugin that sends logs to ArvanCloud CloudLogs service. The plugin supports: - GZIP compression - Configurable timestamp formats - Custom log fields mapping - Retry logic and error handling The plugin has been tested with ArvanCloud CloudLogs API and successfully delivers logs with proper formatting and timestamps. Signed-off-by: Javad Nasrolahi <[email protected]>
javadnasrolahi
force-pushed
the
add-out-arvancloud-cloudlogs
branch
from
c414de0 to
b14d48d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch adds a new output plugin that sends logs to ArvanCloud CloudLogs service. The plugin supports:
The plugin has been tested with ArvanCloud CloudLogs API and successfully delivers logs with proper formatting and timestamps.
Summary by CodeRabbit