out_opentelemetry: Add 'encoding' option for OTLP/HTTP JSON export#11844
out_opentelemetry: Add 'encoding' option for OTLP/HTTP JSON export#11844meehanman wants to merge 1 commit into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2bd72a7804
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (strcasecmp(tmp, "json") == 0) { | ||
| ctx->use_json_encoding = FLB_TRUE; | ||
| } |
There was a problem hiding this comment.
Allowing encoding=json here breaks OTLP/gRPC exports: ctx->use_json_encoding drives the logs/metrics/traces JSON paths, but opentelemetry_post() still wraps that payload in gRPC framing (application/grpc) when grpc on is active. OTLP/gRPC expects protobuf messages, so this combination sends invalid wire payloads and causes export failures for gRPC users who enable the new option.
Useful? React with 👍 / 👎.
| if (ctx->use_json_encoding) { | ||
| return process_metrics_json(event_chunk, ctx); | ||
| } |
There was a problem hiding this comment.
This early return bypasses the existing metrics path that calls append_labels(ctx, cmt) before encoding, so configured add_label entries are silently dropped whenever encoding=json is used. That changes exported metric content compared to protobuf mode and regresses a documented plugin option.
Useful? React with 👍 / 👎.
| if (ctx->use_json_encoding) { | ||
| return otel_process_logs_json(event_chunk, ctx); | ||
| } |
There was a problem hiding this comment.
Returning to otel_process_logs_json() here skips the protobuf log-building path that enforces logs_max_resources and logs_max_scopes, so those limits are ignored under encoding=json. This can produce much larger-than-configured payloads and defeats users' resource/scope cap settings.
Useful? React with 👍 / 👎.
meehanman
force-pushed
the
out_otel-json-export
branch
from
723bc8a to
dc3bcbc
Compare
|
Moving to draft to test changes |
…encoder Introduces a new `encoding` config property for the out_opentelemetry plugin that accepts `protobuf` (default, existing behaviour) or `json` (new OTLP/HTTP JSON). gRPC + JSON is rejected at context-creation time with a clear error per the OTLP specification. New config property ------------------- encoding protobuf (default) | json When json, all log and metrics flushes use OTLP/HTTP JSON encoding. Profiles always use protobuf (no JSON encoder exists for cprofiles). Streaming log encoder (flb_opentelemetry_logs_to_otlp_json) ------------------------------------------------------------ Replace the previous AST-based encoder that called flb_json_mut_obj/ arr/str for every node (~80 malloc() calls per log record) with a single-pass streaming encoder that writes directly into a growing flb_sds_t output buffer. Allocation cost per flush drops from ~80×N heap opeations to ~4 geometric SDS reallocs regardless of N. Key properties: - Produces byte-identical output to the AST path for all unit tests - Handles all option flags: logs_body_key (singular + plural), require_otel_metadata, body_key_attributes, max_resources, max_scopes - Enforces max_scopes per resource via a stack-allocated (resource_hash → scope_count) tracker — no heap allocation - _pretty variant retains the AST path (debug tool, not hot path) Metrics encoder --------------- - Eliminate the intermediate msgpack re-encode step in process_metrics_json(); call flb_opentelemetry_metrics_to_otlp_json() directly after label injection, cutting encoding from 3 msgpack passes to 1 - Replace per-GROUP_START msgpack_sbuffer malloc+serialize for resource/ scope identity hashing with a zero-allocation XXH3 streaming hash (cfl_hash_64bits_reset/update/digest) - Check flb_sds_cat_safe() return value and propagate FLB_RETRY on allocation failure instead of continuing with a corrupt buffer - Remove_encode_msgpack.h include (no longer needed) Bug fixes --------- - Pass max_resources/max_scopes into ensure_default_logs_scope_state() so the default-scope path (require_otel_metadata=false) respects caps - Return 1 (skip record) instead of -1 (fatal) when a resource/scope limit is hit; treat it as a dropped record, not a pipeline error - Set skip_current_group=FLB_TRUE when append_logs_resource_state() fails after the resource-limit check - Fix sign-comparison in resource/scope limit guards: compare counts as size_t rather than casting to int - Move the "profiles JSON encoding is not supported" warning from process_profiles() (per-flush) to context creation (once at startup) - Return FLB_ERROR (not FLB_RETRY) when body_keys calloc fails in otel_process_logs_json() to avoid an infinite OOM retry loop Tests (tests/internal/opentelemetry.c) --------------------------------------- Add 7 new unit tests covering the JSON encoding path: - test_opentelemetry_logs_otlp_json_body_keys_array - test_opentelemetry_logs_otlp_json_body_key_attributes - test_opentelemetry_metrics_otlp_json_add_label - test_opentelemetry_logs_otlp_json_max_resources_cap - test_opentelemetry_logs_otlp_json_max_scopes_cap - test_opentelemetry_logs_otlp_json_body_key_dollar_prefix - test_opentelemetry_metrics_otlp_json_protobuf_default Verified: 26/26 unit tests pass, Valgrind 0 errors, integration test produces valid OTLP/HTTP JSON accepted by mock receiver. Signed-off-by: Dean Meehan <d3an.meehan@hotmail.com>
meehanman
force-pushed
the
out_otel-json-export
branch
from
d2aa5e2 to
7e511b7
Compare
1 participant
The OTLP/HTTP specification (v1.10.0) states:
Prior to this change, the
out_opentelemetryplugin only supported binary Protobuf (Content-Type: application/x-protobuf). This PR adds anencodingconfiguration option so users can select JSON encoding (Content-Type: application/json) when their OTLP/HTTP receiver requires or prefers it.Changes
New
encodingconfig option (out_opentelemetry)protobuf(default)application/x-protobufjsonapplication/jsonflb_opentelemetry_*_to_otlp_json()helpersPer the OTLP spec, JSON encoding applies to OTLP/HTTP only. The
encodingoption is silently ignored whengrpc onis set (gRPC always uses protobuf framing). Profiles have no JSON encoder in the current codebase; protobuf is used and a warning is emitted.content_typeparameter threaded throughopentelemetry_post()Previously the Content-Type was hardcoded to
application/x-protobuf. The parameter is now passed explicitly at each call site, keeping existing callers unaffected.Config map description fixes
grpc: removed misleadingautovalue (not valid forFLB_CONFIG_MAP_BOOL; accepted values areon/offonly).logs_body_key: was copy-pasted fromlogs_uriand incorrectly said "Specify an optional HTTP URI for the target OTel endpoint." Corrected to describe the record accessor pattern used to populate the OTLP log body.$-prefixed body key lookup fixRecord accessor patterns (e.g.
$message) include a$prefix that does not appear in the raw msgpack key. Abody_key_lookup_name()helper now strips the prefix before searching the record map, applied consistently in both the JSON and Protobuf encoders.Spec references
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
Example configuration (YAML)
Example configuration (classic .conf)
Debug log output
Valgrind output
Documentation
fluent/fluent-bit-docs#2587
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.