-
Notifications
You must be signed in to change notification settings - Fork 537
feat(otelcol.receiver): add awss3 receiver #4928
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
c63c5e3
feat: add awss3receiver component template
x1unix d515b74
fix: imports
x1unix 7e3465a
feat: map config
x1unix a11648f
feat: remove currently not supported Encodings and Notification blocks
x1unix e0a3d88
fix: filename
x1unix ad4a8a1
feat: add arguments test
x1unix 120eb90
feat: register component
x1unix 29c4888
feat: add awss3 otelconverter
x1unix 7a44998
chore: update changelog
x1unix 697f3a7
fix: state type
x1unix 1005973
feat: add component docs
x1unix 1811faa
chore: make generate-gh-issue-templates
x1unix cc40b3c
fix: docs link typo
x1unix aaf3d5f
fix: otelconvert fixtures
x1unix 27a70ae
fix: typo
x1unix 8a1962c
fix: doc
x1unix 9a2d580
chore: go generate ./internal/tools/docs_generator/
x1unix 64790ec
feat: map awss3receiver defaults to SetToDefault
x1unix 70a6072
feat: add config defaults break change test
x1unix 0ccb8f9
fix: apply doc suggestions from code review
x1unix 14173ba
fix: tests
x1unix a7f7e61
fix: Apply suggestions from code review
x1unix 4fa3a29
fix: mention that awss3 supports any kind of telemetry
x1unix 54b8197
fix: codegen
x1unix File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
221 changes: 221 additions & 0 deletions
221
docs/sources/reference/components/otelcol/otelcol.receiver.awss3.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,221 @@ | ||
| --- | ||
| canonical: https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.receiver.awss3/ | ||
| description: Learn about otelcol.receiver.awss3 | ||
| labels: | ||
| stage: experimental | ||
| products: | ||
| - oss | ||
| title: otelcol.receiver.awss3 | ||
| --- | ||
|
|
||
| # `otelcol.receiver.awss3` | ||
|
|
||
| {{< docs/shared lookup="stability/experimental.md" source="alloy" version="<ALLOY_VERSION>" >}} | ||
|
|
||
| `otelcol.receiver.awss3` receives telemetry stored in S3 by the [AWS S3 Exporter](./otelcol.exporter.awss3.md). | ||
|
|
||
| {{< admonition type="warning" >}} | ||
| `otelcol.receiver.awss3` is a wrapper over the upstream OpenTelemetry Collector [`awss3`][] receiver. | ||
| Bug reports or feature requests will be redirected to the upstream repository, if necessary. | ||
|
|
||
| [`awss3`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/{{< param "OTEL_VERSION" >}}/receiver/awss3receiver | ||
| {{< /admonition >}} | ||
|
|
||
| The receiver has two modes of operation: | ||
|
|
||
| - **Time Range Mode** - Specify start and end to fetch data from a specific time range. | ||
| - **SQS Message Mode** - Subscribe to SQS messages to process objects as they arrive. | ||
|
|
||
| The receiver supports the following encodings: | ||
|
|
||
| - `otlp_json` (OpenTelemetry Protocol format represented as JSON) with a suffix of `.json` | ||
| - `otlp_proto` (OpenTelemetry Protocol format represented as Protocol Buffers) with a suffix of `.binpb` | ||
|
|
||
| {{< admonition type="note" >}} | ||
| Currently, `otelcol.receiver.awss3` receiver doesn't support encoding extensions. | ||
| {{< /admonition >}} | ||
|
|
||
| You can specify multiple `otelcol.receiver.awss3` components by giving them different labels. | ||
|
|
||
| ## Usage | ||
|
|
||
| ```alloy | ||
| otelcol.receiver.awss3 "<LABEL>" { | ||
| start_time = "..." | ||
| end_time = "..." | ||
|
|
||
| s3downloader { | ||
| s3_bucket = "..." | ||
| s3_prefix = "..." | ||
| } | ||
|
|
||
| output { | ||
| logs = [...] | ||
| metrics = [...] | ||
| trace = [...] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Arguments | ||
|
|
||
| You can use the following arguments with `otelcol.receiver.awss3`: | ||
|
|
||
| | Name | Type | Description | Default | Required | | ||
| | ------------ | -------- | ------------------------------------------- | ------- | ----------------------------- | | ||
| | `start_time` | `string` | The time at which to start retrieving data. | | Required if fetching by time. | | ||
| | `end_time` | `string` | The time at which to stop retrieving data. | | Required if fetching by time. | | ||
|
|
||
| The `start_time` and `end_time` fields use one of the following time formats: RFC3339, `YYYY-MM-DD HH:MM`, or `YYYY-MM-DD`. When using `YYYY-MM-DD`, the time defaults to `00:00`. | ||
|
|
||
| {{< admonition type="note" >}} | ||
| Time-based configuration (`start_time` and `end_time` arguments) can't be combined together with [`sqs`][] block. | ||
|
|
||
| [`sqs`]: #sqs | ||
|
|
||
| {{< /admonition >}} | ||
|
|
||
| Refer to the upstream receiver [documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/{{< param "OTEL_VERSION" >}}/receiver/awss3receiver#configuration) for more details. | ||
|
|
||
| ## Blocks | ||
|
|
||
| You can use the following blocks with `otelcol.receiver.awss3`: | ||
|
|
||
| | Block | Description | Required | | ||
| | ------------------------------ | ---------------------------------------------------------------------------- | ----------------------------------------- | | ||
| | [`s3downloader`][s3downloader] | Configures S3 downloader. | yes | | ||
| | [`sqs`][sqs] | Configures SQS queue configuration for receiving object change notification. | Required if fetching by SQS notification. | | ||
| | [`output`][output] | Configures where to send received telemetry data. | yes | | ||
|
|
||
| [s3downloader]: #s3downloader | ||
| [sqs]: #sqs | ||
| [output]: #output | ||
|
|
||
| ### `s3downloader` | ||
|
|
||
| {{< badge text="Required" >}} | ||
|
|
||
| The `s3downloader` block contains AWS S3 downloader related configuration to control things like bucket, prefix, batching, connections, retries, etc. | ||
|
|
||
| The following arguments are supported: | ||
|
|
||
| | Name | Type | Description | Default | Required | | ||
| | ----------------------- | -------- | ----------------------------------------------------------------------------------------------------- | ------------- | -------- | | ||
| | `s3_bucket` | `string` | S3 bucket. | | yes | | ||
| | `s3_prefix` | `string` | Prefix for the S3 key (root directory inside bucket). | | yes | | ||
| | `endpoint_partition_id` | `string` | Partition id to use if `endpoint` is specified. | `"aws"` | no | | ||
| | `endpoint` | `string` | Overrides the endpoint used by the exporter instead of constructing it from `region` and `s3_bucket`. | | no | | ||
| | `file_prefix` | `string` | Prefix used to filter files for download. | | no | | ||
| | `region` | `string` | AWS region. | `"us-east-1"` | no | | ||
| | `s3_force_path_style` | `bool` | When enabled, forces the request to use [path-style addressing][s3-force-path-style-ref]. | `false` | no | | ||
| | `s3_partition` | `string` | Time granularity of S3 key: hour or minute. | `"minute"` | no | | ||
|
|
||
| [s3-force-path-style-ref]: http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html | ||
|
|
||
| ### `sqs` | ||
|
|
||
| The `sqs` block holds SQS queue configuration for receiving object change notifications. | ||
|
|
||
| The following arguments are supported: | ||
|
|
||
| | Name | Type | Description | Default | Required | | ||
| | ------------------------ | -------- | --------------------------------------------------------------- | ------- | -------- | | ||
| | `queue_url` | `string` | The URL of the SQS queue that receives S3 bucket notifications. | | yes | | ||
| | `region` | `string` | AWS region of the SQS queue. | | yes | | ||
| | `endpoint` | `string` | Custom endpoint for the SQS service. | | no | | ||
| | `max_number_of_messages` | `int` | Maximum number of messages to retrieve in a single SQS request. | `10` | no | | ||
| | `wait_time_seconds` | `int` | Wait time in seconds for long polling SQS requests. | `20` | no | | ||
|
|
||
| {{< admonition type="note" >}} | ||
| You must configure your S3 bucket to send event notifications to the SQS queue. | ||
| Time-based configuration (`start_time`/`end_time`) and SQS configuration can't be used together. | ||
| {{< /admonition >}} | ||
|
|
||
| ### `output` | ||
|
|
||
| {{< badge text="Required" >}} | ||
|
|
||
| {{< docs/shared lookup="reference/components/output-block.md" source="alloy" version="<ALLOY_VERSION>" >}} | ||
|
|
||
| ## Exported fields | ||
|
|
||
| `otelcol.receiver.awss3` doesn't export any fields. | ||
|
|
||
| ## Component health | ||
|
|
||
| `otelcol.receiver.awss3` is only reported as unhealthy if given an invalid configuration. | ||
|
|
||
| ## Debug information | ||
|
|
||
| `otelcol.receiver.awss3` doesn't expose any component-specific debug information. | ||
|
|
||
| ## Example | ||
|
|
||
| This example forwards received traces through a batch processor before finally sending it to an OTLP-capable endpoint: | ||
|
|
||
| ```alloy | ||
| // Time range mode: | ||
| otelcol.receiver.awss3 "default" { | ||
| start_time = "2024-01-01 01:00" | ||
| end_time = "2024-01-02" | ||
|
|
||
| s3downloader { | ||
| region = "us-west-1" | ||
| s3_bucket = "mybucket" | ||
| s3_prefix = "trace" | ||
| s3_partition = "minute" | ||
| } | ||
|
|
||
| output { | ||
| traces = [otelcol.processor.batch.default.input] | ||
| } | ||
| } | ||
|
|
||
| // SQS message mode: | ||
| otelcol.receiver.awss3 "sqs_traces" { | ||
| s3downloader { | ||
| region = "us-east-1" | ||
| s3_bucket = "mybucket" | ||
| s3_prefix = "mytrace" | ||
| } | ||
|
|
||
| sqs { | ||
| queue_url = "https://sqs.us-east-1.amazonaws.com/123456789012/test-queue" | ||
| region = "us-east-1" | ||
| } | ||
|
|
||
| output { | ||
| traces = [otelcol.processor.batch.default.input] | ||
| } | ||
| } | ||
|
|
||
| otelcol.processor.batch "default" { | ||
| output { | ||
| metrics = [otelcol.exporter.otlp.default.input] | ||
| logs = [otelcol.exporter.otlp.default.input] | ||
| traces = [otelcol.exporter.otlp.default.input] | ||
| } | ||
| } | ||
|
|
||
| otelcol.exporter.otlp "default" { | ||
| client { | ||
| endpoint = sys.env("<OTLP_ENDPOINT>") | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| <!-- START GENERATED COMPATIBLE COMPONENTS --> | ||
|
|
||
| ## Compatible components | ||
|
|
||
| `otelcol.receiver.awss3` can accept arguments from the following components: | ||
|
|
||
| - Components that export [OpenTelemetry `otelcol.Consumer`](../../../compatibility/#opentelemetry-otelcolconsumer-exporters) | ||
|
|
||
|
|
||
| {{< admonition type="note" >}} | ||
| Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. | ||
| Refer to the linked documentation for more details. | ||
| {{< /admonition >}} | ||
|
|
||
| <!-- END GENERATED COMPATIBLE COMPONENTS --> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.