Cloudwatch Receiver

Status
Stability	alpha: logs
Distributions	contrib
Issues
Code coverage
Code Owners	@schmikei | Seeking more code owners!
Emeritus	@djaglowski

Receives Cloudwatch events from AWS Cloudwatch via the AWS SDK for Cloudwatch Logs

Getting Started

This receiver uses the AWS SDK as mode of authentication, which includes Credentials File and IMDS authentication for EC2 instances.

Configuration

Top Level Parameters

Parameter	Notes	type	Description
region	required	string	The AWS recognized region string
profile	optional	string	The AWS profile used to authenticate, if none is specified the default is chosen from the list of profiles
imds_endpoint	optional	string	A way of specifying a custom URL to be used by the EC2 IMDS client to validate the session. If unset, and the environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT has a value the client will use the value of the environment variable as the endpoint for operation calls.
logs	optional	Logs	Configuration for Logs ingestion of this receiver
storage	optional	string	The ID of a storage extension to be used for state persistence.

Logs Parameters

Parameter	Type	Default	Description
start_from	String	Read all available logs from the beginning	Timestamp in RFC3339 format (e.g., 2006-01-02T15:04:05Z07:00) indicating where to start reading logs
poll_interval	Duration	1 minute	Time to wait between log requests
max_events_per_request	Integer	1,000	The maximum number of events to process per request to Cloudwatch
groups	Optional	All Log Groups and Streams	Configuration for Log Groups, by default all Log Groups and Log Streams will be collected.

Group Parameters

autodiscover and named are ways to control and filter which log groups and log streams which are collected from. They are mutually exclusive and are incompatible to be configured at the same time.

• autodiscover
  • limit: (optional; default = 50) Limits the number of discovered log groups. This does not limit how large each API call to discover the log groups will be.
  • prefix: (optional) A prefix for log groups to limit the number of log groups discovered.
    • Only one of prefix or pattern can be specified. If both are omitted, all log streams up to the limit are collected.
  • pattern: (optional) A case-sensitive substring (not a regular expression) that must be present in the log group names, used to limit the number of log groups discovered.
    • Only one of prefix or pattern can be specified. If both are omitted, all log streams up to the limit are collected.
  • account_identifiers: (optional) A list of AWS account IDs to filter log groups by. Only log groups from the specified accounts will be discovered.
  • include_linked_accounts: (optional; default = false) When using a monitoring account, set this to true to have autodiscovery return log groups in the accounts listed in account_identifiers. If this is true and account_identifiers contains a null value, all log groups in the monitoring account and in all linked source accounts are returned (behavior is handled by the AWS CloudWatch Logs API).
  • streams: (optional) If streams is omitted, then all streams will be attempted to retrieve events from.
    • names: A list of full log stream names to filter the discovered log groups to collect from.
    • prefixes: A list of prefixes to filter the discovered log groups to collect from.
• named
  • This is a map of log group name to stream filtering options
    • streams: (optional)
      • names: A list of full log stream names to filter the discovered log groups to collect from.
      • prefixes: A list of prefixes to filter the discovered log groups to collect from.

Autodiscovery Example Configuration

awscloudwatch:
  region: us-west-1
  logs:
    poll_interval: 1m
    groups:
      autodiscover:
        limit: 100
        prefix: /aws/eks/
        streams:
          prefixes: [kube-api-controller]

Autodiscovery with Account ID Filtering Example

awscloudwatch:
  region: us-west-1
  logs:
    poll_interval: 1m
    groups:
      autodiscover:
        limit: 100
        account_identifiers: ["123456789012", "987654321098"]
        include_linked_accounts: true
        prefix: /aws/lambda/

Named Example

awscloudwatch:
  region: us-west-1
  logs:
    poll_interval: 5m
    groups:
      named:
        /aws/eks/dev-0/cluster: 
          names: [kube-apiserver-ea9c831555adca1815ae04b87661klasdj]

Sample Configs

This receiver has a number of sample configs for reference.

1. Default

   • Minimal configuration of the receiver
   • Performs autodiscovery
   • Collects all log groups and log streams

2. Autodiscover Filtering Log Groups

   • Performs autodiscovery
   • Only collects log groups matching a prefix
   • Limits the number of discovered Log Groups

3. Autodiscover Filtering Log Streams

   • Performs autodiscovery for all Log Groups
   • Filters log streams

4. Named Groups

   • Specifies and only collects from the desired Log Groups
   • Does not attempt autodiscovery

5. Named Groups Filter Log Streams

   • Specifies the names of the log groups to collect
   • Does not attempt autodiscovery
   • Only collects from log streams matching a prefix