|
| 1 | +## `aws_cloudwatch_input` operator |
| 2 | + |
| 3 | +The `aws_cloudwatch_input` operator reads logs from AWS Cloudwatch Logs using [AWS's SDK](https://github.com/aws/aws-sdk-go). |
| 4 | + |
| 5 | +Fields `log_group`, `log_stream`,`region`, and `event_id` are promoted to resource field. The `Timestamp` field of the event is parsed as the entry's timestamp. |
| 6 | + |
| 7 | +Credentials are used in the following order. |
| 8 | + |
| 9 | +- Environment Variables (Details [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html)) |
| 10 | +- Shared Credentials file (Details [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html)), |
| 11 | +- Shared Configuration file (if SharedConfig is enabled details [here](https://docs.aws.amazon.com/sdkref/latest/guide/creds-config-files.html)) , |
| 12 | +- EC2 Instance Metadata (credentials only details [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-metadata.html)). |
| 13 | + |
| 14 | +You can provide `profile` to specify which credential set to use from a Shared Credentials file. |
| 15 | + |
| 16 | +### Configuration Fields |
| 17 | + |
| 18 | +| Field | Default | Description | |
| 19 | +| --- | --- | --- | |
| 20 | +| `id` | `aws_cloudwatch_input` | A unique identifier for the operator. | |
| 21 | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | |
| 22 | +| `log_group_name` | required | The Cloudwatch Logs Log Group Name. | |
| 23 | +| `region` | required | The AWS Region to be used. | |
| 24 | +| `log_stream_name_prefix` | | The log stream name prefix to use. This will find any log stream name in the group with the starting prefix. Cannot be used with `log_stream_names` | |
| 25 | +| `log_stream_names` | | An array of log stream names to get events from. Cannot be used with `log_stream_name_prefix` | |
| 26 | +| `profile` | | Profile to use for authentication. Details on named profiles can be found [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) | |
| 27 | +| `event_limit` | `10000` | The maximum number of events to return per call. | |
| 28 | +| `poll_interval` | `1m` | The duration between event calls. | |
| 29 | +| `start_at` | `end` | At startup, where to start reading events. Options are `beginning` or `end` | |
| 30 | + |
| 31 | +### Log Stream Name Prefix |
| 32 | + |
| 33 | +The log_stream_prefix allows the use of "directives" such as `%Y` (4-digit year) and `%d` (2-digit zero-padded day). These directives are based on `strptime` directives. There are a limited set of the `strptime` directives. These directives are listed below. When directive is detected within the prefix it will replace the first occurance of directive with the data indicated in the description. |
| 34 | + |
| 35 | +#### Supported directives |
| 36 | + |
| 37 | +| Directive | Description | |
| 38 | +| :---: | :--- | |
| 39 | +| %Y | Year, zero-padded | |
| 40 | +| %y | Year, last two digits, zero-padded | |
| 41 | +| %m | Month, zero-padded | |
| 42 | +| %q | Month as a unpadded number | |
| 43 | +| %b | Abbreviated month name | |
| 44 | +| %h | Abbreviated month name | |
| 45 | +| %B | Full month name | |
| 46 | +| %d | Day of the month, zero-padded | |
| 47 | +| %g | Day of the month, unpadded | |
| 48 | +| %a | Abbreviated weekday name | |
| 49 | +| %A | Full weekday name | |
| 50 | + |
| 51 | +### Example Configurations |
| 52 | + |
| 53 | +#### Simple AWS Cloudwatch Logs Example Input |
| 54 | + |
| 55 | +Configuration: |
| 56 | + |
| 57 | +```yaml |
| 58 | +pipeline: |
| 59 | +- type: aws_cloudwatch_input |
| 60 | + LogGroupName: "/aws/lambda/service" |
| 61 | + Region: us-east-2 |
| 62 | +``` |
| 63 | +
|
| 64 | +### Simple AWS Cloudwatch Logs Example Output |
| 65 | +
|
| 66 | +```json |
| 67 | +{ |
| 68 | + "timestamp": "2021-05-10T13:00:55.023-04:00", |
| 69 | + "severity": 0, |
| 70 | + "record": { |
| 71 | + "event_id": "36142060744975733945009868546041203920891749688822923267", |
| 72 | + "ingestion_time": 1620666055330, |
| 73 | + "log_stream_name": "2021/05/10/[$LATEST]ff09d08f2836494690a1bd6b77365502", |
| 74 | + "message": "REPORT RequestId: 291fe36c-116a-42fd-a563-a8615671bab9\tDuration: 4577.28 ms\tBilled Duration: 4578 ms\tMemory Size: 128 MB\tMax Memory Used: 68 MB\tInit Duration: 401.54 ms\t\n" |
| 75 | + } |
| 76 | +} |
| 77 | +``` |
| 78 | + |
| 79 | +#### Log Stream Prefix Directives Example Input |
| 80 | + |
| 81 | +Configuration: |
| 82 | + |
| 83 | +```yaml |
| 84 | +pipeline: |
| 85 | +- type: aws_cloudwatch_input |
| 86 | + log_group_name: "/aws/lambda/service" |
| 87 | + region: us-east-2 |
| 88 | + log_stream_name_prefix: "%Y/%m/%d" |
| 89 | +``` |
| 90 | +
|
| 91 | +### Log Stream Prefix Directives Example Output |
| 92 | +
|
| 93 | +```json |
| 94 | +{ |
| 95 | + "timestamp": "2021-05-12T13:03:47.941-04:00", |
| 96 | + "severity": 0, |
| 97 | + "resource": { |
| 98 | + "event_id": "36145918169946098276207227425947415203911741965970309123", |
| 99 | + "log_group": "/aws/lambda/service", |
| 100 | + "log_stream": "2021/05/12/[$LATEST]0f36de8f623a491c9305990130201669", |
| 101 | + "region": "us-east-2" |
| 102 | + }, |
| 103 | + "record": { |
| 104 | + "ingestion_time": 1620839035104, |
| 105 | + "message": "REPORT RequestId: d64685ba-913b-456f-acd7-d00021416e68\tDuration: 1852.30 ms\tBilled Duration: 1853 ms\tMemory Size: 128 MB\tMax Memory Used: 68 MB\t\n" |
| 106 | + } |
| 107 | +} |
| 108 | +``` |
| 109 | + |
| 110 | +#### Log Stream Names Example Input |
| 111 | + |
| 112 | +Configuration: |
| 113 | + |
| 114 | +```yaml |
| 115 | +pipeline: |
| 116 | +- type: aws_cloudwatch_input |
| 117 | + log_group_name: "/aws/lambda/service" |
| 118 | + region: us-east-2 |
| 119 | + log_stream_names: |
| 120 | + - "2021/05/09/[$LATEST]62e990bb0e72460c95b1dcfc5d96adc5" |
| 121 | + - "2021/05/08/[$LATEST]84d663604b6845e987d278272455ed95" |
| 122 | +``` |
| 123 | +
|
| 124 | +### Log Stream Names Example Output |
| 125 | +
|
| 126 | +```json |
| 127 | +{ |
| 128 | + "timestamp": "2021-05-09T13:04:02.686-04:00", |
| 129 | + "severity": 0, |
| 130 | + "resource": { |
| 131 | + "event_id": "36140138145615327091042663253954182481286730645124743171", |
| 132 | + "log_group": "/aws/lambda/service", |
| 133 | + "log_stream": "2021/05/09/[$LATEST]62e990bb0e72460c95b1dcfc5d96adc5", |
| 134 | + "region": "us-east-2" |
| 135 | + }, |
| 136 | + "record": { |
| 137 | + "ingestion_time": 1620579849837, |
| 138 | + "message": "REPORT RequestId: 346b9fa2-9117-4d41-89f8-071f0100213b\tDuration: 1865.27 ms\tBilled Duration: 1866 ms\tMemory Size: 128 MB\tMax Memory Used: 68 MB\t\n" |
| 139 | + } |
| 140 | +} |
| 141 | +``` |
0 commit comments