[Feature] Support AWS service monitoring

[wu-sheng]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

In the recent several months, mostly due to @pg-yang @yswdqz hard work in these two iterations. We are now having more and more infrastructure monitoring, e.g. Linux, Kubernetes, Database, Gateway, MQ server(TODO, such as Kafka), Scheduler(TODO, such as Airflow)

Let's begin to do some research for monitoring cloud infra. As we know, this would be long-term work, this issue should be a catalog to list all potential components/services

List to be completed

• EKS @pg-yang - Support monitoring AWS Cloud EKS #10199 plan to release in v9.4.0
• Amazon S3 @pg-yang Support Amazon Simple Storage Service (AWS S3) metrics monitoring #10407 plan to release in v9.4.0
• Amazon DynamoDB @yswdqz Support Monitoring DynamoDB. #10418 plan to release in v9.4.0
• Amazon API Gateway @pg-yang Support Amazon API Gateway monitoring #10645 plan to release in v9.5.0

Use case

We wouldn't trace the cloud infra from the code level, this is unrealistic. But most of good cloud services expose metrics through Prometheus, OpenTelemetry, or some cloud APIs. We could leverage those and integrate these monitoring in SkyWalking.

Making all metrics in one place side by side with their application performance data, and alerting through the same system.

@pg-yang told me, he would work on this after virtual MQ.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[pg-yang]

Yes

[pg-yang]

I updated the task list to add EKS and put it first as it's supported directly by https://github.com/aws-observability/aws-otel-collector, which is more clear and easy than others. Let's begin with EKS

[wu-sheng]

@pg-yang I noticed this, https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/awscloudwatchreceiver/README.md

I think we could use this for reading metrics of S3 and DynamoDB.

[pg-yang]

Let's pick up S3 as the second task

[wu-sheng]

@pg-yang, @yswdqz told me, he has interests to follow this too.
I think you could share what you have about workflow, data formats and important docs here.
And mostly, your next PR(aw-otel-receiver) would be widely used. We should follow the dataflow path you found and share.

[pg-yang]

Very happy that work with @yswdqz. I will share the information about aw-otel-receiver tonight. 🔢

[pg-yang]

From what I know

1. Amazon CloudWatch could collect metrics from most AWS services(e.g. S3, API Gateway).
2. CloudWatch metrics stream is a feature of CloudWatch, it could stream CloudWatch metrics to a destination of your choice. Actually by my test, the only option of destination is Amazon Kinesis Data Firehose
3. Amazon Kinesis Data Firehose could send data to HTTP endpoint(refer to Choose HTTP Endpoint), and the HTTP endpoint should follow the specification

So we could develop a module(let's name it aws-firehose-receiver temporarily). aws-firehose-receiver will follow the specification to receive HTTP request from Amazon Kinesis Data Firehose, and convert the HTTP request to OTLP request, and then invoke org.apache.skywalking.oap.server.receiver.otel.otlp.OpenTelemetryMetricHandler#export for reusing code in otel-receiver-plugin.

Here is the data flow of S3(others are similar)

graph LR;
S3 --> CloudWatch --> CloudWatch-stream --> Amazon-Kinesis-Data-Firehose  -->  aws-firehose-receiver --> OpenTelemetryMetricHandler

Loading

Here is some doc maybe you want to read

1. CloudWatch Metric Streams – Send AWS Metrics to Partners and to Your Apps in Real Time
2. S3 metrics with Cloud Watch
3. API Gateway metrics with Cloud Watch
4. DynamoDB metrics with Cloud Watch

Here is S3 metrics sample data in JSON format

{"metric_stream_name":"QuickFull-QwkHX6","account_id":"008835616606","region":"ap-northeast-1","namespace":"AWS/S3","metric_name":"HeadRequests","dimensions":{"BucketName":"metricstreams-quickfull-qwkhx6-cyp2xuov","FilterId":"test-"},"timestamp":1673092920000,"value":{"max":1.0,"min":1.0,"sum":1.0,"count":1.0},"unit":"Count"}
{"metric_stream_name":"QuickFull-QwkHX6","account_id":"008835616606","region":"ap-northeast-1","namespace":"AWS/S3","metric_name":"4xxErrors","dimensions":{"BucketName":"metricstreams-quickfull-qwkhx6-cyp2xuov","FilterId":"test-"},"timestamp":1673092920000,"value":{"max":0.0,"min":0.0,"sum":0.0,"count":1.0},"unit":"Count"}

Notice, CloudWatch metrics stream support JSON, OpenTelemetry format as output, maybe OpenTelemetry format is more convenient to convert HTTP request to OTLP request.

[yswdqz]

Thank you, I will take a try !