Support Amazon Simple Storage Service (AWS S3) metrics monitoring

.github/workflows/skywalking.yaml

@@ -670,6 +670,8 @@ jobs:
             config: test/e2e-v2/cases/aws/eks/e2e.yaml
           - name: Windows
             config: test/e2e-v2/cases/win/e2e.yaml
+          - name: AWS Cloud S3
+            config: test/e2e-v2/cases/aws/s3/e2e.yaml
     steps:
       - uses: actions/checkout@v3
         with:

docs/en/changes/changes.md

@@ -95,6 +95,7 @@
 * Add Python HTTPX module component ID(7019).
 * Migrate tests from junit 4 to junit 5.
 * Refactor http-based alarm plugins and extract common logic to `HttpAlarmCallback`.
+* Support Amazon Simple Storage Service (Amazon S3) metrics monitoring
 
 #### UI
 

docs/en/setup/backend/aws-firehose-receiver.md

@@ -14,6 +14,12 @@ Usually, the [AWS CloudWatch metrics](https://docs.aws.amazon.com/AmazonCloudWat
 CloudWatch metrics with S3 -->  CloudWatch Metric Stream (OpenTelemetry formart) --> Kinesis Data Firehose Delivery Stream --> AWS Firehose receiver(OAP) --> OpenTelemetry receiver(OAP)
 ```
 
+## Supported metrics
+
+| Description                             | Configuration File                  | Data Source                                                                                                                                       |
+|-----------------------------------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|
+| Metrics of AWS Cloud S3                 | otel-rules/aws-s3/s3-service.yaml   | AWS CloudWatcher Metrics Stream -> AWS Firehose delivery stream -> SkyWalking OAP Server with [AWS Firehose receiver](./aws-firehose-receiver.md) |
+
 ## Notice
 
 1. Only OpenTelemetry format is supported (refer to [Metric streams output formats](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-formats.html))

docs/en/setup/backend/backend-aws-s3-monitoring.md

@@ -0,0 +1,37 @@
+# AWS Cloud S3 monitoring
+Amazon Simple Storage Service (Amazon S3) is an object storage service. SkyWalking leverages [AWS Kinesis Data Firehose receiver](./aws-firehose-receiver.md) to transfer the CloudWatch metrics of s3 to
+[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/meter.md).
+
+### Data flow
+1. AWS CloudWatch collect metrics for S3, refer to [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html)
+2. [CloudWatch metric streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html) stream CloudWatch metrics of S3 to AWS Kinesis Data Firehose
+3. AWS Kinesis Data Firehose delivery metrics to [AWS Kinesis Data Firehose receiver](./aws-firehose-receiver.md) through the HTTP endpoint
+
+### Set up
+1. Create CloudWatch metrics configuration for S3, refer to [S3 metrics configuration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/metrics-configurations.html)
+2. Create an Amazon Kinesis Data Firehose Delivery Stream, and set [AWS Kinesis Data Firehose receiver](./aws-firehose-receiver.md)'s address as HTTP(s) Destination, refer to [Create Delivery Stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html)
+3. Create CloudWatch metric stream, and select the Firehose Delivery Stream which has been created above, set `Select namespaces` to `AWS/S3`, `Select output format` to `OpenTelemetry 0.7`. refer to [CloudWatch Metric Streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html)
+
+### S3 Monitoring
+
+SkyWalking observes CloudWatch metrics of the S3 bucket, which is cataloged as a `LAYER: AWS_S3` `Service` in the OAP.
+
+#### Supported Metrics
+
+| Monitoring Panel           | Unit  | Metric Name                 | Catalog        | Description                                                                                                                                | Data Source                                                                                                       |
+|----------------------------|-------|-----------------------------|----------------|--------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------|
+| 4xx Errors                 | count | aws_s3_4xx                  | Service        | The number of HTTP 4xx client error status code requests made to the S3 bucket                                                             | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
+| 5xx Errors                 | count | aws_s3_5xx                  | Service        | The number of HTTP 5xx client error status code requests made to the S3 bucket                                                             | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
+| Downloaded                 | bytes | aws_s3_downloaded_bytes     | Service        | The number of bytes downloaded for requests made to an Amazon S3 bucket                                                                    | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
+| Uploaded                   | bytes | aws_s3_uploaded_bytes       | Service        | The number of bytes uploaded for requests made to an Amazon S3 bucket                                                                      | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
+| Request Average Latency    | bytes | aws_s3_request_latency      | Service        | The average of elapsed per-request time from the first byte received to the last byte sent to an Amazon S3 bucket                          | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
+| First Byte Average Latency | bytes | aws_s3_request_latency      | Service        | The average of per-request time from the complete request being received by an Amazon S3 bucket to when the response starts to be returned | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
+| All Requests               | bytes | aws_s3_delete_requests      | Service        | The number of HTTP All requests made for objects in an Amazon S3 bucket                                                                    | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
+| Get Requests               | bytes | aws_s3_delete_requests      | Service        | The number of HTTP Get requests made for objects in an Amazon S3 bucket                                                                    | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
+| Put Requests               | bytes | aws_s3_delete_requests      | Service        | The number of HTTP PUT requests made for objects in an Amazon S3 bucket                                                                    | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
+| Delete Requests            | bytes | aws_s3_delete_requests      | Service        | The number of HTTP Delete requests made for objects in an Amazon S3 bucket                                                                 | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
+
+### Customizations
+You can customize your own metrics/expression/dashboard panel.
+The metrics definition and expression rules are found in `/config/otel-rules/aws-s3/`.
+The AWS Cloud EKS dashboard panel configurations are found in `/config/ui-initialized-templates/aws_s3`.

docs/menu.yml

@@ -189,6 +189,8 @@ catalog:
             catalog:
               - name: "EKS Monitoring"
                 path: "/en/setup/backend/backend-aws-eks-monitoring"
+              - name: "S3 Monitoring"
+                path: "/en/setup/backend/backend-aws-s3-monitoring"
           - name: "Browser Monitoring"
             path: "/en/setup/service-agent/browser-agent"
           - name: "Gateway Monitoring"

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java

@@ -146,8 +146,12 @@ public enum Layer {
     /**
      * Windows Machine
      */
-    OS_WINDOWS(23, true);
+    OS_WINDOWS(23, true),
 
+    /**
+     * Amazon Simple Storage Service (Amazon S3) is an object storage service provided by AWS Cloud
+     */
+    AWS_S3(24, true);
 
     private final int value;
     /**

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java

@@ -62,6 +62,7 @@ public class UITemplateInitializer {
         Layer.VIRTUAL_MQ.name(),
         Layer.AWS_EKS.name(),
         Layer.OS_WINDOWS.name(),
+        Layer.AWS_S3.name(),
         "custom"
     };
     private final UITemplateManagementService uiTemplateManagementService;

oap-server/server-starter/src/main/resources/application.yml

@@ -356,7 +356,7 @@ receiver-otel:
   selector: ${SW_OTEL_RECEIVER:default}
   default:
     enabledHandlers: ${SW_OTEL_RECEIVER_ENABLED_HANDLERS:"oc,otlp"}
-    enabledOtelRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_RULES:"apisix,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows"}
+    enabledOtelRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_RULES:"apisix,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*"}
 
 receiver-zipkin:
   selector: ${SW_RECEIVER_ZIPKIN:-}

oap-server/server-starter/src/main/resources/otel-rules/aws-s3/s3-service.yaml

@@ -0,0 +1,55 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This will parse a textual representation of a duration. The formats
+# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS}
+# with days considered to be exactly 24 hours.
+# <p>
+# Examples:
+# <pre>
+#    "PT20.345S" -- parses as "20.345 seconds"
+#    "PT15M"     -- parses as "15 minutes" (where a minute is 60 seconds)
+#    "PT10H"     -- parses as "10 hours" (where an hour is 3600 seconds)
+#    "P2D"       -- parses as "2 days" (where a day is 24 hours or 86400 seconds)
+#    "P2DT3H4M"  -- parses as "2 days, 3 hours and 4 minutes"
+#    "P-6H3M"    -- parses as "-6 hours and +3 minutes"
+#    "-P6H3M"    -- parses as "-6 hours and -3 minutes"
+#    "-P-6H+3M"  -- parses as "+6 hours and -3 minutes"
+# </pre>
+
+filter: "{ tags -> {tags.cloud_provider == 'aws' && tags.Namespace == 'AWS/S3' } }"
+expSuffix: tag({tags -> tags.bucket = 'aws-s3::' + tags.BucketName}).service(['bucket'], Layer.AWS_S3)
+metricPrefix: aws_s3
+metricsRules:
+  - name: 4xx
+    exp: amazonaws_com_AWS_S3_4xxErrors_sum.downsampling(SUM)
+  - name: 5xx
+    exp: amazonaws_com_AWS_S3_5xxErrors_sum.downsampling(SUM)
+  - name: downloaded_bytes
+    exp: amazonaws_com_AWS_S3_BytesDownloaded_sum.downsampling(SUM)
+  - name: uploaded_bytes
+    exp: amazonaws_com_AWS_S3_BytesUploaded_sum.downsampling(SUM)
+  - name: request_latency
+    exp: amazonaws_com_AWS_S3_TotalRequestLatency_sum.div(amazonaws_com_AWS_S3_TotalRequestLatency_count)
+  - name: first_latency_bytes
+    exp: amazonaws_com_AWS_S3_FirstByteLatency_sum.div(amazonaws_com_AWS_S3_FirstByteLatency_count)
+  - name: all_requests
+    exp: amazonaws_com_AWS_S3_AllRequests_sum.downsampling(SUM)
+  - name: put_requests
+    exp: amazonaws_com_AWS_S3_PutRequests_sum.downsampling(SUM)
+  - name: get_requests
+    exp: amazonaws_com_AWS_S3_GetRequests_sum.downsampling(SUM)
+  - name: delete_requests
+    exp: amazonaws_com_AWS_S3_DeleteRequests_sum.downsampling(SUM)

oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_eks/aws_eks-root.json

@@ -50,7 +50,7 @@
           "graph": {
             "fontColor":"blue",
             "backgroundColor":"white",
-            "content":"Provide AWS Could monitoring through AWS Container Insights Receiver",
+            "content":"Provide AWS Cloud EKS monitoring through AWS Container Insights Receiver",
             "fontSize":14,
             "textAlign":"left",
             "url":"https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-aws-eks-monitoring/"

oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_s3/aws_s3-root.json

@@ -0,0 +1,68 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+[
+  {
+    "id": "AWS_S3-Root",
+    "configuration": {
+      "children": [
+        {
+          "x": 0,
+          "y": 2,
+          "w": 24,
+          "h": 51,
+          "i": "0",
+          "type": "Widget",
+          "widget": {
+            "title": ""
+          },
+          "graph": {
+            "type": "ServiceList",
+            "dashboardName": "AWS_S3-service",
+            "fontSize": 12,
+            "showXAxis": false,
+            "showYAxis": false,
+            "showGroup": true
+          },
+          "moved": false
+        },
+        {
+          "x": 0,
+          "y": 0,
+          "w": 24,
+          "h": 2,
+          "i": "100",
+          "type": "Text",
+          "graph": {
+            "fontColor":"blue",
+            "backgroundColor":"white",
+            "content":"Provide AWS Cloud S3 monitoring through AWS FireHose Receiver",
+            "fontSize":14,
+            "textAlign":"left",
+            "url":"https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-aws-s3-monitoring/"
+          },
+          "moved": false
+        }
+      ],
+      "id": "AWS_S3-Root",
+      "layer": "AWS_S3",
+      "entity": "All",
+      "name": "AWS_S3-Root",
+      "isRoot": true
+    }
+  }
+]