diff --git a/.github/workflows/skywalking.yaml b/.github/workflows/skywalking.yaml index fe6e025d684c..e74bbae57a40 100644 --- a/.github/workflows/skywalking.yaml +++ b/.github/workflows/skywalking.yaml @@ -670,6 +670,8 @@ jobs: config: test/e2e-v2/cases/aws/eks/e2e.yaml - name: Windows config: test/e2e-v2/cases/win/e2e.yaml + - name: AWS Cloud S3 + config: test/e2e-v2/cases/aws/s3/e2e.yaml steps: - uses: actions/checkout@v3 with: diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index 153b33de3cfa..47f800f617d5 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -95,6 +95,7 @@ * Add Python HTTPX module component ID(7019). * Migrate tests from junit 4 to junit 5. * Refactor http-based alarm plugins and extract common logic to `HttpAlarmCallback`. +* Support Amazon Simple Storage Service (Amazon S3) metrics monitoring #### UI diff --git a/docs/en/setup/backend/aws-firehose-receiver.md b/docs/en/setup/backend/aws-firehose-receiver.md index e422c46f4da7..254a3c7f81eb 100644 --- a/docs/en/setup/backend/aws-firehose-receiver.md +++ b/docs/en/setup/backend/aws-firehose-receiver.md @@ -14,6 +14,12 @@ Usually, the [AWS CloudWatch metrics](https://docs.aws.amazon.com/AmazonCloudWat CloudWatch metrics with S3 --> CloudWatch Metric Stream (OpenTelemetry formart) --> Kinesis Data Firehose Delivery Stream --> AWS Firehose receiver(OAP) --> OpenTelemetry receiver(OAP) ``` +## Supported metrics + +| Description | Configuration File | Data Source | +|-----------------------------------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------| +| Metrics of AWS Cloud S3 | otel-rules/aws-s3/s3-service.yaml | AWS CloudWatcher Metrics Stream -> AWS Firehose delivery stream -> SkyWalking OAP Server with [AWS Firehose receiver](./aws-firehose-receiver.md) | + ## Notice 1. Only OpenTelemetry format is supported (refer to [Metric streams output formats](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-formats.html)) diff --git a/docs/en/setup/backend/backend-aws-s3-monitoring.md b/docs/en/setup/backend/backend-aws-s3-monitoring.md new file mode 100644 index 000000000000..390502be7969 --- /dev/null +++ b/docs/en/setup/backend/backend-aws-s3-monitoring.md @@ -0,0 +1,37 @@ +# AWS Cloud S3 monitoring +Amazon Simple Storage Service (Amazon S3) is an object storage service. SkyWalking leverages [AWS Kinesis Data Firehose receiver](./aws-firehose-receiver.md) to transfer the CloudWatch metrics of s3 to +[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/meter.md). + +### Data flow +1. AWS CloudWatch collect metrics for S3, refer to [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) +2. [CloudWatch metric streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html) stream CloudWatch metrics of S3 to AWS Kinesis Data Firehose +3. AWS Kinesis Data Firehose delivery metrics to [AWS Kinesis Data Firehose receiver](./aws-firehose-receiver.md) through the HTTP endpoint + +### Set up +1. Create CloudWatch metrics configuration for S3, refer to [S3 metrics configuration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/metrics-configurations.html) +2. Create an Amazon Kinesis Data Firehose Delivery Stream, and set [AWS Kinesis Data Firehose receiver](./aws-firehose-receiver.md)'s address as HTTP(s) Destination, refer to [Create Delivery Stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html) +3. Create CloudWatch metric stream, and select the Firehose Delivery Stream which has been created above, set `Select namespaces` to `AWS/S3`, `Select output format` to `OpenTelemetry 0.7`. refer to [CloudWatch Metric Streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html) + +### S3 Monitoring + +SkyWalking observes CloudWatch metrics of the S3 bucket, which is cataloged as a `LAYER: AWS_S3` `Service` in the OAP. + +#### Supported Metrics + +| Monitoring Panel | Unit | Metric Name | Catalog | Description | Data Source | +|----------------------------|-------|-----------------------------|----------------|--------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------| +| 4xx Errors | count | aws_s3_4xx | Service | The number of HTTP 4xx client error status code requests made to the S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) | +| 5xx Errors | count | aws_s3_5xx | Service | The number of HTTP 5xx client error status code requests made to the S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) | +| Downloaded | bytes | aws_s3_downloaded_bytes | Service | The number of bytes downloaded for requests made to an Amazon S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) | +| Uploaded | bytes | aws_s3_uploaded_bytes | Service | The number of bytes uploaded for requests made to an Amazon S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) | +| Request Average Latency | bytes | aws_s3_request_latency | Service | The average of elapsed per-request time from the first byte received to the last byte sent to an Amazon S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) | +| First Byte Average Latency | bytes | aws_s3_request_latency | Service | The average of per-request time from the complete request being received by an Amazon S3 bucket to when the response starts to be returned | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) | +| All Requests | bytes | aws_s3_delete_requests | Service | The number of HTTP All requests made for objects in an Amazon S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) | +| Get Requests | bytes | aws_s3_delete_requests | Service | The number of HTTP Get requests made for objects in an Amazon S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) | +| Put Requests | bytes | aws_s3_delete_requests | Service | The number of HTTP PUT requests made for objects in an Amazon S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) | +| Delete Requests | bytes | aws_s3_delete_requests | Service | The number of HTTP Delete requests made for objects in an Amazon S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) | + +### Customizations +You can customize your own metrics/expression/dashboard panel. +The metrics definition and expression rules are found in `/config/otel-rules/aws-s3/`. +The AWS Cloud EKS dashboard panel configurations are found in `/config/ui-initialized-templates/aws_s3`. diff --git a/docs/menu.yml b/docs/menu.yml index 1a74b095b762..0882f8bab94d 100644 --- a/docs/menu.yml +++ b/docs/menu.yml @@ -189,6 +189,8 @@ catalog: catalog: - name: "EKS Monitoring" path: "/en/setup/backend/backend-aws-eks-monitoring" + - name: "S3 Monitoring" + path: "/en/setup/backend/backend-aws-s3-monitoring" - name: "Browser Monitoring" path: "/en/setup/service-agent/browser-agent" - name: "Gateway Monitoring" diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java index dd4da5ad0592..a939c5765341 100644 --- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java @@ -146,8 +146,12 @@ public enum Layer { /** * Windows Machine */ - OS_WINDOWS(23, true); + OS_WINDOWS(23, true), + /** + * Amazon Simple Storage Service (Amazon S3) is an object storage service provided by AWS Cloud + */ + AWS_S3(24, true); private final int value; /** diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java index 49539ff305a3..4ec3ac58c7f7 100644 --- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java @@ -62,6 +62,7 @@ public class UITemplateInitializer { Layer.VIRTUAL_MQ.name(), Layer.AWS_EKS.name(), Layer.OS_WINDOWS.name(), + Layer.AWS_S3.name(), "custom" }; private final UITemplateManagementService uiTemplateManagementService; diff --git a/oap-server/server-starter/src/main/resources/application.yml b/oap-server/server-starter/src/main/resources/application.yml index a08843be1d9c..a42c55bd2645 100644 --- a/oap-server/server-starter/src/main/resources/application.yml +++ b/oap-server/server-starter/src/main/resources/application.yml @@ -356,7 +356,7 @@ receiver-otel: selector: ${SW_OTEL_RECEIVER:default} default: enabledHandlers: ${SW_OTEL_RECEIVER_ENABLED_HANDLERS:"oc,otlp"} - enabledOtelRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_RULES:"apisix,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows"} + enabledOtelRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_RULES:"apisix,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*"} receiver-zipkin: selector: ${SW_RECEIVER_ZIPKIN:-} diff --git a/oap-server/server-starter/src/main/resources/otel-rules/aws-s3/s3-service.yaml b/oap-server/server-starter/src/main/resources/otel-rules/aws-s3/s3-service.yaml new file mode 100644 index 000000000000..6cfbf1718c7f --- /dev/null +++ b/oap-server/server-starter/src/main/resources/otel-rules/aws-s3/s3-service.yaml @@ -0,0 +1,55 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This will parse a textual representation of a duration. The formats +# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS} +# with days considered to be exactly 24 hours. +#

+# Examples: +# 

+#    "PT20.345S" -- parses as "20.345 seconds"
+#    "PT15M"     -- parses as "15 minutes" (where a minute is 60 seconds)
+#    "PT10H"     -- parses as "10 hours" (where an hour is 3600 seconds)
+#    "P2D"       -- parses as "2 days" (where a day is 24 hours or 86400 seconds)
+#    "P2DT3H4M"  -- parses as "2 days, 3 hours and 4 minutes"
+#    "P-6H3M"    -- parses as "-6 hours and +3 minutes"
+#    "-P6H3M"    -- parses as "-6 hours and -3 minutes"
+#    "-P-6H+3M"  -- parses as "+6 hours and -3 minutes"
+#
+ +filter: "{ tags -> {tags.cloud_provider == 'aws' && tags.Namespace == 'AWS/S3' } }" +expSuffix: tag({tags -> tags.bucket = 'aws-s3::' + tags.BucketName}).service(['bucket'], Layer.AWS_S3) +metricPrefix: aws_s3 +metricsRules: + - name: 4xx + exp: amazonaws_com_AWS_S3_4xxErrors_sum.downsampling(SUM) + - name: 5xx + exp: amazonaws_com_AWS_S3_5xxErrors_sum.downsampling(SUM) + - name: downloaded_bytes + exp: amazonaws_com_AWS_S3_BytesDownloaded_sum.downsampling(SUM) + - name: uploaded_bytes + exp: amazonaws_com_AWS_S3_BytesUploaded_sum.downsampling(SUM) + - name: request_latency + exp: amazonaws_com_AWS_S3_TotalRequestLatency_sum.div(amazonaws_com_AWS_S3_TotalRequestLatency_count) + - name: first_latency_bytes + exp: amazonaws_com_AWS_S3_FirstByteLatency_sum.div(amazonaws_com_AWS_S3_FirstByteLatency_count) + - name: all_requests + exp: amazonaws_com_AWS_S3_AllRequests_sum.downsampling(SUM) + - name: put_requests + exp: amazonaws_com_AWS_S3_PutRequests_sum.downsampling(SUM) + - name: get_requests + exp: amazonaws_com_AWS_S3_GetRequests_sum.downsampling(SUM) + - name: delete_requests + exp: amazonaws_com_AWS_S3_DeleteRequests_sum.downsampling(SUM) diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_eks/aws_eks-root.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_eks/aws_eks-root.json index 401538bbc772..d5e8206817eb 100644 --- a/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_eks/aws_eks-root.json +++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_eks/aws_eks-root.json @@ -50,7 +50,7 @@ "graph": { "fontColor":"blue", "backgroundColor":"white", - "content":"Provide AWS Could monitoring through AWS Container Insights Receiver", + "content":"Provide AWS Cloud EKS monitoring through AWS Container Insights Receiver", "fontSize":14, "textAlign":"left", "url":"https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-aws-eks-monitoring/" diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_s3/aws_s3-root.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_s3/aws_s3-root.json new file mode 100644 index 000000000000..459071991fe2 --- /dev/null +++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_s3/aws_s3-root.json @@ -0,0 +1,68 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +[ + { + "id": "AWS_S3-Root", + "configuration": { + "children": [ + { + "x": 0, + "y": 2, + "w": 24, + "h": 51, + "i": "0", + "type": "Widget", + "widget": { + "title": "" + }, + "graph": { + "type": "ServiceList", + "dashboardName": "AWS_S3-service", + "fontSize": 12, + "showXAxis": false, + "showYAxis": false, + "showGroup": true + }, + "moved": false + }, + { + "x": 0, + "y": 0, + "w": 24, + "h": 2, + "i": "100", + "type": "Text", + "graph": { + "fontColor":"blue", + "backgroundColor":"white", + "content":"Provide AWS Cloud S3 monitoring through AWS FireHose Receiver", + "fontSize":14, + "textAlign":"left", + "url":"https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-aws-s3-monitoring/" + }, + "moved": false + } + ], + "id": "AWS_S3-Root", + "layer": "AWS_S3", + "entity": "All", + "name": "AWS_S3-Root", + "isRoot": true + } + } +] diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_s3/aws_s3-service.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_s3/aws_s3-service.json new file mode 100644 index 000000000000..6c4a39279946 --- /dev/null +++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_s3/aws_s3-service.json @@ -0,0 +1,399 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +[ + { + "id": "AWS_S3-service", + "configuration": { + "children": [ + { + "x": 0, + "y": 0, + "w": 6, + "h": 5, + "i": "1", + "type": "Widget", + "widget": { + "title": "4xx Count" + }, + "graph": { + "type": "Card", + "fontSize": 14, + "textAlign": "center", + "showUnit": true + }, + "metrics": [ + "aws_s3_4xx" + ], + "metricTypes": [ + "readMetricsValue" + ] + }, + { + "x": 6, + "y": 0, + "w": 6, + "h": 5, + "i": "2", + "type": "Widget", + "widget": { + "title": "5xx Count" + }, + "graph": { + "type": "Card", + "fontSize": 14, + "textAlign": "center", + "showUnit": true + }, + "metrics": [ + "aws_s3_4xx" + ], + "metricTypes": [ + "readMetricsValue" + ] + }, + { + "x": 12, + "y": 0, + "w": 6, + "h": 5, + "i": "3", + "type": "Widget", + "widget": { + "title": "All Request Count" + }, + "graph": { + "type": "Card", + "fontSize": 14, + "textAlign": "center", + "showUnit": true + }, + "metrics": [ + "aws_s3_all_requests" + ], + "metricTypes": [ + "readMetricsValue" + ] + }, + { + "x": 18, + "y": 0, + "w": 6, + "h": 5, + "i": "4", + "type": "Widget", + "widget": { + "title": "Request Average Latency (ms)" + }, + "graph": { + "type": "Card", + "fontSize": 14, + "textAlign": "center", + "showUnit": true + }, + "metrics": [ + "aws_s3_request_latency" + ], + "metricTypes": [ + "readMetricsValue" + ] + }, + { + "x": 0, + "y": 5, + "w": 6, + "h": 13, + "i": "5", + "type": "Widget", + "widget": { + "title": "4xx Count" + }, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": false, + "showXAxis": true, + "showYAxis": true + }, + "metrics": [ + "aws_s3_4xx" + ], + "metricTypes": [ + "readMetricsValues" + ] + }, + { + "x": 6, + "y": 5, + "w": 6, + "h": 13, + "i": "6", + "type": "Widget", + "widget": { + "title": "5xx Count" + }, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": false, + "showXAxis": true, + "showYAxis": true + }, + "metrics": [ + "aws_s3_4xx" + ], + "metricTypes": [ + "readMetricsValues" + ] + }, + { + "x": 12, + "y": 5, + "w": 6, + "h": 13, + "i": "7", + "type": "Widget", + "widget": { + "title": "Request Average Latency (ms)" + }, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": false, + "showXAxis": true, + "showYAxis": true + }, + "metrics": [ + "aws_s3_request_latency" + ], + "metricTypes": [ + "readMetricsValues" + ] + }, + { + "x": 18, + "y": 5, + "w": 6, + "h": 13, + "i": "8", + "type": "Widget", + "widget": { + "title": "First Byte Average Latency (ms)" + }, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": false, + "showXAxis": true, + "showYAxis": true + }, + "metrics": [ + "aws_s3_first_latency_bytes" + ], + "metricTypes": [ + "readMetricsValues" + ] + }, + { + "x": 0, + "y": 18, + "w": 6, + "h": 13, + "i": "9", + "type": "Widget", + "widget": { + "title": "All Request Count" + }, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": false, + "showXAxis": true, + "showYAxis": true + }, + "metrics": [ + "aws_s3_all_requests" + ], + "metricTypes": [ + "readMetricsValues" + ] + }, + { + "x": 6, + "y": 18, + "w": 6, + "h": 13, + "i": "10", + "type": "Widget", + "widget": { + "title": "Get Request Count" + }, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": false, + "showXAxis": true, + "showYAxis": true + }, + "metrics": [ + "aws_s3_get_requests" + ], + "metricTypes": [ + "readMetricsValues" + ] + }, + { + "x": 12, + "y": 18, + "w": 6, + "h": 13, + "i": "11", + "type": "Widget", + "widget": { + "title": "Put Request Count" + }, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": false, + "showXAxis": true, + "showYAxis": true + }, + "metrics": [ + "aws_s3_put_requests" + ], + "metricTypes": [ + "readMetricsValues" + ] + }, + { + "x": 18, + "y": 18, + "w": 6, + "h": 13, + "i": "12", + "type": "Widget", + "widget": { + "title": "Delete Request Count" + }, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": false, + "showXAxis": true, + "showYAxis": true + }, + "metrics": [ + "aws_s3_delete_requests" + ], + "metricTypes": [ + "readMetricsValues" + ] + }, + { + "x": 0, + "y": 41, + "w": 12, + "h": 13, + "i": "13", + "type": "Widget", + "widget": { + "title": "Downloaded (KB)" + }, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": false, + "showXAxis": true, + "showYAxis": true, + "legend": { + "asTable": true, + "toTheRight": true, + "min": true, + "max": true, + "mean": true, + "total": true + } + }, + "metrics": [ + "aws_s3_downloaded_bytes" + ], + "metricTypes": [ + "readMetricsValues" + ], + "metricConfig": [ + { + "calculation": "byteToKB" + } + ] + }, + { + "x": 12, + "y": 41, + "w": 12, + "h": 13, + "i": "14", + "type": "Widget", + "widget": { + "title": "Uploaded (KB)" + }, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": false, + "showXAxis": true, + "showYAxis": true, + "legend": { + "asTable": true, + "toTheRight": true, + "min": true, + "max": true, + "mean": true, + "total": true + } + }, + "metrics": [ + "aws_s3_uploaded_bytes" + ], + "metricTypes": [ + "readMetricsValues" + ], + "metricConfig": [ + { + "calculation": "byteToKB" + } + ] + } + ], + "layer": "AWS_S3", + "entity": "Service", + "name": "AWS_S3-service", + "id": "AWS_S3-service", + "isRoot": false + } + } +] diff --git a/test/e2e-v2/cases/aws/s3/docker-compose.yml b/test/e2e-v2/cases/aws/s3/docker-compose.yml new file mode 100644 index 000000000000..dcd2cfc5af6f --- /dev/null +++ b/test/e2e-v2/cases/aws/s3/docker-compose.yml @@ -0,0 +1,51 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: '2.1' + +services: + oap: + extends: + file: ../../../script/docker-compose/base-compose.yml + service: oap + ports: + - 12800 + networks: + e2e: + + sender: + image: "eclipse-temurin:8-jre" + volumes: + - ./../../../java-test-service/e2e-mock-sender/target/e2e-mock-sender-2.0.0.jar:/e2e-mock-sender-2.0.0.jar + - ./mock-data:/data/otel-metrics + command: [ "java", "-jar", "/e2e-mock-sender-2.0.0.jar" ] + environment: + OAP_HOST: oap + OAP_GRPC_PORT: 11800 + OTEL_METRICS_DATA_PATH: /data/otel-metrics + networks: + - e2e + ports: + - 9093:9093 + healthcheck: + test: ["CMD", "sh", "-c", "nc -nz 127.0.0.1 9093"] + interval: 5s + timeout: 60s + retries: 120 + depends_on: + oap: + condition: service_healthy +networks: + e2e: diff --git a/test/e2e-v2/cases/aws/s3/e2e.yaml b/test/e2e-v2/cases/aws/s3/e2e.yaml new file mode 100644 index 000000000000..5cde5b61d16b --- /dev/null +++ b/test/e2e-v2/cases/aws/s3/e2e.yaml @@ -0,0 +1,44 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This file is used to show how to write configuration files and can be used to test. + +setup: + env: compose + file: docker-compose.yml + timeout: 20m + init-system-environment: ../../../script/env + steps: + - name: set PATH + command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH + - name: install yq + command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq + - name: install swctl + command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl + +trigger: + action: http + interval: 3s + times: 10 + url: http://localhost:9093/otel-metrics/send + method: GET + +verify: + retry: + count: 10 + interval: 3s + cases: + - includes: + - ./s3-cases.yaml diff --git a/test/e2e-v2/cases/aws/s3/expected/metrics-has-value.yml b/test/e2e-v2/cases/aws/s3/expected/metrics-has-value.yml new file mode 100644 index 000000000000..ce1f3d83c82f --- /dev/null +++ b/test/e2e-v2/cases/aws/s3/expected/metrics-has-value.yml @@ -0,0 +1,19 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{{- contains . }} +- key: {{ notEmpty .key }} + value: {{ ge .value 0 }} +{{- end }} diff --git a/test/e2e-v2/cases/aws/s3/expected/service.yml b/test/e2e-v2/cases/aws/s3/expected/service.yml new file mode 100644 index 000000000000..f6021171009b --- /dev/null +++ b/test/e2e-v2/cases/aws/s3/expected/service.yml @@ -0,0 +1,24 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + {{- contains . }} +- id: {{ b64enc "aws-s3::skywalking" }}.1 + name: aws-s3::skywalking + group: aws-s3 + shortname: skywalking + normal: true + layers: + - AWS_S3 + {{- end }} diff --git a/test/e2e-v2/cases/aws/s3/mock-data/data-1.json b/test/e2e-v2/cases/aws/s3/mock-data/data-1.json new file mode 100644 index 000000000000..7d97bca7bf96 --- /dev/null +++ b/test/e2e-v2/cases/aws/s3/mock-data/data-1.json @@ -0,0 +1,132 @@ +{ + "resourceMetrics": [ + { + "resource": { + "attributes": [ + { + "key": "cloud.provider", + "value": { + "stringValue": "aws" + } + }, + { + "key": "cloud.account.id", + "value": { + "stringValue": "xxxxxxxx" + } + }, + { + "key": "cloud.region", + "value": { + "stringValue": "ap-northeast-1" + } + }, + { + "key": "aws.exporter.arn", + "value": { + "stringValue": "arn:aws:cloudwatch:ap-northeast-1:xxxxxxxx:metric-stream/CustomFull-CmHV2P" + } + } + ] + }, + "scopeMetrics": [ + { + "metrics": [ + { + "name": "amazonaws.com/AWS/S3/4xxErrors", + "unit": "{Count}", + "summary": { + "dataPoints": [ + { + "startTimeUnixNano": "1674547500000000000", + "timeUnixNano": "1674547560000000000", + "count": "1", + "quantileValues": [ + { + }, + { + "quantile": 1.0 + } + ], + "attributes": [ + { + "key": "Namespace", + "value": { + "stringValue": "AWS/S3" + } + }, + { + "key": "MetricName", + "value": { + "stringValue": "4xxErrors" + } + }, + { + "key": "BucketName", + "value": { + "stringValue": "skywalking" + } + }, + { + "key": "FilterId", + "value": { + "stringValue": "test" + } + } + ] + } + ] + } + }, + { + "name": "amazonaws.com/AWS/S3/5xxErrors", + "unit": "{Count}", + "summary": { + "dataPoints": [ + { + "startTimeUnixNano": "1674547500000000000", + "timeUnixNano": "1674547560000000000", + "count": "1", + "quantileValues": [ + { + }, + { + "quantile": 1.0 + } + ], + "attributes": [ + { + "key": "Namespace", + "value": { + "stringValue": "AWS/S3" + } + }, + { + "key": "MetricName", + "value": { + "stringValue": "5xxErrors" + } + }, + { + "key": "BucketName", + "value": { + "stringValue": "skywalking" + } + }, + { + "key": "FilterId", + "value": { + "stringValue": "test" + } + } + ] + } + ] + } + } + ] + } + ] + } + ] +} diff --git a/test/e2e-v2/cases/aws/s3/s3-cases.yaml b/test/e2e-v2/cases/aws/s3/s3-cases.yaml new file mode 100644 index 000000000000..1c6ebb8c87a2 --- /dev/null +++ b/test/e2e-v2/cases/aws/s3/s3-cases.yaml @@ -0,0 +1,28 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This file is used to show how to write configuration files and can be used to test. + + cases: + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql service ls + expected: expected/service.yml + - query: | + swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \ + --name=aws_s3_4xx --service-name aws-s3::skywalking |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: | + swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \ + --name=aws_s3_5xx --service-name aws-s3::skywalking |yq e 'to_entries' - + expected: expected/metrics-has-value.yml