diff --git a/.github/workflows/skywalking.yaml b/.github/workflows/skywalking.yaml index e74bbae57a40..62b1a07c44b4 100644 --- a/.github/workflows/skywalking.yaml +++ b/.github/workflows/skywalking.yaml @@ -672,6 +672,8 @@ jobs: config: test/e2e-v2/cases/win/e2e.yaml - name: AWS Cloud S3 config: test/e2e-v2/cases/aws/s3/e2e.yaml + - name: AWS Cloud DynamoDB + config: test/e2e-v2/cases/aws/dynamodb/e2e.yaml steps: - uses: actions/checkout@v3 with: diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index 58d774a72c64..34d5b4ffde18 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -99,6 +99,7 @@ * Refactor http-based alarm plugins and extract common logic to `HttpAlarmCallback`. * Support Amazon Simple Storage Service (Amazon S3) metrics monitoring * Support process Sum metrics with AGGREGATION_TEMPORALITY_DELTA case +* Support Amazon DynamoDB monitoring. #### UI diff --git a/docs/en/setup/backend/aws-firehose-receiver.md b/docs/en/setup/backend/aws-firehose-receiver.md index 57260261261d..6f3b783d81a5 100644 --- a/docs/en/setup/backend/aws-firehose-receiver.md +++ b/docs/en/setup/backend/aws-firehose-receiver.md @@ -19,6 +19,8 @@ CloudWatch metrics with S3 --> CloudWatch Metric Stream (OpenTelemetry formart) | Description | Configuration File | Data Source | |-----------------------------------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------| | Metrics of AWS Cloud S3 | otel-rules/aws-s3/s3-service.yaml | AWS CloudWatcher Metrics Stream -> AWS Firehose delivery stream -> SkyWalking OAP Server with [AWS Firehose receiver](./aws-firehose-receiver.md) | +| Metrics of AWS DynamoDB | otel-rules/aws-dynamodb/dynamodb-service.yaml | AWS CloudWatcher Metrics Stream -> AWS Firehose delivery stream -> SkyWalking OAP Server with [AWS Firehose receiver](./aws-firehose-receiver.md) | +| Metrics of AWS DynamoDB | otel-rules/aws-dynamodb/dynamodb-endpoint.yaml | AWS CloudWatcher Metrics Stream -> AWS Firehose delivery stream -> SkyWalking OAP Server with [AWS Firehose receiver](./aws-firehose-receiver.md) | ## Notice diff --git a/docs/en/setup/backend/backend-aws-dynamodb-monitoring.md b/docs/en/setup/backend/backend-aws-dynamodb-monitoring.md new file mode 100644 index 000000000000..15bce3fde7e1 --- /dev/null +++ b/docs/en/setup/backend/backend-aws-dynamodb-monitoring.md @@ -0,0 +1,36 @@ +# AWS DynamoDb monitoring +SkyWalking leverages Amazon Kinesis Data Filehose with [Amazon CloudWatch](https://aws.amazon.com/cn/cloudwatch/) to transfer the metrics into the [Meter System](./../../concepts-and-designs/meter.md). + +### Data flow +1. Amazon CloudWatch fetches metrics from DynamoDB and pushes metrics to SkyWalking OAP Server via Amazon Kinesis data firehose. +2. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results. + +### Set up +1. Create CloudWatch metrics configuration for DynamoDB, refer to [DynamoDB metrics configuration](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/monitoring-cloudwatch.html) + Create an Amazon Kinesis Data Firehose Delivery Stream, and set [AWS Kinesis Data Firehose receiver](./aws-firehose-receiver.md)'s address as HTTP(s) Destination, refer to [Create Delivery Stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html)3. Create a [metric stream](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html), set namespace to DynanoDB, and set `Kinesis Data Firehose` to the firehose you just created. +2. Config [aws-firehose-receiver](aws-firehose-receiver.md) to receive data. +3. Create CloudWatch metric stream, and select the Firehose Delivery Stream which has been created above, set `Select namespaces` to `AWS/DynamoDB`, `Select output format` to `OpenTelemetry 0.7`. refer to [CloudWatch Metric Streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html) + +### DynamoDB Monitoring +DynamoDB monitoring provides monitoring of the status and resources of the DynamoDB server. AWS user id is cataloged as a `Layer: AWS_DYNAMODB` `Service` in OAP. +Each DynamoDB table is cataloged as an `Endpoint` in OAP. + +#### Supported Metrics +| Monitoring Panel | Unit | Metric Name | Description | Data Source | +|-----|-----|-----|-----|--| +| Read Usage | unit/s | consumed_read_capacity_units
provisioned_read_capacity_units| The situation of read capacity units consumed and provisioned over the specified time period | Amazon CloudWatch | +| Write Usage | unit/s | consumed_write_capacity_units
provisioned_write_capacity_units| The situation of write capacity units consumed and provisioned over the specified time period | Amazon CloudWatch | +| Successful Request Latency | ms | get_successful_request_latency
put_successful_request_latency
query_successful_request_latency
scan_successful_request_latency | The latency of successful request | Amazon CloudWatch | +| TTL Deleted Item count | | time_to_live_deleted_item_count | The count of items deleted by TTL | Amazon CloudWatch | +| Throttle Events| | read_throttle_events
write_throttle_events | Requests to DynamoDB that exceed the provisioned read/write capacity units for a table or a global secondary index. | Amazon CloudWatch | +| Throttled Requests | | read_throttled_requests
write_throttled_requests | Requests to DynamoDB that exceed the provisioned throughput limits on a resource (such as a table or an index). | Amazon CloudWatch | +| Scan/Query Operation Returned Item Ccount | | scan_returned_item_count
query_returned_item_count
| The number of items returned by Query, Scan or ExecuteStatement (select) operations during the specified time period. | Amazon CloudWatch | +| System Errors | | read_system_errors
write_system_errors | The requests to DynamoDB or Amazon DynamoDB Streams that generate an HTTP 500 status code during the specified time period. | Amazon CloudWatch | +| User Errors | | user_errors | Requests to DynamoDB or Amazon DynamoDB Streams that generate an HTTP 400 status code during the specified time period.| Amazon CloudWatch | +| Condition Checked Fail Requests | | conditional_check_failed_requests | The number of failed attempts to perform conditional writes. | Amazon CloudWatch | +| Transaction Conflict | | transaction_conflict | Rejected item-level requests due to transactional conflicts between concurrent requests on the same items. | Amazon CloudWatch | + +### Customizations +You can customize your own metrics/expression/dashboard panel. +The metrics definition and expression rules are found in `/config/otel-rules/aws-dynamodb`. +The DynamoDB dashboard panel configurations are found in `/config/ui-initialized-templates/aws_dynamodb`. \ No newline at end of file diff --git a/docs/menu.yml b/docs/menu.yml index 0882f8bab94d..c51779b0951f 100644 --- a/docs/menu.yml +++ b/docs/menu.yml @@ -191,6 +191,8 @@ catalog: path: "/en/setup/backend/backend-aws-eks-monitoring" - name: "S3 Monitoring" path: "/en/setup/backend/backend-aws-s3-monitoring" + - name: "AWS DynamoDB" + path: "/en/setup/backend/backend-aws-dynamodb-monitoring" - name: "Browser Monitoring" path: "/en/setup/service-agent/browser-agent" - name: "Gateway Monitoring" @@ -203,6 +205,8 @@ catalog: path: "/en/setup/backend/backend-mysql-monitoring" - name: "PostgreSQL Server" path: "/en/setup/backend/backend-postgresql-monitoring" + - name: "AWS DynamoDB" + path: "/en/setup/backend/backend-aws-dynamodb-monitoring" - name: "Self Observability" catalog: - name: "OAP self telemetry" diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java index a939c5765341..2b0e7ce0df6c 100644 --- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java @@ -151,7 +151,13 @@ public enum Layer { /** * Amazon Simple Storage Service (Amazon S3) is an object storage service provided by AWS Cloud */ - AWS_S3(24, true); + AWS_S3(24, true), + + /* + * Amazon DynamoDB is a fully managed NoSQL database service that provides + * fast and predictable performance with seamless scalability. + */ + AWS_DYNAMODB(25, true); private final int value; /** diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java index 4ec3ac58c7f7..243e25e001e4 100644 --- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java @@ -63,6 +63,7 @@ public class UITemplateInitializer { Layer.AWS_EKS.name(), Layer.OS_WINDOWS.name(), Layer.AWS_S3.name(), + Layer.AWS_DYNAMODB.name(), "custom" }; private final UITemplateManagementService uiTemplateManagementService; diff --git a/oap-server/server-starter/src/main/resources/application.yml b/oap-server/server-starter/src/main/resources/application.yml index de052da12c29..35163a15f1d0 100644 --- a/oap-server/server-starter/src/main/resources/application.yml +++ b/oap-server/server-starter/src/main/resources/application.yml @@ -356,7 +356,7 @@ receiver-otel: selector: ${SW_OTEL_RECEIVER:default} default: enabledHandlers: ${SW_OTEL_RECEIVER_ENABLED_HANDLERS:"oc,otlp"} - enabledOtelRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_RULES:"apisix,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*"} + enabledOtelRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_RULES:"apisix,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*,aws-dynamodb/*"} receiver-zipkin: selector: ${SW_RECEIVER_ZIPKIN:-} diff --git a/oap-server/server-starter/src/main/resources/otel-rules/aws-dynamodb/dynamodb-endpoint.yaml b/oap-server/server-starter/src/main/resources/otel-rules/aws-dynamodb/dynamodb-endpoint.yaml new file mode 100644 index 000000000000..70ce6b4c7fa3 --- /dev/null +++ b/oap-server/server-starter/src/main/resources/otel-rules/aws-dynamodb/dynamodb-endpoint.yaml @@ -0,0 +1,86 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This will parse a textual representation of a duration. The formats +# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS} +# with days considered to be exactly 24 hours. +#

+# Examples: +# 

+#    "PT20.345S" -- parses as "20.345 seconds"
+#    "PT15M"     -- parses as "15 minutes" (where a minute is 60 seconds)
+#    "PT10H"     -- parses as "10 hours" (where an hour is 3600 seconds)
+#    "P2D"       -- parses as "2 days" (where a day is 24 hours or 86400 seconds)
+#    "P2DT3H4M"  -- parses as "2 days, 3 hours and 4 minutes"
+#    "P-6H3M"    -- parses as "-6 hours and +3 minutes"
+#    "-P6H3M"    -- parses as "-6 hours and -3 minutes"
+#    "-P-6H+3M"  -- parses as "+6 hours and -3 minutes"
+#
+ +filter: "{ tags -> tags.Namespace == 'AWS/DynamoDB' }" # The OpenTelemetry job name +expPrefix: tag({tags -> tags.host_name = 'aws-dynamodb::' + tags.cloud_account_id}) +expSuffix: service(['host_name'], Layer.AWS_DYNAMODB).endpoint(['host_name'], ['TableName'], Layer.AWS_DYNAMODB) +metricPrefix: aws_dynamodb +metricsRules: + # table metrics + - name: endpoint_consumed_write_capacity_units + exp: amazonaws_com_AWS_DynamoDB_ConsumedWriteCapacityUnits + + - name: endpoint_consumed_read_capacity_units + exp: amazonaws_com_AWS_DynamoDB_ConsumedReadCapacityUnits + + - name: endpoint_provisioned_read_capacity_units + exp: amazonaws_com_AWS_DynamoDB_ProvisionedReadCapacityUnits + + - name: endpoint_provisioned_write_capacity_units + exp: amazonaws_com_AWS_DynamoDB_ProvisionedWriteCapacityUnits + + # table operation metrics + - name: endpoint_get_successful_request_latency + exp: amazonaws_com_AWS_DynamoDB_SuccessfulRequestLatency.tagMatch('Operation','GetItem|BatchGetItem') + - name: endpoint_put_successful_request_latency + exp: amazonaws_com_AWS_DynamoDB_SuccessfulRequestLatency.tagMatch('Operation','PutItem|BatchWriteItem') + - name: endpoint_query_successful_request_latency + exp: amazonaws_com_AWS_DynamoDB_SuccessfulRequestLatency.tagEqual('Operation','Query') + - name: endpoint_scan_successful_request_latency + exp: amazonaws_com_AWS_DynamoDB_SuccessfulRequestLatency.tagEqual('Operation','Scan') + + - name: endpoint_scan_returned_item_count + exp: amazonaws_com_AWS_DynamoDB_ReturnedItemCount.tagEqual('Operation','Scan') + - name: endpoint_query_returned_item_count + exp: amazonaws_com_AWS_DynamoDB_ReturnedItemCount.tagEqual('Operation','Query') + + - name: endpoint_time_to_live_deleted_item_count + exp: amazonaws_com_AWS_DynamoDB_TimeToLiveDeletedItemCount + + - name: endpoint_read_throttled_requests + exp: amazonaws_com_AWS_DynamoDB_ThrottledRequests.tagMatch('Operation','GetItem|Scan|Query|BatchGetItem') + - name: endpoint_write_throttled_requests + exp: amazonaws_com_AWS_DynamoDB_ThrottledRequests.tagMatch('Operation','PutItem|UpdateItem|DeleteItem|BatchWriteItem') + + - name: endpoint_read_throttle_events + exp: amazonaws_com_AWS_DynamoDB_ReadThrottleEvents + - name: endpoint_write_throttle_events + exp: amazonaws_com_AWS_DynamoDB_WriteThrottleEvents + - name: endpoint_read_system_errors + exp: amazonaws_com_AWS_DynamoDB_SystemErrors.tagMatch('Operation','GetItem|Scan|Query|BatchGetItem|TransactGetItems') + - name: endpoint_write_system_errors + exp: amazonaws_com_AWS_DynamoDB_SystemErrors.tagMatch('Operation','PutItem|UpdateItem|DeleteItem|BatchWriteItem|TransactWriteItems') + - name: endpoint_user_errors + exp: amazonaws_com_AWS_DynamoDB_UserErrors + - name: endpoint_conditional_check_failed_requests + exp: amazonaws_com_AWS_DynamoDB_ConditionalCheckFailedRequests + - name: endpoint_transaction_conflict + exp: amazonaws_com_AWS_DynamoDB_TransactionConflict \ No newline at end of file diff --git a/oap-server/server-starter/src/main/resources/otel-rules/aws-dynamodb/dynamodb-service.yaml b/oap-server/server-starter/src/main/resources/otel-rules/aws-dynamodb/dynamodb-service.yaml new file mode 100644 index 000000000000..7a66838fa898 --- /dev/null +++ b/oap-server/server-starter/src/main/resources/otel-rules/aws-dynamodb/dynamodb-service.yaml @@ -0,0 +1,108 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This will parse a textual representation of a duration. The formats +# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS} +# with days considered to be exactly 24 hours. +#

+# Examples: +# 

+#    "PT20.345S" -- parses as "20.345 seconds"
+#    "PT15M"     -- parses as "15 minutes" (where a minute is 60 seconds)
+#    "PT10H"     -- parses as "10 hours" (where an hour is 3600 seconds)
+#    "P2D"       -- parses as "2 days" (where a day is 24 hours or 86400 seconds)
+#    "P2DT3H4M"  -- parses as "2 days, 3 hours and 4 minutes"
+#    "P-6H3M"    -- parses as "-6 hours and +3 minutes"
+#    "-P6H3M"    -- parses as "-6 hours and -3 minutes"
+#    "-P-6H+3M"  -- parses as "+6 hours and -3 minutes"
+#
+ +filter: "{ tags -> tags.Namespace == 'AWS/DynamoDB' }" # The OpenTelemetry job name +expPrefix: tag({tags -> tags.host_name = 'aws-dynamodb::' + tags.cloud_account_id}) +expSuffix: service(['host_name'], Layer.AWS_DYNAMODB) +metricPrefix: aws_dynamodb +metricsRules: + # account metrics + - name: account_max_writes + exp: amazonaws_com_AWS_DynamoDB_AccountMaxWrites + - name: account_max_reads + exp: amazonaws_com_AWS_DynamoDB_AccountMaxReads + + - name: account_max_table_level_writes + exp: amazonaws_com_AWS_DynamoDB_AccountMaxTableLevelWrites + - name: account_max_table_level_reads + exp: amazonaws_com_AWS_DynamoDB_AccountMaxTableLevelReads + + - name: max_provisioned_write_capacity_utilization + exp: amazonaws_com_AWS_DynamoDB_MaxProvisionedTableWriteCapacityUtilization + - name: max_provisioned_read_capacity_utilization + exp: amazonaws_com_AWS_DynamoDB_MaxProvisionedTableReadCapacityUtilization + + - name: account_provisioned_read_capacity_utilization + exp: amazonaws_com_AWS_DynamoDB_AccountProvisionedReadCapacityUtilization + - name: account_provisioned_write_capacity_utilization + exp: amazonaws_com_AWS_DynamoDB_AccountProvisionedWriteCapacityUtilization + + # table metrics + - name: consumed_write_capacity_units + exp: amazonaws_com_AWS_DynamoDB_ConsumedWriteCapacityUnits + + - name: consumed_read_capacity_units + exp: amazonaws_com_AWS_DynamoDB_ConsumedReadCapacityUnits + + - name: provisioned_read_capacity_units + exp: amazonaws_com_AWS_DynamoDB_ProvisionedReadCapacityUnits + + - name: provisioned_write_capacity_units + exp: amazonaws_com_AWS_DynamoDB_ProvisionedWriteCapacityUnits + + # table operation metrics + - name: get_successful_request_latency + exp: amazonaws_com_AWS_DynamoDB_SuccessfulRequestLatency.tagMatch('Operation','GetItem|BatchGetItem') + - name: put_successful_request_latency + exp: amazonaws_com_AWS_DynamoDB_SuccessfulRequestLatency.tagMatch('Operation','PutItem|BatchWriteItem') + - name: query_successful_request_latency + exp: amazonaws_com_AWS_DynamoDB_SuccessfulRequestLatency.tagEqual('Operation','Query') + - name: scan_successful_request_latency + exp: amazonaws_com_AWS_DynamoDB_SuccessfulRequestLatency.tagEqual('Operation','Scan') + - name: scan_returned_item_count + exp: amazonaws_com_AWS_DynamoDB_ReturnedItemCount.tagEqual('Operation','Scan') + - name: query_returned_item_count + exp: amazonaws_com_AWS_DynamoDB_ReturnedItemCount.tagEqual('Operation','Query') + + - name: time_to_live_deleted_item_count + exp: amazonaws_com_AWS_DynamoDB_TimeToLiveDeletedItemCount + + - name: read_throttled_requests + exp: amazonaws_com_AWS_DynamoDB_ThrottledRequests.tagMatch('Operation','GetItem|Scan|Query|BatchGetItem') + - name: write_throttled_requests + exp: amazonaws_com_AWS_DynamoDB_ThrottledRequests.tagMatch('Operation','PutItem|UpdateItem|DeleteItem|BatchWriteItem') + + - name: read_throttle_events + exp: amazonaws_com_AWS_DynamoDB_ReadThrottleEvents + - name: write_throttle_events + exp: amazonaws_com_AWS_DynamoDB_WriteThrottleEvents + - name: read_system_errors + exp: amazonaws_com_AWS_DynamoDB_SystemErrors.tagMatch('Operation','GetItem|Scan|Query|BatchGetItem|TransactGetItems') + - name: write_system_errors + exp: amazonaws_com_AWS_DynamoDB_SystemErrors.tagMatch('Operation','PutItem|UpdateItem|DeleteItem|BatchWriteItem|TransactWriteItems') + - name: user_errors + exp: amazonaws_com_AWS_DynamoDB_UserErrors + - name: conditional_check_failed_requests + exp: amazonaws_com_AWS_DynamoDB_ConditionalCheckFailedRequests + - name: transaction_conflict + exp: amazonaws_com_AWS_DynamoDB_TransactionConflict + + diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_dynamodb/aws_dynamodb-endpoint.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_dynamodb/aws_dynamodb-endpoint.json new file mode 100644 index 000000000000..00201d70878b --- /dev/null +++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_dynamodb/aws_dynamodb-endpoint.json @@ -0,0 +1,346 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +[ + { + "id":"AWS-DynamoDB-Endpoint", + "configuration":{ + "children":[ + { + "x":6, + "y":10, + "w":6, + "h":11, + "i":"0", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_endpoint_read_throttled_requests", + "aws_dynamodb_endpoint_write_throttled_requests" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Throttled Requests" + } + }, + { + "x":0, + "y":10, + "w":6, + "h":11, + "i":"1", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_endpoint_read_throttle_events", + "aws_dynamodb_endpoint_write_throttle_events" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Throttle Events" + } + }, + { + "x":0, + "y":21, + "w":6, + "h":11, + "i":"2", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_endpoint_read_system_errors", + "aws_dynamodb_endpoint_write_system_errors" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"System Errors" + } + }, + { + "x":6, + "y":21, + "w":6, + "h":11, + "i":"3", + "type":"Widget", + "metricTypes":[ + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_endpoint_user_errors" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"User Error" + } + }, + { + "x":12, + "y":21, + "w":6, + "h":11, + "i":"4", + "type":"Widget", + "metricTypes":[ + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_endpoint_conditional_check_failed_requests" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Conditional Check Failed Requests" + } + }, + { + "x":18, + "y":21, + "w":6, + "h":11, + "i":"5", + "type":"Widget", + "metricTypes":[ + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_endpoint_transaction_conflict" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Transaction Conflict" + } + }, + { + "x":0, + "y":0, + "w":6, + "h":10, + "i":"6", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_endpoint_provisioned_read_capacity_units", + "aws_dynamodb_endpoint_consumed_read_capacity_units" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Read Usage (unit/s)" + } + }, + { + "x":6, + "y":0, + "w":6, + "h":10, + "i":"7", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_endpoint_consumed_write_capacity_units", + "aws_dynamodb_endpoint_provisioned_write_capacity_units" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Write Usage (unit/s)" + } + }, + { + "x":12, + "y":0, + "w":6, + "h":10, + "i":"8", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues", + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_endpoint_get_successful_request_latency", + "aws_dynamodb_endpoint_put_successful_request_latency", + "aws_dynamodb_endpoint_query_successful_request_latency", + "aws_dynamodb_endpoint_scan_successful_request_latency" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Successful Request Latency (ms)" + } + }, + { + "x":18, + "y":0, + "w":6, + "h":10, + "i":"9", + "type":"Widget", + "metricTypes":[ + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_endpoint_time_to_live_deleted_item_count" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"TTL Deleted Item Count" + } + }, + { + "x":12, + "y":10, + "w":6, + "h":11, + "i":"11", + "type":"Widget", + "metricTypes":[ + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_endpoint_scan_returned_item_count" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Scan Operation Returned Item Count" + } + }, + { + "x":18, + "y":10, + "w":6, + "h":11, + "i":"12", + "type":"Widget", + "metricTypes":[ + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_endpoint_query_returned_item_count" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Query Operation Returned Item Count" + } + } + ], + "layer":"AWS_DYNAMODB", + "entity":"Endpoint", + "name":"AWS-DynamoDB-Endpoint", + "id":"AWS-DynamoDB-Endpoint" + } + } +] \ No newline at end of file diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_dynamodb/aws_dynamodb-root.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_dynamodb/aws_dynamodb-root.json new file mode 100644 index 000000000000..1c66ceb69f65 --- /dev/null +++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_dynamodb/aws_dynamodb-root.json @@ -0,0 +1,64 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +[ + { + "id":"AWS-DynamoDB-Root", + "configuration":{ + "children":[ + { + "x":0, + "y":0, + "w":24, + "h":3, + "i":"1", + "type":"Text", + "graph":{ + "fontColor":"blue", + "backgroundColor":"white", + "content":"Provide DynamoDB monitoring through Amazon CloudWatch", + "fontSize":14, + "textAlign":"left", + "url":"https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-aws-dynamodb-monitoring/" + } + }, + { + "x":0, + "y":3, + "w":24, + "h":43, + "i":"0", + "type":"Widget", + "graph":{ + "type":"ServiceList", + "dashboardName":"AWS-DynamoDB-Service", + "fontSize":12, + "showXAxis":false, + "showYAxis":false, + "showGroup":true + } + } + + ], + "layer":"AWS_DYNAMODB", + "entity":"All", + "name":"AWS-DynamoDB-Root", + "id":"AWS-DynamoDB-Root", + "isRoot":true + } + } +] \ No newline at end of file diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_dynamodb/aws_dynamodb-service.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_dynamodb/aws_dynamodb-service.json new file mode 100644 index 000000000000..89a575254b18 --- /dev/null +++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/aws_dynamodb/aws_dynamodb-service.json @@ -0,0 +1,462 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +[ + { + "id":"AWS-DynamoDB-Service", + "configuration":{ + "children":[ + { + "x":0, + "y":0, + "w":24, + "h":36, + "i":"2", + "type":"Tab", + "children":[ + { + "name":"Inspections", + "children":[ + { + "x":6, + "y":21, + "w":6, + "h":11, + "i":"0", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_read_throttled_requests", + "aws_dynamodb_write_throttled_requests" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Throttled Requests" + } + }, + { + "x":0, + "y":21, + "w":6, + "h":11, + "i":"1", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_read_throttle_events", + "aws_dynamodb_write_throttle_events" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Throttle Events" + } + }, + { + "x":0, + "y":32, + "w":6, + "h":11, + "i":"2", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_read_system_errors", + "aws_dynamodb_write_system_errors" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"System Errors" + } + }, + { + "x":6, + "y":32, + "w":6, + "h":11, + "i":"3", + "type":"Widget", + "metricTypes":[ + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_user_errors" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"User Error" + } + }, + { + "x":12, + "y":32, + "w":6, + "h":11, + "i":"4", + "type":"Widget", + "metricTypes":[ + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_conditional_check_failed_requests" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Conditional Check Failed Requests" + } + }, + { + "x":18, + "y":32, + "w":6, + "h":11, + "i":"5", + "type":"Widget", + "metricTypes":[ + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_transaction_conflict" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Transaction Conflict" + } + }, + { + "x":0, + "y":11, + "w":6, + "h":10, + "i":"6", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_provisioned_read_capacity_units", + "aws_dynamodb_consumed_read_capacity_units" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Read Usage (unit/s)" + } + }, + { + "x":6, + "y":11, + "w":6, + "h":10, + "i":"7", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_consumed_write_capacity_units", + "aws_dynamodb_provisioned_write_capacity_units" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Write Usage (unit/s)" + } + }, + { + "x":12, + "y":11, + "w":6, + "h":10, + "i":"8", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues", + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_get_successful_request_latency", + "aws_dynamodb_put_successful_request_latency", + "aws_dynamodb_query_successful_request_latency", + "aws_dynamodb_scan_successful_request_latency" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Successful Request Latency (ms)" + } + }, + { + "x":18, + "y":11, + "w":6, + "h":10, + "i":"9", + "type":"Widget", + "metricTypes":[ + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_time_to_live_deleted_item_count" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"TTL Deleted Item Count" + } + }, + { + "x":12, + "y":21, + "w":6, + "h":11, + "i":"11", + "type":"Widget", + "metricTypes":[ + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_scan_returned_item_count" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Scan Operation Returned Item Count" + } + }, + { + "x":18, + "y":21, + "w":6, + "h":11, + "i":"12", + "type":"Widget", + "metricTypes":[ + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_query_returned_item_count" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Query Operation Returned Item Count" + } + }, + { + "x":8, + "y":0, + "w":8, + "h":11, + "i":"13", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_account_max_table_level_writes", + "aws_dynamodb_account_max_table_level_reads" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Account Max Table Level Writes / Reads" + } + }, + { + "x":16, + "y":0, + "w":8, + "h":11, + "i":"14", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_account_provisioned_read_capacity_utilization", + "aws_dynamodb_account_provisioned_write_capacity_utilization" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Account Provisioned Read / Write Capacity Utilization" + } + }, + { + "x":0, + "y":0, + "w":8, + "h":11, + "i":"15", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "aws_dynamodb_account_max_writes", + "aws_dynamodb_account_max_reads" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Account Max Reads / Writes" + } + } + ] + }, + { + "name":"Tables", + "children":[ + { + "x":0, + "y":0, + "w":24, + "h":29, + "i":"0", + "type":"Widget", + "graph":{ + "type":"EndpointList", + "dashboardName":"AWS-DynamoDB-Endpoint", + "fontSize":12 + } + } + ] + } + ] + } + ], + "layer":"AWS_DYNAMODB", + "entity":"Service", + "name":"AWS-DynamoDB-Service", + "id":"AWS-DynamoDB-Service", + "isRoot":false + } + } +] \ No newline at end of file diff --git a/test/e2e-v2/cases/aws/dynamodb/docker-compose.yml b/test/e2e-v2/cases/aws/dynamodb/docker-compose.yml new file mode 100644 index 000000000000..39262bbea821 --- /dev/null +++ b/test/e2e-v2/cases/aws/dynamodb/docker-compose.yml @@ -0,0 +1,51 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: '2.1' + +services: + oap: + extends: + file: ../../../script/docker-compose/base-compose.yml + service: oap + ports: + - 12800 + networks: + e2e: + + sender: + image: "eclipse-temurin:8-jre" + volumes: + - ./../../../java-test-service/e2e-mock-sender/target/e2e-mock-sender-2.0.0.jar:/e2e-mock-sender-2.0.0.jar + - ./mock-data:/data/otel-metrics + command: [ "java", "-jar", "/e2e-mock-sender-2.0.0.jar" ] + environment: + OAP_HOST: oap + OAP_GRPC_PORT: 11800 + OTEL_METRICS_DATA_PATH: /data/otel-metrics + networks: + - e2e + ports: + - 9093:9093 + healthcheck: + test: ["CMD", "sh", "-c", "nc -nz 127.0.0.1 9093"] + interval: 5s + timeout: 60s + retries: 120 + depends_on: + oap: + condition: service_healthy +networks: + e2e: \ No newline at end of file diff --git a/test/e2e-v2/cases/aws/dynamodb/dynamodb-cases.yaml b/test/e2e-v2/cases/aws/dynamodb/dynamodb-cases.yaml new file mode 100644 index 000000000000..a67cf6a3b56a --- /dev/null +++ b/test/e2e-v2/cases/aws/dynamodb/dynamodb-cases.yaml @@ -0,0 +1,81 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This file is used to show how to write configuration files and can be used to test. + +cases: + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql service ls + expected: expected/service.yml + - query: | + swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \ + --name=account_max_writes --service-name aws-dynamodb::xxxxxxxx |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: | + swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \ + --name=account_max_reads --service-name aws-dynamodb::xxxxxxxx |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: | + swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \ + --name=max_provisioned_write_capacity_utilization --service-name aws-dynamodb::xxxxxxxx |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: | + swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \ + --name=max_provisioned_read_capacity_utilization --service-name aws-dynamodb::xxxxxxxx |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: | + swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \ + --name=account_max_table_level_reads --service-name aws-dynamodb::xxxxxxxx |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: | + swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \ + --name=account_max_table_level_writes --service-name aws-dynamodb::xxxxxxxx |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: | + swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \ + --name=account_provisioned_read_capacity_utilization --service-name aws-dynamodb::xxxxxxxx |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: | + swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \ + --name=account_provisioned_write_capacity_utilization --service-name aws-dynamodb::xxxxxxxx |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: | + swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \ + --name=time_to_live_deleted_item_count --service-name aws-dynamodb::xxxxxxxx |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: | + swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \ + --name=user_errors --service-name aws-dynamodb::xxxxxxxx |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: | + swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \ + --name=system_errors --service-name aws-dynamodb::xxxxxxxx |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: | + swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \ + --name=scan_returned_item_count --service-name aws-dynamodb::xxxxxxxx |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: | + swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \ + --name=query_returned_item_count --service-name aws-dynamodb::xxxxxxxx |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + + - query: | + swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \ + --name=read_throttled_requests --service-name aws-dynamodb::xxxxxxxx |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: | + swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \ + --name=write_throttled_requests --service-name aws-dynamodb::xxxxxxxx |yq e 'to_entries' - + expected: expected/metrics-has-value.yml \ No newline at end of file diff --git a/test/e2e-v2/cases/aws/dynamodb/e2e.yaml b/test/e2e-v2/cases/aws/dynamodb/e2e.yaml new file mode 100644 index 000000000000..e99cba86dd3c --- /dev/null +++ b/test/e2e-v2/cases/aws/dynamodb/e2e.yaml @@ -0,0 +1,44 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This file is used to show how to write configuration files and can be used to test. + +setup: + env: compose + file: docker-compose.yml + timeout: 20m + init-system-environment: ../../../script/env + steps: + - name: set PATH + command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH + - name: install yq + command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq + - name: install swctl + command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl + +trigger: + action: http + interval: 3s + times: 10 + url: http://localhost:9093/otel-metrics/send + method: GET + +verify: + retry: + count: 10 + interval: 3s + cases: + - includes: + - ./dynamodb-cases.yaml \ No newline at end of file diff --git a/test/e2e-v2/cases/aws/dynamodb/expected/metrics-has-value.yml b/test/e2e-v2/cases/aws/dynamodb/expected/metrics-has-value.yml new file mode 100644 index 000000000000..61632d4fd386 --- /dev/null +++ b/test/e2e-v2/cases/aws/dynamodb/expected/metrics-has-value.yml @@ -0,0 +1,19 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + {{- contains . }} +- key: {{ notEmpty .key }} + value: {{ ge .value 0 }} + {{- end }} \ No newline at end of file diff --git a/test/e2e-v2/cases/aws/dynamodb/expected/service.yml b/test/e2e-v2/cases/aws/dynamodb/expected/service.yml new file mode 100644 index 000000000000..1fe817b0be11 --- /dev/null +++ b/test/e2e-v2/cases/aws/dynamodb/expected/service.yml @@ -0,0 +1,24 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{{- contains . }} +- id: {{ b64enc "aws-dynamodb::xxxxxxxx" }}.1 + name: aws-dynamodb::xxxxxxxx + group: aws-dynamodb + shortname: xxxxxxxx + normal: true + layers: + - AWS_DYNAMODB + {{- end }} \ No newline at end of file diff --git a/test/e2e-v2/cases/aws/dynamodb/mock-data/data-1.json b/test/e2e-v2/cases/aws/dynamodb/mock-data/data-1.json new file mode 100644 index 000000000000..f4882e142fa4 --- /dev/null +++ b/test/e2e-v2/cases/aws/dynamodb/mock-data/data-1.json @@ -0,0 +1,556 @@ +{ + "resourceMetrics": [ + { + "resource": { + "attributes": [ + { + "key": "cloud.provider", + "value": { + "stringValue": "aws" + } + }, + { + "key": "cloud.account.id", + "value": { + "stringValue": "xxxxxxxx" + } + }, + { + "key": "cloud.region", + "value": { + "stringValue": "ap-northeast-1" + } + }, + { + "key": "aws.exporter.arn", + "value": { + "stringValue": "arn:aws:cloudwatch:ap-northeast-1:xxxxxxxx:metric-stream/CustomFull-CmHV2P" + } + } + ] + }, + "scopeMetrics": [ + { + "metrics": [ + { + "name": "amazonaws.com/AWS/DynamoDB/ProvisionedWriteCapacityUnits", + "unit": "{Count}", + "summary": { + "dataPoints": [ + { + "startTimeUnixNano": "1676809020000000000", + "timeUnixNano": "1676809080000000000", + "count": "1", + "sum": "2.0", + "quantileValues": [ + { + "value": 2.0 + }, + { + "quantile": 1.0, + "value": 2.0 + } + ], + "attributes": [ + { + "key": "Namespace", + "value": { + "stringValue": "AWS/DynamoDB" + } + }, + { + "key": "MetricName", + "value": { + "stringValue": "ProvisionedWriteCapacityUnits" + } + }, + { + "key": "TableName", + "value": { + "stringValue": "test" + } + } + ] + } + ] + } + }, + { + "name": "amazonaws.com/AWS/DynamoDB/AccountMaxTableLevelWrites", + "unit": "{Count}", + "summary": { + "dataPoints": [ + { + "startTimeUnixNano": "1676809080000000000", + "timeUnixNano": "1676809140000000000", + "count": "1", + "sum": "40000.0", + "quantileValues": [ + { + "value": 40000.0 + }, + { + "quantile": 1.0, + "value": 40000.0 + } + ], + "attributes": [ + { + "key": "Namespace", + "value": { + "stringValue": "AWS/DynamoDB" + } + }, + { + "key": "MetricName", + "value": { + "stringValue": "AccountMaxTableLevelWrites" + } + } + ] + } + ] + } + }, + { + "name": "amazonaws.com/AWS/DynamoDB/AccountProvisionedWriteCapacityUtilization", + "unit": "%", + "summary": { + "dataPoints": [ + { + "startTimeUnixNano": "1676809080000000000", + "timeUnixNano": "1676809140000000000", + "count": "1", + "quantileValues": [ + { + }, + { + "quantile": 1.0 + } + ], + "attributes": [ + { + "key": "Namespace", + "value": { + "stringValue": "AWS/DynamoDB" + } + }, + { + "key": "MetricName", + "value": { + "stringValue": "AccountProvisionedWriteCapacityUtilization" + } + } + ] + } + ] + } + }, + { + "name": "amazonaws.com/AWS/DynamoDB/AccountProvisionedReadCapacityUtilization", + "unit": "%", + "summary": { + "dataPoints": [ + { + "startTimeUnixNano": "1676809080000000000", + "timeUnixNano": "1676809140000000000", + "count": "1", + "sum": "2.0", + "quantileValues": [ + { + "value": 2.0 + }, + { + "quantile": 1.0, + "value": 2.0 + } + ], + "attributes": [ + { + "key": "Namespace", + "value": { + "stringValue": "AWS/DynamoDB" + } + }, + { + "key": "MetricName", + "value": { + "stringValue": "ProvisionedReadCapacityUnits" + } + } + ] + } + ] + } + }, + { + "name": "amazonaws.com/AWS/DynamoDB/MaxProvisionedTableWriteCapacityUtilization", + "unit": "%", + "summary": { + "dataPoints": [ + { + "startTimeUnixNano": "1676809320000000000", + "timeUnixNano": "1676809380000000000", + "count": "1", + "sum": "0.01", + "quantileValues": [ + { + "value": 0.01 + }, + { + "quantile": 1.0, + "value": 0.01 + } + ], + "attributes": [ + { + "key": "Namespace", + "value": { + "stringValue": "AWS/DynamoDB" + } + }, + { + "key": "MetricName", + "value": { + "stringValue": "MaxProvisionedTableWriteCapacityUtilization" + } + } + ] + } + ] + } + }, + { + "name": "amazonaws.com/AWS/DynamoDB/AccountMaxTableLevelReads", + "unit": "{Count}", + "summary": { + "dataPoints": [ + { + "startTimeUnixNano": "1676809320000000000", + "timeUnixNano": "1676809380000000000", + "count": "1", + "sum": "40000.0", + "quantileValues": [ + { + "value": 40000.0 + }, + { + "quantile": 1.0, + "value": 40000.0 + } + ], + "attributes": [ + { + "key": "Namespace", + "value": { + "stringValue": "AWS/DynamoDB" + } + }, + { + "key": "MetricName", + "value": { + "stringValue": "AccountMaxTableLevelReads" + } + } + ] + } + ] + } + }, + { + "name": "amazonaws.com/AWS/DynamoDB/AccountMaxWrites", + "unit": "{Count}", + "summary": { + "dataPoints": [ + { + "startTimeUnixNano": "1676809320000000000", + "timeUnixNano": "1676809380000000000", + "count": "1", + "sum": "80000.0", + "quantileValues": [ + { + "value": 80000.0 + }, + { + "quantile": 1.0, + "value": 80000.0 + } + ], + "attributes": [ + { + "key": "Namespace", + "value": { + "stringValue": "AWS/DynamoDB" + } + }, + { + "key": "MetricName", + "value": { + "stringValue": "AccountMaxWrites" + } + } + ] + } + ] + } + }, + { + "name": "amazonaws.com/AWS/DynamoDB/MaxProvisionedTableReadCapacityUtilization", + "unit": "%", + "summary": { + "dataPoints": [ + { + "startTimeUnixNano": "1676809320000000000", + "timeUnixNano": "1676809380000000000", + "count": "1", + "sum": "0.01", + "quantileValues": [ + { + "value": 0.01 + }, + { + "quantile": 1.0, + "value": 0.01 + } + ], + "attributes": [ + { + "key": "Namespace", + "value": { + "stringValue": "AWS/DynamoDB" + } + }, + { + "key": "MetricName", + "value": { + "stringValue": "MaxProvisionedTableReadCapacityUtilization" + } + } + ] + } + ] + } + }, + { + "name": "amazonaws.com/AWS/DynamoDB/TimeToLiveDeletedItemCount", + "unit": "{Count}", + "summary": { + "dataPoints": [ + { + "startTimeUnixNano": "1676809260000000000", + "timeUnixNano": "1676809320000000000", + "count": "1", + "sum": "2.0", + "quantileValues": [ + { + }, + { + "quantile": 1.0 + } + ], + "attributes": [ + { + "key": "Namespace", + "value": { + "stringValue": "AWS/DynamoDB" + } + }, + { + "key": "MetricName", + "value": { + "stringValue": "TimeToLiveDeletedItemCount" + } + }, + { + "key": "TableName", + "value": { + "stringValue": "test" + } + } + ] + } + ] + } + }, + { + "name": "amazonaws.com/AWS/DynamoDB/ReturnedItemCount", + "unit": "{Count}", + "summary": { + "dataPoints": [ + { + "startTimeUnixNano": "1676809260000000000", + "timeUnixNano": "1676809320000000000", + "count": "1", + "sum": "2.0", + "quantileValues": [ + { + }, + { + "quantile": 1.0 + } + ], + "attributes": [ + { + "key": "Namespace", + "value": { + "stringValue": "AWS/DynamoDB" + } + }, + { + "key": "MetricName", + "value": { + "stringValue": "ReturnedItemCount" + } + }, + { + "key": "Operation", + "value": { + "stringValue": "Scan" + } + }, + { + "key": "TableName", + "value": { + "stringValue": "test" + } + } + ] + } + ] + } + }, + { + "name": "amazonaws.com/AWS/DynamoDB/ReturnedItemCount", + "unit": "{Count}", + "summary": { + "dataPoints": [ + { + "startTimeUnixNano": "1676809260000000000", + "timeUnixNano": "1676809320000000000", + "count": "1", + "sum": "2.0", + "quantileValues": [ + { + }, + { + "quantile": 1.0 + } + ], + "attributes": [ + { + "key": "Namespace", + "value": { + "stringValue": "AWS/DynamoDB" + } + }, + { + "key": "MetricName", + "value": { + "stringValue": "ReturnedItemCount" + } + }, + { + "key": "Operation", + "value": { + "stringValue": "Query" + } + }, + { + "key": "TableName", + "value": { + "stringValue": "test" + } + } + ] + } + ] + } + }, + { + "name": "amazonaws.com/AWS/DynamoDB/UserErrors", + "unit": "{Count}", + "summary": { + "dataPoints": [ + { + "startTimeUnixNano": "1676809260000000000", + "timeUnixNano": "1676809320000000000", + "count": "1", + "sum": "2.0", + "quantileValues": [ + { + }, + { + "quantile": 1.0 + } + ], + "attributes": [ + { + "key": "Namespace", + "value": { + "stringValue": "AWS/DynamoDB" + } + }, + { + "key": "MetricName", + "value": { + "stringValue": "UserErrors" + } + }, + { + "key": "TableName", + "value": { + "stringValue": "test" + } + } + ] + } + ] + } + }, + { + "name": "amazonaws.com/AWS/DynamoDB/SystemErrors", + "unit": "{Count}", + "summary": { + "dataPoints": [ + { + "startTimeUnixNano": "1676809260000000000", + "timeUnixNano": "1676809320000000000", + "count": "1", + "sum": "2.0", + "quantileValues": [ + { + }, + { + "quantile": 1.0 + } + ], + "attributes": [ + { + "key": "Namespace", + "value": { + "stringValue": "AWS/DynamoDB" + } + }, + { + "key": "MetricName", + "value": { + "stringValue": "SystemErrors" + } + }, + { + "key": "TableName", + "value": { + "stringValue": "test" + } + } + ] + } + ] + } + } + ] + } + ] + } + ] +} \ No newline at end of file