Skip to content

Support Amazon API Gateway monitoring #10645

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wu-sheng merged 7 commits into from
Apr 5, 2023
Merged

Support Amazon API Gateway monitoring #10645

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
bdea591
Support Amazon API Gateway monitoring
pg-yang Apr 2, 2023
3ff1511
lowcase
pg-yang Apr 2, 2023
e9c9900
fix e2e
pg-yang Apr 2, 2023
7a6897f
update api gateway doc
pg-yang Apr 4, 2023
15fc590
revert enabledOtelRules config, add aws gateway to gateway menu doc too
pg-yang Apr 4, 2023
04b065b
use colon replace space
pg-yang Apr 4, 2023
3226d7a
Merge branch 'master' into api-gateway-v2
wu-sheng Apr 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/skywalking.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -673,9 +673,10 @@ jobs:
config: test/e2e-v2/cases/aws/s3/e2e.yaml
- name: AWS Cloud DynamoDB
config: test/e2e-v2/cases/aws/dynamodb/e2e.yaml

- name: PromQL Service
config: test/e2e-v2/cases/promql/e2e.yaml
- name: AWS API Gateway
config: test/e2e-v2/cases/aws/api-gateway/e2e.yaml
steps:
- uses: actions/checkout@v3
with:
Expand Down
1 change: 1 addition & 0 deletions docs/en/changes/changes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
* Support collecting memory buff/cache metrics in VM monitoring.
* PromQL: Remove empty values from the query result, fix `/api/v1/metadata` param `limit` could cause out of bound.
* Support monitoring the total number metrics of k8s StatefulSet and DaemonSet.
* Support Amazon API Gateway monitoring.

#### UI
* Revert: cpm5d function. This feature is cancelled from backend.
Expand Down
46 changes: 46 additions & 0 deletions docs/en/setup/backend/backend-aws-api-gateway-monitoring.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# AWS API Gateway monitoring
Amazon API Gateway is an AWS service for creating, publishing, maintaining, monitoring, and securing REST, HTTP, and WebSocket APIs. SkyWalking leverages [AWS Kinesis Data Firehose receiver](./aws-firehose-receiver.md) to transfer the CloudWatch metrics of API Gateway(HTTP and REST APIs) to
[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/meter.md).

### Data flow
1. AWS CloudWatch collect metrics for API Gateway(REST and HTTP APIs), refer to [API Gateway HTTP APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-metrics.html) and [API Gateway REST APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html)
2. [CloudWatch metric streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html) stream CloudWatch metrics of API Gateway to AWS Kinesis Data Firehose
3. AWS Kinesis Data Firehose delivery metrics to [AWS Kinesis Data Firehose receiver](./aws-firehose-receiver.md) through the HTTP endpoint

### Set up
1. Enable CloudWatch metrics for API Gateway
2. Create an Amazon Kinesis Data Firehose Delivery Stream, and set [AWS Kinesis Data Firehose receiver](./aws-firehose-receiver.md)'s address as HTTP(s) Destination, refer to [Create Delivery Stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html)
3. Create CloudWatch metric stream, and select the Firehose Delivery Stream which has been created above, set `Select namespaces` to `AWS/ApiGateway`, `Select output format` to `OpenTelemetry 0.7`. refer to [CloudWatch Metric Streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html)

### Gateway Monitoring

SkyWalking observes CloudWatch metrics of the AWS API Gateway, which is cataloged as a `LAYER: AWS_GATEWAY` `Service` in the OAP. Meanwhile, the routes would be recognized as `LAYER: AWS_GATEWAY` `endpoint`s

#### Supported Metrics

| Monitoring Panel | Unit | Metric Name | Catalog | Description | Data Source |
|-------------------------------------|-------|------------------------------------------|------------|---------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Request Count | count | aws_gateway_service_count | Service | The total number API requests in a given period. | [API Gateway HTTP APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-metrics.html) and [API Gateway REST APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html) |
| 4xx Count | count | aws_gateway_service_4xx | Service | The number of client-side errors captured in a given period. | [API Gateway HTTP APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-metrics.html) and [API Gateway REST APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html) |
| 5xx Count | count | aws_gateway_service_5xx | Service | The number of server-side errors captured in a given period. | [API Gateway HTTP APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-metrics.html) and [API Gateway REST APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html) |
| Request Average Latency | ms | aws_gateway_service_latency | Service | The time between when API Gateway receives a request from a client and when it returns a response to the client. | [API Gateway HTTP APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-metrics.html) and [API Gateway REST APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html) |
| Request Average Integration Latency | ms | aws_gateway_service_integration_latency | Service | The time between when API Gateway relays a request to the backend and when it receives a response from the backend. | [API Gateway HTTP APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-metrics.html) and [API Gateway REST APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html) |
| Data Processed | KB | aws_gateway_service_data_processed | Service | The amount of data processed | [API Gateway HTTP APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-metrics.html) |
| Cache Hit Count Rate | % | aws_gateway_service_cache_hit_rate | Service | The number of requests served from the API cache | [API Gateway REST APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html) |
| Cache Miss Count Rate | % | aws_gateway_service_cache_miss_rate | Service | The number of requests served from the backend | [API Gateway REST APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html) |
| Request Count | count | aws_gateway_endpoint_count | Endpoint | The total number API requests in a given period. | [API Gateway HTTP APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-metrics.html) and [API Gateway REST APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html) |
| 4xx Count | count | aws_gateway_endpoint_4xx | Endpoint | The number of client-side errors captured in a given period. | [API Gateway HTTP APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-metrics.html) and [API Gateway REST APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html) |
| 5xx Count | count | aws_gateway_endpoint_5xx | Endpoint | The number of server-side errors captured in a given period. | [API Gateway HTTP APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-metrics.html) and [API Gateway REST APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html) |
| Request Average Latency | ms | aws_gateway_endpoint_latency | Endpoint | The time between when API Gateway receives a request from a client and when it returns a response to the client. | [API Gateway HTTP APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-metrics.html) and [API Gateway REST APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html) |
| Request Average Integration Latency | ms | aws_gateway_endpoint_integration_latency | Endpoint | The time between when API Gateway relays a request to the backend and when it receives a response from the backend. | [API Gateway HTTP APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-metrics.html) and [API Gateway REST APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html) |
| Data Processed | KB | aws_gateway_endpoint_data_processed | Endpoint | The amount of data processed | [API Gateway HTTP APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-metrics.html) |
| Cache Hit Count Rate | % | aws_gateway_endpoint_cache_hit_rate | Endpoint | The number of requests served from the API cache | [API Gateway REST APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html) |
| Cache Miss Count Rate | % | aws_gateway_endpoint_cache_miss_rate | Endpoint | The number of requests served from the backend | [API Gateway REST APIs monitoring with CloudWatch](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html) |




### Customizations
You can customize your own metrics/expression/dashboard panel.
The metrics definition and expression rules are found in `/config/otel-rules/aws-gateway/`.
The AWS Cloud EKS dashboard panel configurations are found in `/config/ui-initialized-templates/aws_gateway`.
4 changes: 4 additions & 0 deletions docs/menu.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -205,12 +205,16 @@ catalog:
path: "/en/setup/backend/backend-aws-s3-monitoring"
- name: "AWS DynamoDB"
path: "/en/setup/backend/backend-aws-dynamodb-monitoring"
- name: "AWS API Gateway"
path: "/en/setup/backend/backend-aws-api-gateway-monitoring"
Comment thread
pg-yang marked this conversation as resolved.
- name: "Browser Monitoring"
path: "/en/setup/service-agent/browser-agent"
- name: "Gateway Monitoring"
catalog:
- name: "APISIX Monitoring"
path: "/en/setup/backend/backend-apisix-monitoring"
- name: "AWS API Gateway"
path: "/en/setup/backend/backend-aws-api-gateway-monitoring"
- name: "Database Monitoring"
catalog:
- name: "MySQL/MariaDB Server"
Expand Down
8 changes: 6 additions & 2 deletions ...ter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -464,7 +464,7 @@ public SampleFamily instance(List<String> serviceKeys, List<String> instanceKeys
return instance(serviceKeys, Const.POINT, instanceKeys, Const.POINT, layer, null);
}

public SampleFamily endpoint(List<String> serviceKeys, List<String> endpointKeys, Layer layer) {
public SampleFamily endpoint(List<String> serviceKeys, List<String> endpointKeys, String delimiter, Layer layer) {
Comment thread
pg-yang marked this conversation as resolved.
Preconditions.checkArgument(serviceKeys.size() > 0);
Preconditions.checkArgument(endpointKeys.size() > 0);
ExpressionParsingContext.get().ifPresent(ctx -> {
Expand All @@ -475,7 +475,11 @@ public SampleFamily endpoint(List<String> serviceKeys, List<String> endpointKeys
if (this == EMPTY) {
return EMPTY;
}
return createMeterSamples(new EndpointEntityDescription(serviceKeys, endpointKeys, layer, Const.POINT));
return createMeterSamples(new EndpointEntityDescription(serviceKeys, endpointKeys, layer, delimiter));
}

public SampleFamily endpoint(List<String> serviceKeys, List<String> endpointKeys, Layer layer) {
return endpoint(serviceKeys, endpointKeys, Const.POINT, layer);
}

public SampleFamily process(List<String> serviceKeys, List<String> serviceInstanceKeys, List<String> processKeys, String layerKey) {
Expand Down
13 changes: 10 additions & 3 deletions ...erver/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,8 @@ public enum Layer {
K8S_SERVICE(17, true),

/**
* MySQL Server, also known as mysqld, is a single multithreaded program that does most of the work in a MySQL installation.
* MySQL Server, also known as mysqld, is a single multithreaded program that does most of the work in a MySQL
* installation.
*/
MYSQL(18, true),

Expand Down Expand Up @@ -153,11 +154,17 @@ public enum Layer {
*/
AWS_S3(24, true),

/*
/**
* Amazon DynamoDB is a fully managed NoSQL database service that provides
* fast and predictable performance with seamless scalability.
*/
AWS_DYNAMODB(25, true);
AWS_DYNAMODB(25, true),

/**
* Amazon API Gateway is an AWS service for creating, publishing, maintaining, monitoring, and securing REST, HTTP,
* and WebSocket APIs at any scale.
*/
AWS_GATEWAY(26, true);

private final int value;
/**
Expand Down
1 change: 1 addition & 0 deletions ...a/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ public class UITemplateInitializer {
Layer.OS_WINDOWS.name(),
Layer.AWS_S3.name(),
Layer.AWS_DYNAMODB.name(),
Layer.AWS_GATEWAY.name(),
"custom"
};
private final UITemplateManagementService uiTemplateManagementService;
Expand Down
Loading