diff --git a/packages/couchbase/_dev/build/docs/README.md b/packages/couchbase/_dev/build/docs/README.md index 15a094c05c2..336ec8fb202 100644 --- a/packages/couchbase/_dev/build/docs/README.md +++ b/packages/couchbase/_dev/build/docs/README.md @@ -12,10 +12,10 @@ For example, you could use the data from this integration to know when there are The Couchbase integration collects metrics data. -Metrics give you insight into the state of the Couchbase. Metrics data streams collected by the Couchbase integration include [Bucket](https://docs.couchbase.com/server/current/rest-api/rest-buckets-summary.html), [Cluster](https://docs.couchbase.com/server/current/rest-api/rest-cluster-details.html), [Cache](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#cache), [Couchbase Lite Replication](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#cbl_replication_pull), [Database](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#database), [Delta Sync](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#delta_sync), [GSI views](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#gsi_views), [Import](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#shared_bucket_import), [Resource Utilization](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#resource_utilization), [Security](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#security), and [XDCR](https://docs.couchbase.com/server/current/rest-api/rest-bucket-stats.html) metrics from [Couchbase](https://www.couchbase.com/) so that the user could monitor and troubleshoot the performance of the Couchbase instances. +Metrics give you insight into the state of the Couchbase. Metrics data streams collected by the Couchbase integration include [Bucket](https://docs.couchbase.com/server/current/rest-api/rest-buckets-summary.html), [Cluster](https://docs.couchbase.com/server/current/rest-api/rest-cluster-details.html), [Cache](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#cache), [Couchbase Lite Replication](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#cbl_replication_pull), [Database](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#database), [Delta Sync](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#delta_sync), [Eventing](https://developer.couchbase.com/tutorial-monitoring-eventing-service?learningPath=learn/couchbase-monitoring-guide#get-cluster-eventing-service-stats), [GSI views](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#gsi_views), [Import](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#shared_bucket_import), [Index](https://developer.couchbase.com/tutorial-monitoring-index-service#get-cluster-index-service-stats), [Query](https://developer.couchbase.com/tutorial-monitoring-query-service?learningPath=learn/couchbase-monitoring-guide#get-cluster-query-service-stats), [Resource Utilization](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#resource_utilization), [Security](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#security), and [XDCR](https://docs.couchbase.com/server/current/rest-api/rest-bucket-stats.html) metrics from [Couchbase](https://www.couchbase.com/) so that the user could monitor and troubleshoot the performance of the Couchbase instances. This integration uses: -- `http` metricbeat module to collect `bucket`, `cluster`, and `xdcr` metrics. +- `http` metricbeat module to collect `bucket`, `cluster`, `eventing`, `index`, `query`, and `xdcr` metrics. - `prometheus` metricbeat module to collect `cache`, `cbl_replication`, `database_stats`, `delta_sync`, `gsi_views`, `import`, `resource`, and `security` metrics. Note: For Couchbase cluster setup, there is an ideal scenario of a single host with administrator access for the entire cluster to collect metrics. Providing multiple hosts from the same cluster might lead to data duplication. In the case of multiple clusters, adding a new integration to collect data from different cluster hosts is a good option. @@ -95,6 +95,14 @@ Global Secondary Indexes (GSI) support queries made by the Query Service. {{fields "miscellaneous"}} +### Query Index + +This is the `query_index` data stream. The Query service enables you to issue queries to extract data from the Couchbase server. The Index collects statistics provided by the Index service. + +{{event "query_index"}} + +{{fields "query_index"}} + ### Resource Utilization This is the `resource` data stream. The Resource Utilization metrics are related to [MemStats](https://golang.org/pkg/runtime/#MemStats) records statistics about the memory allocator. diff --git a/packages/couchbase/changelog.yml b/packages/couchbase/changelog.yml index b9d007f7b1e..fb4b7ab805a 100644 --- a/packages/couchbase/changelog.yml +++ b/packages/couchbase/changelog.yml @@ -1,4 +1,9 @@ # newer versions go on top +- version: "0.8.0" + changes: + - description: Couchbase integration package with "query_index" data stream. + type: enhancement + link: https://github.com/elastic/integrations/pull/4100 - version: "0.7.0" changes: - description: Couchbase integration package with "cache" data stream. diff --git a/packages/couchbase/data_stream/query_index/_dev/test/pipeline/test-common-config.yml b/packages/couchbase/data_stream/query_index/_dev/test/pipeline/test-common-config.yml new file mode 100644 index 00000000000..c39dc386179 --- /dev/null +++ b/packages/couchbase/data_stream/query_index/_dev/test/pipeline/test-common-config.yml @@ -0,0 +1,2 @@ +dynamic_fields: + event.ingested: ".*" diff --git a/packages/couchbase/data_stream/query_index/_dev/test/pipeline/test-query-index-metrics.json b/packages/couchbase/data_stream/query_index/_dev/test/pipeline/test-query-index-metrics.json new file mode 100644 index 00000000000..2fcf7a4f12a --- /dev/null +++ b/packages/couchbase/data_stream/query_index/_dev/test/pipeline/test-query-index-metrics.json @@ -0,0 +1,3270 @@ +{ + "events": [ + { + "http": { + "query_index": { + "op": { + "samples": { + "timestamp": [ + 1659353663825, + 1659353664826, + 1659353665827, + 1659353666828, + 1659353667829, + 1659353668830, + 1659353669831, + 1659353670832, + 1659353671833, + 1659353672834, + 1659353673835, + 1659353674836, + 1659353675837, + 1659353676838, + 1659353677839, + 1659353678840, + 1659353679841, + 1659353680842, + 1659353681843, + 1659353682844, + 1659353683845, + 1659353684847, + 1659353685847, + 1659353686848, + 1659353687849, + 1659353688850, + 1659353689851, + 1659353690852, + 1659353691853, + 1659353692854, + 1659353693855, + 1659353694856, + 1659353695857, + 1659353696858, + 1659353697859, + 1659353698860, + 1659353699861, + 1659353700862, + 1659353701863, + 1659353702864, + 1659353703865, + 1659353704866, + 1659353705867, + 1659353706868, + 1659353707869, + 1659353708870, + 1659353709871, + 1659353710872, + 1659353711873, + 1659353712874, + 1659353713875, + 1659353714878, + 1659353715878, + 1659353716879, + 1659353717881, + 1659353718881, + 1659353719882, + 1659353720883, + 1659353721884, + 1659353722885 + ], + "index_memory_quota": [ + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912, + 536870912 + ], + "index_memory_used": [ + 481669616, + 481669616, + 481669616, + 481669616, + 481669616, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320, + 416846320 + ], + "index_ram_percent": [ + 89.71795737743378, + 89.71795737743378, + 89.71795737743378, + 89.71795737743378, + 89.71795737743378, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753, + 77.64367759227753 + ], + "index_remaining_ram": [ + 55201296, + 55201296, + 55201296, + 55201296, + 55201296, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592, + 120024592 + ] + }, + "samplesCount": 60, + "isPersistent": false, + "lastTStamp": 1659353722885, + "interval": 1000 + } + } + } + }, + { + "http": { + "query_index": { + "op": { + "samples": { + "query_avg_req_time": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_avg_svc_time": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_avg_response_size": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_avg_result_count": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_active_requests": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_errors": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_invalid_requests": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_queued_requests": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_request_time": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_requests": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_requests_1000ms": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_requests_250ms": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_requests_5000ms": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_requests_500ms": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_result_count": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_result_size": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_selects": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_service_time": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "query_warnings": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "timestamp": [ + 1659353866029, + 1659353867030, + 1659353868031, + 1659353869032, + 1659353870033, + 1659353871034, + 1659353872035, + 1659353873036, + 1659353874037, + 1659353875038, + 1659353876039, + 1659353877040, + 1659353878041, + 1659353879042, + 1659353880043, + 1659353881044, + 1659353882045, + 1659353883046, + 1659353884047, + 1659353885048, + 1659353886049, + 1659353887050, + 1659353888051, + 1659353889052, + 1659353890053, + 1659353891054, + 1659353892055, + 1659353893056, + 1659353894057, + 1659353895058, + 1659353896059, + 1659353897060, + 1659353898061, + 1659353899062, + 1659353900063, + 1659353901064, + 1659353902065, + 1659353903066, + 1659353904067, + 1659353905068, + 1659353906069, + 1659353907070, + 1659353908071, + 1659353909072, + 1659353910073, + 1659353911074, + 1659353912075, + 1659353913076, + 1659353914077, + 1659353915078, + 1659353916079, + 1659353917080, + 1659353918081, + 1659353919082, + 1659353920083, + 1659353921084, + 1659353922085, + 1659353923086, + 1659353924087, + 1659353925088 + ] + }, + "samplesCount": 60, + "isPersistent": false, + "lastTStamp": 1659353925088, + "interval": 1000 + } + } + } + }, + { + "http": { + "query_index": { + "op": { + "samples": { + "timestamp": [ + 1659353936099, + 1659353937100, + 1659353938101, + 1659353939102, + 1659353940103, + 1659353941104, + 1659353942105, + 1659353943106, + 1659353944107, + 1659353945108, + 1659353946109, + 1659353947110, + 1659353948111, + 1659353949112, + 1659353950113, + 1659353951114, + 1659353952115, + 1659353953116, + 1659353954117, + 1659353955118, + 1659353956119, + 1659353957120, + 1659353958121, + 1659353959122, + 1659353960123, + 1659353961124, + 1659353962125, + 1659353963126, + 1659353964127, + 1659353965128, + 1659353966129, + 1659353967130, + 1659353968131, + 1659353969132, + 1659353970133, + 1659353971134, + 1659353972135, + 1659353973136, + 1659353974137, + 1659353975138, + 1659353976139, + 1659353977140, + 1659353978141, + 1659353979142, + 1659353980143, + 1659353981144, + 1659353982145, + 1659353983146, + 1659353984147, + 1659353985148, + 1659353986149, + 1659353987150, + 1659353988151, + 1659353989152, + 1659353990153, + 1659353991154, + 1659353992155, + 1659353993156, + 1659353994157, + 1659353995158 + ], + "eventing/bucket_op_exception_count": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/checkpoint_failure_count": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/dcp_backlog": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/failed_count": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/n1ql_op_exception_count": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/on_delete_failure": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/on_delete_success": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/on_update_failure": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/on_update_success": [ + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467 + ], + "eventing/processed_count": [ + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467 + ], + "eventing/timeout_count": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/timer_callback_failure": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/timer_callback_success": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/travel-sample/tenant_agent_02/Test/bucket_op_exception_count": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/travel-sample/tenant_agent_02/Test/checkpoint_failure_count": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/travel-sample/tenant_agent_02/Test/dcp_backlog": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/travel-sample/tenant_agent_02/Test/failed_count": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/travel-sample/tenant_agent_02/Test/n1ql_op_exception_count": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/travel-sample/tenant_agent_02/Test/on_delete_failure": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/travel-sample/tenant_agent_02/Test/on_delete_success": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/travel-sample/tenant_agent_02/Test/on_update_failure": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/travel-sample/tenant_agent_02/Test/on_update_success": [ + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467 + ], + "eventing/travel-sample/tenant_agent_02/Test/processed_count": [ + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252325, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467, + 1252467 + ], + "eventing/travel-sample/tenant_agent_02/Test/timeout_count": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/travel-sample/tenant_agent_02/Test/timer_callback_failure": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "eventing/travel-sample/tenant_agent_02/Test/timer_callback_success": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + "samplesCount": 60, + "isPersistent": false, + "lastTStamp": 1659353995158, + "interval": 1000 + } + } + } + } + ] +} \ No newline at end of file diff --git a/packages/couchbase/data_stream/query_index/_dev/test/pipeline/test-query-index-metrics.json-expected.json b/packages/couchbase/data_stream/query_index/_dev/test/pipeline/test-query-index-metrics.json-expected.json new file mode 100644 index 00000000000..41e1cb1bd9d --- /dev/null +++ b/packages/couchbase/data_stream/query_index/_dev/test/pipeline/test-query-index-metrics.json-expected.json @@ -0,0 +1,79 @@ +{ + "expected": [ + { + "couchbase": { + "query_index": { + "ram": { + "pct": 77.64367759227753, + "remaining": 120024592 + } + } + }, + "ecs": { + "version": "8.3.0" + }, + "event": { + "category": [ + "database" + ], + "kind": "metric", + "module": "couchbase", + "type": [ + "info" + ] + } + }, + { + "couchbase": { + "query_index": { + "query": { + "request_time": { + "avg": 0 + }, + "requests": 0, + "result": { + "count": 0 + } + } + } + }, + "ecs": { + "version": "8.3.0" + }, + "event": { + "category": [ + "database" + ], + "kind": "metric", + "module": "couchbase", + "type": [ + "info" + ] + } + }, + { + "couchbase": { + "query_index": { + "eventing": { + "failed": { + "count": 0 + } + } + } + }, + "ecs": { + "version": "8.3.0" + }, + "event": { + "category": [ + "database" + ], + "kind": "metric", + "module": "couchbase", + "type": [ + "info" + ] + } + } + ] +} \ No newline at end of file diff --git a/packages/couchbase/data_stream/query_index/_dev/test/system/test-default-config.yml b/packages/couchbase/data_stream/query_index/_dev/test/system/test-default-config.yml new file mode 100644 index 00000000000..2734146004c --- /dev/null +++ b/packages/couchbase/data_stream/query_index/_dev/test/system/test-default-config.yml @@ -0,0 +1,6 @@ +input: http/metrics +vars: + hosts: + - http://Administrator:password@{{Hostname}}:{{Port[0]}} +data_stream: + vars: ~ diff --git a/packages/couchbase/data_stream/query_index/agent/stream/stream.yml.hbs b/packages/couchbase/data_stream/query_index/agent/stream/stream.yml.hbs new file mode 100644 index 00000000000..21960f1b367 --- /dev/null +++ b/packages/couchbase/data_stream/query_index/agent/stream/stream.yml.hbs @@ -0,0 +1,23 @@ +metricsets: ["json"] +namespace: "query_index" +hosts: +{{#each hosts}} + - "{{this}}/pools/default/buckets/@eventing/stats" + - "{{this}}/pools/default/buckets/@index/stats" + - "{{this}}/pools/default/buckets/@query/stats" +{{/each}} +{{#if ssl}} +{{ssl}} +{{/if}} +period: {{period}} +tags: +{{#each tags as |tag|}} + - {{tag}} +{{/each}} +{{#contains tags "forwarded"}} +publisher_pipeline.disable_host: true +{{/contains}} +{{#if processors}} +processors: +{{processors}} +{{/if}} diff --git a/packages/couchbase/data_stream/query_index/elasticsearch/ingest_pipeline/default.yml b/packages/couchbase/data_stream/query_index/elasticsearch/ingest_pipeline/default.yml new file mode 100644 index 00000000000..9bf0ec3614e --- /dev/null +++ b/packages/couchbase/data_stream/query_index/elasticsearch/ingest_pipeline/default.yml @@ -0,0 +1,114 @@ +--- +description: Pipeline for parsing Couchbase Query Index metrics. +processors: +- set: + field: ecs.version + value: 8.3.0 + ignore_failure: true + ignore_empty_value: true +- set: + field: event.type + value: [info] + ignore_failure: true + ignore_empty_value: true +- set: + field: event.kind + value: metric + ignore_failure: true + ignore_empty_value: true +- set: + field: event.category + value: [database] + ignore_failure: true + ignore_empty_value: true +- set: + field: event.module + value: couchbase + override: true + ignore_failure: true + ignore_empty_value: true +- script: + lang: painless + source: |- + def getLastElement(def arr) { + def last_index = arr.length - 1; + return arr[last_index]; + } + if (ctx.http?.query_index?.op?.samples != null) { + if (ctx.http.query_index.op.samples.index_remaining_ram != null) { + ctx.index_remaining_ram = getLastElement(ctx.http.query_index.op.samples.index_remaining_ram); + } + if (ctx.http.query_index.op.samples.index_ram_percent != null) { + ctx.index_ram_percent = getLastElement(ctx.http.query_index.op.samples.index_ram_percent); + } + if (ctx.http.query_index.op.samples.query_avg_req_time != null) { + ctx.query_avg_req_time = getLastElement(ctx.http.query_index.op.samples.query_avg_req_time); + } + if (ctx.http.query_index.op.samples.query_requests != null) { + ctx.query_requests = getLastElement(ctx.http.query_index.op.samples.query_requests); + } + if (ctx.http.query_index.op.samples.query_result_count != null) { + ctx.query_result_count = getLastElement(ctx.http.query_index.op.samples.query_result_count); + } + if (ctx.http.query_index.op.samples['eventing/failed_count'] != null) { + ctx.eventing_failed_count = getLastElement(ctx.http.query_index.op.samples['eventing/failed_count']); + } + } + ignore_failure: true + description: Get the last value from an array response. +- rename: + field: index_remaining_ram + target_field: couchbase.query_index.ram.remaining + ignore_missing: true + ignore_failure: true +- rename: + field: index_ram_percent + target_field: couchbase.query_index.ram.pct + ignore_missing: true + ignore_failure: true +- rename: + field: query_avg_req_time + target_field: couchbase.query_index.query.request_time.avg + ignore_missing: true + ignore_failure: true +- rename: + field: query_requests + target_field: couchbase.query_index.query.requests + ignore_missing: true + ignore_failure: true +- rename: + field: query_result_count + target_field: couchbase.query_index.query.result.count + ignore_missing: true + ignore_failure: true +- rename: + field: eventing_failed_count + target_field: couchbase.query_index.eventing.failed.count + ignore_missing: true + ignore_failure: true +- script: + description: Drops null/empty values recursively. + lang: painless + source: | + boolean drop(Object o) { + if (o == null || o == "") { + return true; + } else if (o instanceof Map) { + ((Map) o).values().removeIf(v -> drop(v)); + return (((Map) o).size() == 0); + } else if (o instanceof List) { + ((List) o).removeIf(v -> drop(v)); + return (((List) o).length == 0); + } + return false; + } + drop(ctx); +- remove: + field: + - http + ignore_missing: true + ignore_failure: true +on_failure: +- set: + field: error.message + value: "{{{_ingest.on_failure_message}}}" diff --git a/packages/couchbase/data_stream/query_index/fields/base-fields.yml b/packages/couchbase/data_stream/query_index/fields/base-fields.yml new file mode 100644 index 00000000000..7c798f4534c --- /dev/null +++ b/packages/couchbase/data_stream/query_index/fields/base-fields.yml @@ -0,0 +1,12 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. diff --git a/packages/couchbase/data_stream/query_index/fields/ecs.yml b/packages/couchbase/data_stream/query_index/fields/ecs.yml new file mode 100644 index 00000000000..a1549907d13 --- /dev/null +++ b/packages/couchbase/data_stream/query_index/fields/ecs.yml @@ -0,0 +1,24 @@ +- external: ecs + name: ecs.version +- external: ecs + name: error.message +- external: ecs + name: event.category +- external: ecs + name: event.dataset +- external: ecs + name: event.duration +- external: ecs + name: event.ingested +- external: ecs + name: event.kind +- external: ecs + name: event.module +- external: ecs + name: event.type +- external: ecs + name: service.address +- external: ecs + name: service.type +- external: ecs + name: tags diff --git a/packages/couchbase/data_stream/query_index/fields/fields.yml b/packages/couchbase/data_stream/query_index/fields/fields.yml new file mode 100644 index 00000000000..ffec08afb1a --- /dev/null +++ b/packages/couchbase/data_stream/query_index/fields/fields.yml @@ -0,0 +1,34 @@ +- name: couchbase.query_index + type: group + fields: + - name: eventing.failed.count + type: float + metrics_type: gauge + description: Total number of failed eventing function operations. + - name: ram + type: group + fields: + - name: pct + type: float + metrics_type: gauge + description: The percentage of index entries in ram. + - name: remaining + type: float + metrics_type: gauge + description: The amount of memory remaining. + - name: query + type: group + fields: + - name: requests + type: float + metrics_type: gauge + description: Current number of requests per second. + - name: request_time.avg + type: float + metrics_type: gauge + unit: s + description: Average total request time. + - name: result.count + type: float + metrics_type: gauge + description: Number of results returned. diff --git a/packages/couchbase/data_stream/query_index/manifest.yml b/packages/couchbase/data_stream/query_index/manifest.yml new file mode 100644 index 00000000000..16e9ea321fb --- /dev/null +++ b/packages/couchbase/data_stream/query_index/manifest.yml @@ -0,0 +1,31 @@ +title: Query Index metrics +type: metrics +streams: + - input: http/metrics + template_path: stream.yml.hbs + title: Collect Query Index metrics + description: Collect metrics from Couchbase query indexes of buckets. + vars: + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + - name: tags + type: text + title: Tags + multi: true + required: false + show_user: false + default: + - forwarded + - couchbase-query_index + - name: processors + type: yaml + title: Processors + multi: false + required: false + show_user: false + description: Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details. diff --git a/packages/couchbase/data_stream/query_index/sample_event.json b/packages/couchbase/data_stream/query_index/sample_event.json new file mode 100644 index 00000000000..30e1705d141 --- /dev/null +++ b/packages/couchbase/data_stream/query_index/sample_event.json @@ -0,0 +1,84 @@ +{ + "@timestamp": "2022-09-26T07:33:29.682Z", + "agent": { + "ephemeral_id": "dc8dbbf1-9518-4b02-be55-d52cd3e2684e", + "id": "80e6ae5e-391d-4c35-af53-92fea05a9d3a", + "name": "docker-fleet-agent", + "type": "metricbeat", + "version": "8.4.1" + }, + "couchbase": { + "query_index": { + "query": { + "request_time": { + "avg": 0.0178967996 + }, + "requests": 2.5, + "result": { + "count": 1.3 + } + } + } + }, + "data_stream": { + "dataset": "couchbase.query_index", + "namespace": "ep", + "type": "metrics" + }, + "ecs": { + "version": "8.3.0" + }, + "elastic_agent": { + "id": "80e6ae5e-391d-4c35-af53-92fea05a9d3a", + "snapshot": false, + "version": "8.4.1" + }, + "event": { + "agent_id_status": "verified", + "category": [ + "database" + ], + "dataset": "couchbase.query_index", + "duration": 97251822, + "ingested": "2022-09-26T07:33:34Z", + "kind": "metric", + "module": "couchbase", + "type": [ + "info" + ] + }, + "host": { + "architecture": "x86_64", + "containerized": true, + "hostname": "docker-fleet-agent", + "id": "51511c1493f34922b559a964798246ec", + "ip": [ + "172.23.0.7" + ], + "mac": [ + "02:42:ac:17:00:07" + ], + "name": "docker-fleet-agent", + "os": { + "codename": "focal", + "family": "debian", + "kernel": "5.4.0-126-generic", + "name": "Ubuntu", + "platform": "ubuntu", + "type": "linux", + "version": "20.04.4 LTS (Focal Fossa)" + } + }, + "metricset": { + "name": "json", + "period": 10000 + }, + "service": { + "address": "http://elastic-package-service_couchbase_1:8091/pools/default/buckets/@query/stats", + "type": "http" + }, + "tags": [ + "forwarded", + "couchbase-query_index" + ] +} \ No newline at end of file diff --git a/packages/couchbase/docs/README.md b/packages/couchbase/docs/README.md index 325e5ae20ce..c1714dcce99 100644 --- a/packages/couchbase/docs/README.md +++ b/packages/couchbase/docs/README.md @@ -12,10 +12,10 @@ For example, you could use the data from this integration to know when there are The Couchbase integration collects metrics data. -Metrics give you insight into the state of the Couchbase. Metrics data streams collected by the Couchbase integration include [Bucket](https://docs.couchbase.com/server/current/rest-api/rest-buckets-summary.html), [Cluster](https://docs.couchbase.com/server/current/rest-api/rest-cluster-details.html), [Cache](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#cache), [Couchbase Lite Replication](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#cbl_replication_pull), [Database](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#database), [Delta Sync](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#delta_sync), [GSI views](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#gsi_views), [Import](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#shared_bucket_import), [Resource Utilization](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#resource_utilization), [Security](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#security), and [XDCR](https://docs.couchbase.com/server/current/rest-api/rest-bucket-stats.html) metrics from [Couchbase](https://www.couchbase.com/) so that the user could monitor and troubleshoot the performance of the Couchbase instances. +Metrics give you insight into the state of the Couchbase. Metrics data streams collected by the Couchbase integration include [Bucket](https://docs.couchbase.com/server/current/rest-api/rest-buckets-summary.html), [Cluster](https://docs.couchbase.com/server/current/rest-api/rest-cluster-details.html), [Cache](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#cache), [Couchbase Lite Replication](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#cbl_replication_pull), [Database](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#database), [Delta Sync](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#delta_sync), [Eventing](https://developer.couchbase.com/tutorial-monitoring-eventing-service?learningPath=learn/couchbase-monitoring-guide#get-cluster-eventing-service-stats), [GSI views](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#gsi_views), [Import](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#shared_bucket_import), [Index](https://developer.couchbase.com/tutorial-monitoring-index-service#get-cluster-index-service-stats), [Query](https://developer.couchbase.com/tutorial-monitoring-query-service?learningPath=learn/couchbase-monitoring-guide#get-cluster-query-service-stats), [Resource Utilization](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#resource_utilization), [Security](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#security), and [XDCR](https://docs.couchbase.com/server/current/rest-api/rest-bucket-stats.html) metrics from [Couchbase](https://www.couchbase.com/) so that the user could monitor and troubleshoot the performance of the Couchbase instances. This integration uses: -- `http` metricbeat module to collect `bucket`, `cluster`, and `xdcr` metrics. +- `http` metricbeat module to collect `bucket`, `cluster`, `eventing`, `index`, `query`, and `xdcr` metrics. - `prometheus` metricbeat module to collect `cache`, `cbl_replication`, `database_stats`, `delta_sync`, `gsi_views`, `import`, `resource`, and `security` metrics. Note: For Couchbase cluster setup, there is an ideal scenario of a single host with administrator access for the entire cluster to collect metrics. Providing multiple hosts from the same cluster might lead to data duplication. In the case of multiple clusters, adding a new integration to collect data from different cluster hosts is a good option. @@ -902,6 +902,127 @@ An example event for `miscellaneous` looks as following: | tags | List of keywords used to tag each event. | keyword | | +### Query Index + +This is the `query_index` data stream. The Query service enables you to issue queries to extract data from the Couchbase server. The Index collects statistics provided by the Index service. + +An example event for `query_index` looks as following: + +```json +{ + "@timestamp": "2022-09-26T07:33:29.682Z", + "agent": { + "ephemeral_id": "dc8dbbf1-9518-4b02-be55-d52cd3e2684e", + "id": "80e6ae5e-391d-4c35-af53-92fea05a9d3a", + "name": "docker-fleet-agent", + "type": "metricbeat", + "version": "8.4.1" + }, + "couchbase": { + "query_index": { + "query": { + "request_time": { + "avg": 0.0178967996 + }, + "requests": 2.5, + "result": { + "count": 1.3 + } + } + } + }, + "data_stream": { + "dataset": "couchbase.query_index", + "namespace": "ep", + "type": "metrics" + }, + "ecs": { + "version": "8.3.0" + }, + "elastic_agent": { + "id": "80e6ae5e-391d-4c35-af53-92fea05a9d3a", + "snapshot": false, + "version": "8.4.1" + }, + "event": { + "agent_id_status": "verified", + "category": [ + "database" + ], + "dataset": "couchbase.query_index", + "duration": 97251822, + "ingested": "2022-09-26T07:33:34Z", + "kind": "metric", + "module": "couchbase", + "type": [ + "info" + ] + }, + "host": { + "architecture": "x86_64", + "containerized": true, + "hostname": "docker-fleet-agent", + "id": "51511c1493f34922b559a964798246ec", + "ip": [ + "172.23.0.7" + ], + "mac": [ + "02:42:ac:17:00:07" + ], + "name": "docker-fleet-agent", + "os": { + "codename": "focal", + "family": "debian", + "kernel": "5.4.0-126-generic", + "name": "Ubuntu", + "platform": "ubuntu", + "type": "linux", + "version": "20.04.4 LTS (Focal Fossa)" + } + }, + "metricset": { + "name": "json", + "period": 10000 + }, + "service": { + "address": "http://elastic-package-service_couchbase_1:8091/pools/default/buckets/@query/stats", + "type": "http" + }, + "tags": [ + "forwarded", + "couchbase-query_index" + ] +} +``` + +**Exported fields** + +| Field | Description | Type | Unit | +|---|---|---|---| +| @timestamp | Event timestamp. | date | | +| couchbase.query_index.eventing.failed.count | Total number of failed eventing function operations. | float | | +| couchbase.query_index.query.request_time.avg | Average total request time. | float | s | +| couchbase.query_index.query.requests | Current number of requests per second. | float | | +| couchbase.query_index.query.result.count | Number of results returned. | float | | +| couchbase.query_index.ram.pct | The percentage of index entries in ram. | float | | +| couchbase.query_index.ram.remaining | The amount of memory remaining. | float | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | +| data_stream.type | Data stream type. | constant_keyword | | +| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword | | +| error.message | Error message. | match_only_text | | +| event.category | This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy. `event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories. | keyword | | +| event.dataset | Name of the dataset. If an event source publishes more than one type of log or events (e.g. access log, error log), the dataset is used to specify which one the event comes from. It's recommended but not required to start the dataset name with the module name, followed by a dot, then the dataset name. | keyword | | +| event.duration | Duration of the event in nanoseconds. If event.start and event.end are known this value should be the difference between the end and start time. | long | | +| event.ingested | Timestamp when an event arrived in the central data store. This is different from `@timestamp`, which is when the event originally occurred. It's also different from `event.created`, which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: `@timestamp` \< `event.created` \< `event.ingested`. | date | | +| event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. | keyword | | +| event.module | Name of the module this data is coming from. If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module. | keyword | | +| event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword | | +| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | +| service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, `service.type` would be `elasticsearch`. | keyword | | +| tags | List of keywords used to tag each event. | keyword | | + + ### Resource Utilization This is the `resource` data stream. The Resource Utilization metrics are related to [MemStats](https://golang.org/pkg/runtime/#MemStats) records statistics about the memory allocator. diff --git a/packages/couchbase/manifest.yml b/packages/couchbase/manifest.yml index 76fc014ba2f..192b87b11cf 100644 --- a/packages/couchbase/manifest.yml +++ b/packages/couchbase/manifest.yml @@ -1,7 +1,7 @@ format_version: 1.0.0 name: couchbase title: Couchbase -version: 0.7.0 +version: 0.8.0 license: basic description: Collect metrics from Couchbase databases with Elastic Agent. type: integration