[Metrics][Discover] Fix fields dropdown by casting dimension fields to string by MiriamAparicio · Pull Request #242432 · elastic/kibana

MiriamAparicio · 2025-11-10T14:51:40Z

Summary

This PR resolves the data inconsistency between metric charts and the "Filter by value" dropdown, where values visible in charts were missing from the filter list.

Problem

The root cause was that the same dimension field (e.g., attributes.cpu) could have different data types across different indices:

long in one index
keyword in another index

Elasticsearch aggregations and ES|QL queries fail when trying to process fields with inconsistent types, resulting in incomplete dimension values in the dropdown.

Solution

1. Updated get_dimentions.ts:

Migrated from Elasticsearch _search API with terms aggregations to ES|QL
Added explicit type casting to string using ::STRING syntax
Increased dimension value limit from 20 to 1000
Simplified response mapping logic

2. Updated create_esql_query.ts:

Added type casting logic for dimensions in WHERE clauses
Applied the same needsStringCasting() check used in CONCAT operations
Ensures consistent string type handling for null checks

BEFORE

AFTER

jennypavlova

LGTM, thanks for the fix!
Just a question, when I tested with edge lite data, I saw that if we have the 4 values but the charts are supporting 2 each, I saw an error when filtered by it in the charts where the value is not present:

Is that expected, or maybe a data issue on my end?

src/platform/plugins/shared/metrics_experience/server/routes/dimensions/get_dimensions.ts

crespocarlos · 2025-11-11T08:44:22Z

src/platform/plugins/shared/metrics_experience/server/routes/dimensions/get_dimensions.ts

+  const source = fromCommand(indices);
+  const query = source
+    .pipe(
+      evaluate('??dim = ??dim::string', { dim: dimensions[0] }),


For now, it's ok, but in the future, we might support multi-dimensions, and we'll have to rethink this query.

crespocarlos · 2025-11-11T08:44:52Z

src/platform/plugins/shared/metrics_experience/server/routes/dimensions/get_dimentions.ts

+        dim: dimensions[0],
+      }),
+      sort('??dim', { dim: dimensions[0] }),
+      limit(1000)


Should we increase the number of items returned?

I don't know, it's what I saw on your POC, should I keep it to 20 then?

...platform/packages/shared/kbn-unified-metrics-grid/src/common/utils/esql/create_esql_query.ts

MiriamAparicio · 2025-11-11T10:46:29Z

@jennypavlova I couldn't reproduce the error, but I will check as @crespocarlos mention to add it also for the IN clause

src/platform/plugins/shared/metrics_experience/server/routes/dimensions/get_dimensions.ts

crespocarlos

LGTM

…ons-to-string

crespocarlos · 2025-11-12T09:29:28Z

src/platform/plugins/shared/metrics_experience/server/routes/dimensions/get_dimensions.test.ts

@@ -0,0 +1,307 @@
+/*


We already have a functional test covering this, btw.

yes, the one failing 😅
I can remove delete this unit test then

crespocarlos · 2025-11-12T09:35:23Z

src/platform/plugins/shared/metrics_experience/server/routes/dimensions/get_dimensions.ts

+    const response = await esClient.esql(
+      'get_dimensions',
+      {
+        query: query.toString(),


we already call toString() on line 47. Do we need this one too?

MiriamAparicio · 2025-11-12T14:55:36Z

Thanks @crespocarlos for the fix, I tested it and run the api test and works as expected

elasticmachine · 2025-11-12T19:22:23Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 4e59491

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #12 / Alerting alerts_as_data alerts as data flapping should drop tracked alerts early after hitting the alert limit
[job] [logs] FTR Configs #26 / Search Profiler Profile should return profile results for a valid index

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`discover`	1.2MB	1.2MB	+114.0B

History

💔 Build #360608 failed bd23142
💔 Build #360255 failed a795506
💚 Build #360179 succeeded ed9f7b2
💔 Build #360131 failed 101a7a6
💚 Build #359881 succeeded 59559f9

kibanamachine · 2025-11-13T09:36:48Z

Starting backport for target branches: 9.2

https://github.com/elastic/kibana/actions/runs/19327018702

kibanamachine · 2025-11-13T09:44:14Z

💔 All backports failed

Status	Branch	Result
❌	9.2	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 242432

Questions ?

Please refer to the Backport tool documentation

MiriamAparicio · 2025-11-14T07:53:24Z

💚 All backports created successfully

Status	Branch	Result
✅	9.2

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…o string (elastic#242432) Closes elastic#239952 ### Summary This PR resolves the data inconsistency between metric charts and the "Filter by value" dropdown, where values visible in charts were missing from the filter list. #### Problem The root cause was that the same dimension field (e.g., `attributes.cpu`) could have different data types across different indices: - `long` in one index - `keyword` in another index Elasticsearch aggregations and ES|QL queries fail when trying to process fields with inconsistent types, resulting in incomplete dimension values in the dropdown. #### Solution **1. Updated `get_dimentions.ts`:** - Migrated from Elasticsearch `_search` API with `terms` aggregations to ES|QL - Added explicit type casting to string using `::STRING` syntax - Increased dimension value limit from 20 to 1000 - Simplified response mapping logic **2. Updated `create_esql_query.ts`:** - Added type casting logic for dimensions in `WHERE` clauses - Applied the same `needsStringCasting()` check used in `CONCAT` operations - Ensures consistent string type handling for null checks BEFORE <img width="2304" height="560" alt="image" src="https://github.com/user-attachments/assets/e74c73af-838e-4282-b6f7-9778f03e562d" /> AFTER <img width="1043" height="292" alt="Screenshot 2025-11-10 at 14 37 35" src="https://github.com/user-attachments/assets/ade48ce5-1290-4fc7-afb4-57adcf3ebaf3" /> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Carlos Crespo <carloshenrique.leonelcrespo@elastic.co> (cherry picked from commit 08460f6) # Conflicts: # src/platform/packages/shared/kbn-traced-es-client/src/create_traced_es_client.ts # src/platform/plugins/shared/metrics_experience/tsconfig.json

…o string (elastic#242432) Closes elastic#239952 ### Summary This PR resolves the data inconsistency between metric charts and the "Filter by value" dropdown, where values visible in charts were missing from the filter list. #### Problem The root cause was that the same dimension field (e.g., `attributes.cpu`) could have different data types across different indices: - `long` in one index - `keyword` in another index Elasticsearch aggregations and ES|QL queries fail when trying to process fields with inconsistent types, resulting in incomplete dimension values in the dropdown. #### Solution **1. Updated `get_dimentions.ts`:** - Migrated from Elasticsearch `_search` API with `terms` aggregations to ES|QL - Added explicit type casting to string using `::STRING` syntax - Increased dimension value limit from 20 to 1000 - Simplified response mapping logic **2. Updated `create_esql_query.ts`:** - Added type casting logic for dimensions in `WHERE` clauses - Applied the same `needsStringCasting()` check used in `CONCAT` operations - Ensures consistent string type handling for null checks BEFORE <img width="2304" height="560" alt="image" src="https://github.com/user-attachments/assets/e74c73af-838e-4282-b6f7-9778f03e562d" /> AFTER <img width="1043" height="292" alt="Screenshot 2025-11-10 at 14 37 35" src="https://github.com/user-attachments/assets/ade48ce5-1290-4fc7-afb4-57adcf3ebaf3" /> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Carlos Crespo <carloshenrique.leonelcrespo@elastic.co>

Fix fields dropdown by casting dimension fields to string

20b1c22

MiriamAparicio requested a review from a team November 10, 2025 14:51

MiriamAparicio added release_note:skip Skip the PR/issue when compiling release notes backport:version Backport to applied version labels v9.2.0 v9.3.0 v9.2.1 Team:obs-exploration Observability Exploration team labels Nov 10, 2025

jennypavlova self-requested a review November 10, 2025 16:13

jennypavlova approved these changes Nov 10, 2025

View reviewed changes

src/platform/plugins/shared/metrics_experience/server/routes/dimensions/get_dimensions.ts Show resolved Hide resolved

crespocarlos reviewed Nov 11, 2025

View reviewed changes

MiriamAparicio commented Nov 11, 2025

View reviewed changes

src/platform/plugins/shared/metrics_experience/server/routes/dimensions/get_dimensions.ts Show resolved Hide resolved

MiriamAparicio added 2 commits November 11, 2025 11:19

cast key in WHERE IN clause

66ff359

add unit test for get_dimentions

f5071f6

MiriamAparicio force-pushed the 239952-cast-dimensions-to-string branch from 59559f9 to f5071f6 Compare November 11, 2025 11:20

fix file typo

138079c

crespocarlos approved these changes Nov 11, 2025

View reviewed changes

fix test

751603a

MiriamAparicio force-pushed the 239952-cast-dimensions-to-string branch from 101a7a6 to 751603a Compare November 11, 2025 12:54

MiriamAparicio added 2 commits November 11, 2025 15:26

Merge remote-tracking branch 'upstream/main' into 239952-cast-dimensi…

288d995

…ons-to-string

Merge remote-tracking branch 'upstream/main' into 239952-cast-dimensi…

b69c0e9

…ons-to-string

MiriamAparicio force-pushed the 239952-cast-dimensions-to-string branch from ed9f7b2 to b69c0e9 Compare November 11, 2025 15:28

Changes from node scripts/lint_ts_projects --fix

a795506

crespocarlos reviewed Nov 12, 2025

View reviewed changes

fix: esql function

c9bfafa

crespocarlos requested a review from a team as a code owner November 12, 2025 14:36

remove unit test, no needed we have functional test for it

0ec223b

MiriamAparicio and others added 2 commits November 12, 2025 14:58

remove redundant toString

bd23142

Merge branch 'main' into 239952-cast-dimensions-to-string

4e59491

MiriamAparicio merged commit 08460f6 into elastic:main Nov 13, 2025
12 checks passed

MiriamAparicio mentioned this pull request Nov 14, 2025

[9.2] [Metrics][Discover] Fix fields dropdown by casting dimension fields to string (#242432) #242999

Closed

MiriamAparicio added backport:skip This PR does not require backporting and removed backport:version Backport to applied version labels v9.2.0 v9.2.1 labels Nov 14, 2025

Conversation

MiriamAparicio commented Nov 10, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Solution

Uh oh!

jennypavlova left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

crespocarlos Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

crespocarlos Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

MiriamAparicio Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MiriamAparicio commented Nov 11, 2025

Uh oh!

Uh oh!

crespocarlos left a comment

Choose a reason for hiding this comment

Uh oh!

crespocarlos Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

MiriamAparicio Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

crespocarlos Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

MiriamAparicio commented Nov 12, 2025

Uh oh!

elasticmachine commented Nov 12, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Async chunks

History

Uh oh!

Uh oh!

kibanamachine commented Nov 13, 2025

Uh oh!

kibanamachine commented Nov 13, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

MiriamAparicio commented Nov 14, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

MiriamAparicio commented Nov 10, 2025 •

edited by kibanamachine

Loading