functions: save FieldType as value instead of ptr in json function

[yongman]

What problem does this PR solve?

Issue Number: close #10845

Problem Summary:

When TiFlash nextgen evaluates JSON_EXTRACT on a TEXT column with IS NULL / IS NOT NULL filters, the result can be inconsistent with JSON columns.

For example, JSON_EXTRACT(action_params, '$.popup_id') IS NULL may return rows whose extracted value is actually non-null, while IS NOT NULL returns no rows.

The root cause is that the disaggregated columnar path builds temporary FilterConditions, and JSON cast functions keep raw pointers to tipb::FieldType. After the temporary object is destroyed, those pointers can become dangling, so FunctionCastStringAsJson may read invalid FieldType metadata.

What is changed and how it works?

functions: save FieldType as value instead of ptr in json function

Store TiDB FieldType metadata by value in JSON cast functions instead of keeping raw pointers to caller-owned FieldType objects.

Use std::optional<tipb::FieldType> for optional FieldType metadata and update the missing-metadata checks accordingly in:
- FunctionCastJsonAsString
- FunctionCastIntAsJson
- FunctionCastStringAsJson
- FunctionCastTimeAsJson

This avoids dangling FieldType references when JSON cast functions are created from temporary filter conditions, and keeps TEXT-to-JSON cast behavior stable for pushed-down JSON_EXTRACT filters.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Manual test:

Use the SQL in #10845 to create event_log1 with action_params TEXT and event_log2 with action_params JSON, then run with:

SET SESSION tidb_isolation_read_engines='tiflash';

Verify that both TEXT and JSON columns return consistent results:

WHERE JSON_EXTRACT(action_params, '$.popup_id') IS NULL
-- returns 0 rows

WHERE JSON_EXTRACT(action_params, '$.popup_id') IS NOT NULL
-- returns 5 rows

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

Summary by CodeRabbit

• Bug Fixes

  • More consistent JSON casting/extraction behavior when type metadata is available, improving null/value handling in JSON queries.
  • Improved application of pushed-down filters in columnar query execution, yielding more consistent filter results and operator profiling.

• Tests

  • Added fullstack regression test covering JSON extraction and null/non-null filtering with TiFlash-enabled query execution.

[pantheon-ai]

@yongman I've received your pull request and will start the review. I'll conduct a thorough review covering code quality, potential issues, and implementation details.

⏳ This process typically takes 10-30 minutes depending on the complexity of the changes.

_{ℹ️ Learn more details on Pantheon AI.}

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: e7f5a57b-68cf-438f-b5b4-dd6aa66c2b7b

📥 Commits

Reviewing files that changed from the base of the PR and between 7378edc and db8d62f.

📒 Files selected for processing (3)

• dbms/src/Storages/StorageDisaggregated.h
• dbms/src/Storages/StorageDisaggregatedColumnar.cpp
• dbms/src/Storages/StorageDisaggregatedRemote.cpp

💤 Files with no reviewable changes (1)

• dbms/src/Storages/StorageDisaggregatedRemote.cpp

---

📝 Walkthrough

Walkthrough

Refactors JSON cast functions to store TiDB FieldType in std::optional (update setters and has_value() checks), adds a fullstack regression test for json_extract on TEXT vs JSON, and introduces columnar overloads to re-apply pushed-down filters (with conditional header declarations and removal of remote overloads).

Changes

JSON FieldType Optional Refactoring

Layer / File(s)	Summary
Header include and FunctionCastJsonAsString refactoring dbms/src/Functions/FunctionsJson.h	Added <optional> header. Refactored FunctionCastJsonAsString to store tidb_tp as std::optional<tipb::FieldType>. Updated setOutputTiDBFieldType to assign into the optional, and changed execution condition from pointer checks to tidb_tp.has_value() before accessing flen().
FunctionCastIntAsJson refactoring dbms/src/Functions/FunctionsJson.h	Refactored FunctionCastIntAsJson to store input_tidb_tp as std::optional<tipb::FieldType>. Updated setInputTiDBFieldType to assign into the optional, and changed execution condition from input_tidb_tp == nullptr to !input_tidb_tp.has_value().
FunctionCastStringAsJson refactoring dbms/src/Functions/FunctionsJson.h	Refactored FunctionCastStringAsJson to store both input_tidb_tp and output_tidb_tp as std::optional<tipb::FieldType>. Updated both setters to assign into optionals. Changed execution conditions from pointer checks to has_value() checks; output_tidb_tp is dereferenced only when present before calling hasParseToJSONFlag().
FunctionCastTimeAsJson refactoring dbms/src/Functions/FunctionsJson.h	Refactored FunctionCastTimeAsJson to store input_tidb_tp as std::optional<tipb::FieldType>. Updated setInputTiDBFieldType to assign into the optional, and changed timestamp detection logic from input_tidb_tp == nullptr to !input_tidb_tp.has_value().
Fullstack regression test tests/fullstack-test/expr/cast_as_json_issue10845.test	Adds an end-to-end SQL test that sets up tables with TEXT and JSON columns, inserts sample rows, forces TiFlash/MPP execution, validates null vs non-null json_extract results and ordered json_unquote(json_extract(...)) output, then cleans up tables.

Disaggregated Storage: Reapply Pushed-Down Filters

Layer / File(s)	Summary
Header include and conditional declarations dbms/src/Storages/StorageDisaggregated.h	Adds <Common/config.h> and wraps the two filterConditionsWithPushedDownFilters method declarations in #if ENABLE_NEXT_GEN_COLUMNAR so they are only declared when the feature macro is enabled.
Columnar implementations to reapply pushed-down filters dbms/src/Storages/StorageDisaggregatedColumnar.cpp	Adds two overloads that merge table_scan.getPushedDownFilters() into FilterConditions, execute merged conditions via ::DB::executePushedDownFilter(...) when present, and record resulting operator/profile streams into the DAG context or executor path.
Remove remote overloads dbms/src/Storages/StorageDisaggregatedRemote.cpp	Removes the two previous overloads of filterConditionsWithPushedDownFilters(...) from the remote implementation.

Sequence Diagram(s)

sequenceDiagram
  participant TableScan
  participant StorageDisaggregated
  participant FilterConditions
  participant executePushedDownFilter
  participant DAGContext
  TableScan->>StorageDisaggregated: provide pushedDownFilters()
  StorageDisaggregated->>FilterConditions: merge existing filter_conditions + pushedDownFilters
  StorageDisaggregated->>executePushedDownFilter: if merged not empty, execute merged conditions
  executePushedDownFilter->>DAGContext: return operator streams / operator profile infos
  StorageDisaggregated->>DAGContext: record ProfileStreamsMap or addOperatorProfileInfos

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

🐇 I nibble bits and types so bright,

Pointers tucked away, optionals take flight,
Filters re-applied in columnar rows,
Tests hop in where the JSON wind blows,
A rabbit cheers the code tonight.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 6.90% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Out of Scope Changes check	❓ Inconclusive	While primary changes focus on the FieldType refactoring, the PR also modifies StorageDisaggregated.h/StorageDisaggregatedColumnar.cpp/.Remote.cpp by moving/conditionalizing filterConditionsWithPushedDownFilters methods, which appears related to where temporary FilterConditions are constructed but needs clarification.	Clarify whether the storage-layer changes (moving filterConditionsWithPushedDownFilters between files and adding ENABLE_NEXT_GEN_COLUMNAR guards) are necessary for the FieldType fix, or if they should be separated into a distinct PR.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'functions: save FieldType as value instead of ptr in json function' clearly and specifically describes the main change: refactoring JSON functions to store FieldType by value using optional instead of raw pointers.
Description check	✅ Passed	The PR description includes all required template sections: problem statement (issue #10845 with clear explanation), what changed with commit message, checklist with manual test documentation, and release notes.
Linked Issues check	✅ Passed	The PR implements the required changes to fix issue #10845: stores FieldType by value as std::optional instead of raw pointers in FunctionCastJsonAsString, FunctionCastIntAsJson, FunctionCastStringAsJson, and FunctionCastTimeAsJson, eliminating dangling pointer issues and ensuring consistent JSON_EXTRACT behavior on TEXT vs JSON columns.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

dbms/src/Functions/FunctionsJson.h (1)

439-439: 💤 Low value

Refactoring from pointer to value changes ownership semantics.

The change from const tipb::FieldType* to std::optional<tipb::FieldType> is semantically significant: the function now owns a copy of the FieldType rather than holding a reference to external data. This eliminates potential lifetime issues (dangling pointers), which likely addresses the consistency bug mentioned in issue #10845.

The setter copies tipb::FieldType on each call. If tipb::FieldType (a protobuf message) is large, consider adding a move-enabled overload:

void setOutputTiDBFieldType(tipb::FieldType tidb_tp_) { tidb_tp = std::move(tidb_tp_); }

However, the current implementation is correct, and the copy overhead may be acceptable.

Also applies to: 467-467, 530-530

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@dbms/src/Functions/FunctionsJson.h` at line 439, The setter currently copies
a potentially large protobuf (setOutputTiDBFieldType) which can be expensive;
add a move-enabled overload that takes tipb::FieldType by value (or an rvalue
ref) and moves it into the std::optional member (tidb_tp) to avoid the extra
copy, and apply the same change to the other setters flagged in this file (the
other setOutputTiDBFieldType occurrences referenced in the comment).

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@dbms/src/Functions/FunctionsJson.h`:
- Line 439: The setter currently copies a potentially large protobuf
(setOutputTiDBFieldType) which can be expensive; add a move-enabled overload
that takes tipb::FieldType by value (or an rvalue ref) and moves it into the
std::optional member (tidb_tp) to avoid the extra copy, and apply the same
change to the other setters flagged in this file (the other
setOutputTiDBFieldType occurrences referenced in the comment).

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: aa141322-ab1a-4623-ab1a-810339dd1046

📥 Commits

Reviewing files that changed from the base of the PR and between ed4e382 and e74426d.

📒 Files selected for processing (1)

• dbms/src/Functions/FunctionsJson.h

[JaySon-Huang]

Verified the fixed in the tiflash-cse columnar branch.

LGTM

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JaySon-Huang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [JaySon-Huang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-05-15 05:48:16.988262699 +0000 UTC m=+417465.521042028: ☑️ agreed by JaySon-Huang.

[JaySon-Huang]

/cc @windtalker @gengliqi

[windtalker]

I don't understand why "disaggregated columnar path builds temporary FilterConditions"

[yongman]

I don't understand why "disaggregated columnar path builds temporary FilterConditions"

@windtalker The related logic refer to

tiflash/dbms/src/Storages/StorageDisaggregatedRemote.cpp

Lines 126 to 160 in 6f17860

	void StorageDisaggregated::filterConditionsWithPushedDownFilters(
	DAGExpressionAnalyzer & analyzer,
	DAGPipeline & pipeline)
	{
	#if ENABLE_NEXT_GEN_COLUMNAR == 0
	filterConditions(analyzer, pipeline);
	#else
	FilterConditions conditions(filter_conditions.executor_id, filter_conditions.conditions);
	conditions.conditions.MergeFrom(table_scan.getPushedDownFilters());
	if (conditions.hasValue())
	{
	::DB::executePushedDownFilter(conditions, analyzer, log, pipeline);
	auto & profile_streams = context.getDAGContext()->getProfileStreamsMap()[conditions.executor_id];
	pipeline.transform([&profile_streams](auto & stream) { profile_streams.push_back(stream); });
	}
	#endif
	}
	void StorageDisaggregated::filterConditionsWithPushedDownFilters(
	PipelineExecutorContext & exec_context,
	PipelineExecGroupBuilder & group_builder,
	DAGExpressionAnalyzer & analyzer)
	{
	#if ENABLE_NEXT_GEN_COLUMNAR == 0
	filterConditions(exec_context, group_builder, analyzer);
	#else
	FilterConditions conditions(filter_conditions.executor_id, filter_conditions.conditions);
	conditions.conditions.MergeFrom(table_scan.getPushedDownFilters());
	if (conditions.hasValue())
	{
	::DB::executePushedDownFilter(exec_context, group_builder, conditions, analyzer, log);
	context.getDAGContext()->addOperatorProfileInfos(conditions.executor_id, group_builder.getCurProfileInfos());
	}
	#endif
	}

[windtalker]

I don't understand why "disaggregated columnar path builds temporary FilterConditions"

@windtalker The related logic refer to

tiflash/dbms/src/Storages/StorageDisaggregatedRemote.cpp

Lines 126 to 160 in 6f17860

	void StorageDisaggregated::filterConditionsWithPushedDownFilters(
	DAGExpressionAnalyzer & analyzer,
	DAGPipeline & pipeline)
	{
	#if ENABLE_NEXT_GEN_COLUMNAR == 0
	filterConditions(analyzer, pipeline);
	#else
	FilterConditions conditions(filter_conditions.executor_id, filter_conditions.conditions);
	conditions.conditions.MergeFrom(table_scan.getPushedDownFilters());
	if (conditions.hasValue())
	{
	::DB::executePushedDownFilter(conditions, analyzer, log, pipeline);
	auto & profile_streams = context.getDAGContext()->getProfileStreamsMap()[conditions.executor_id];
	pipeline.transform([&profile_streams](auto & stream) { profile_streams.push_back(stream); });
	}
	#endif
	}
	void StorageDisaggregated::filterConditionsWithPushedDownFilters(
	PipelineExecutorContext & exec_context,
	PipelineExecGroupBuilder & group_builder,
	DAGExpressionAnalyzer & analyzer)
	{
	#if ENABLE_NEXT_GEN_COLUMNAR == 0
	filterConditions(exec_context, group_builder, analyzer);
	#else
	FilterConditions conditions(filter_conditions.executor_id, filter_conditions.conditions);
	conditions.conditions.MergeFrom(table_scan.getPushedDownFilters());
	if (conditions.hasValue())
	{
	::DB::executePushedDownFilter(exec_context, group_builder, conditions, analyzer, log);
	context.getDAGContext()->addOperatorProfileInfos(conditions.executor_id, group_builder.getCurProfileInfos());
	}
	#endif
	}

can you give some explainations about why disaggregated columnar path is so special?

[JaySon-Huang]

@windtalker @yongman

The disaggregated columnar path is special because the columnar reader and the classic StorageDeltaMerge path have different contracts for late-materialization filters.

For a TiDB table scan, pushed_down_filter_conditions are the filters pushed down by late materialization and are expected to be executed in the storage layer:

/// pushed_down_filter_conditions is the filter conditions that are
/// pushed down to table scan by late materialization.
/// They will be executed on Storage layer.

In the classic StorageDeltaMerge path, these filters are passed through DAGQueryInfo and then into DM::PushDownExecutor::build(...). PushDownExecutor builds the filter expression actions from pushed_down_filters, so this path treats them as the storage-layer filter executor.

The disaggregated columnar reader is different. It receives the table scan protobuf with pushed_down_filter_conditions, but the current columnar reader contract is weaker. The code comment in StorageDisaggregatedColumnar.cpp says:

// Copy pushed down filters to filter_conditions to make filterConditions works properly.
// Proxy columnar reader use pushed down filters to reduce packs load from disk and has no
// guarantee to filter all useless data, so we rely on the filterConditions to filter data.

So, for the columnar path, table_scan.getPushedDownFilters() can be used by the proxy columnar reader for pruning / reducing pack reads, but it is not sufficient for correctness because it may still return rows that do not satisfy those filters. TiFlash therefore has to re-apply those late-materialization filters in the local pipeline.

That is why filterConditionsWithPushedDownFilters() creates a temporary FilterConditions, merges table_scan.getPushedDownFilters() into the normal selection conditions, and calls executePushedDownFilter(...) after the columnar reader returns data. This temporary merged FilterConditions is also why the dangling tipb::FieldType* issue can be exposed in this path: JSON cast functions may be built from expressions owned by that temporary object.

Code references

• dbms/src/Flash/Coprocessor/TiDBTableScan.h:68: documents that pushed_down_filter_conditions come from late materialization and should be executed on the storage layer.
• dbms/src/Storages/StorageDeltaMerge.cpp:878 and dbms/src/Storages/StorageDeltaMerge.cpp:972: classic StorageDeltaMerge builds a PushDownExecutor from the query info.
• dbms/src/Storages/DeltaMerge/Filter/PushDownExecutor.cpp:226: PushDownExecutor::build(query_info, ...) reads dag_query->pushed_down_filters.
• dbms/src/Storages/DeltaMerge/Filter/PushDownExecutor.cpp:149: PushDownExecutor builds filter expression actions from pushed_down_filters.
• dbms/src/Storages/StorageDisaggregatedColumnar.cpp:348: columnar comment states that the proxy columnar reader uses pushed-down filters to reduce pack loads but does not guarantee filtering all useless data.
• dbms/src/Storages/StorageDisaggregatedRemote.cpp:133 and dbms/src/Storages/StorageDisaggregatedRemote.cpp:152: columnar-enabled filterConditionsWithPushedDownFilters() merges table_scan.getPushedDownFilters() into a temporary FilterConditions and executes it in TiFlash.

[JaySon-Huang]

/test pull-unit-test

[ti-chi-bot]

@yongman: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-unit-test	db8d62f	link	true	/test pull-unit-test

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.