[fix](profile) Disable profile collection by default#57314
Merged
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
TPC-DS: Total hot run time: 189635 ms |
ClickBench: Total hot run time: 27.48 s |
Contributor
Author
|
run feut |
Contributor
FE Regression Coverage ReportIncrement line coverage |
2 similar comments
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
zhiqiang-hhhh
force-pushed
the
fix-profile-memory
branch
from
e4df4f2 to
0135240
Compare
Contributor
Author
|
run buildall |
TPC-DS: Total hot run time: 189691 ms |
ClickBench: Total hot run time: 27.37 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
FE UT Coverage ReportIncrement line coverage |
zhiqiang-hhhh
force-pushed
the
fix-profile-memory
branch
from
0135240 to
4ea39b4
Compare
Contributor
Author
|
run buildall |
Contributor
FE Regression Coverage ReportIncrement line coverage |
zhiqiang-hhhh
force-pushed
the
fix-profile-memory
branch
from
4ea39b4 to
7a39eae
Compare
Contributor
Author
|
run buildall |
zhiqiang-hhhh
force-pushed
the
fix-profile-memory
branch
from
7a39eae to
7f47b0a
Compare
Contributor
Author
|
run buildall |
ClickBench: Total hot run time: 28.29 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
ClickBench: Total hot run time: 28.09 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
yiguolei
approved these changes
Oct 31, 2025
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
github-actions Bot
pushed a commit
that referenced
this pull request
Oct 31, 2025
### What problem does this PR solve? 1. Memory consumption of profile is still a big problem of FE. So disable it by default. 2. For load task, we need to update counter even if profile is not enabled, since final http return msg needs it.
dwdwqfwe
pushed a commit
to dwdwqfwe/doris
that referenced
this pull request
Oct 31, 2025
### What problem does this PR solve? 1. Memory consumption of profile is still a big problem of FE. So disable it by default. 2. For load task, we need to update counter even if profile is not enabled, since final http return msg needs it.
yiguolei
pushed a commit
that referenced
this pull request
Nov 3, 2025
liaoxin01
added a commit
that referenced
this pull request
Jun 25, 2026
### What - Gate the load counter update in `Scanner::_collect_profile_before_close()` on a new virtual `_should_update_load_counters()`: - Base `Scanner` reports only when `_is_load` (classic stream/broker/routine load scanners with src tuple desc). - `FileScanner` additionally reports for `FILE_STREAM` scans: TVF based loads (`http_stream`, group commit) plan the load source as a tvf query scan without src tuple desc (`_is_load` is false), but their WHERE clause filtered rows must still be reported as `NumberUnselectedRows` / counted into `NumberTotalRows`. - Add a deterministic regression case covering INSERT-SELECT / DELETE / UPDATE whose scans filter out all rows. ### Why For DELETE/UPDATE/INSERT INTO ... SELECT executed through the insert path, rows filtered by query scan predicates (including runtime filters) were added to the RuntimeState load counters. When all scanned rows are filtered, `num_rows_load_success()` (total - filtered - unselected) goes negative, BE reports a negative `dpp.norm.ALL`, and FE fails the `insert_max_filter_ratio` check with errors like: ``` Insert has too many filtered data 0/-2 insert_max_filter_ratio is 1.000000 ``` This only triggers with `enable_insert_strict=false` and `insert_max_filter_ratio > 0` (the strict branch only checks `filteredRows > 0`). The intermittency in the field comes from runtime filter arrival timing: rows are only counted when the RF arrives in time to filter inside the scanner. This was historically gated by `if (!enable_profile && !_is_load) return;` (already buggy with `enable_profile=true`), and became unconditional after #57314 removed the early return. Compared to the previous revision of this PR (thrift `skip_query_scan_load_counters` option set by FE for DELETE/UPDATE): - No thrift / FE changes needed. - Also fixes plain INSERT INTO ... SELECT: `AbstractInsertExecutor` sets `query_type=LOAD` for all insert-path commands, so the query-type based gate still let OlapScanner predicate-filtered rows pollute the counters there. - `FILE_STREAM` is a precise discriminator: only the `http_stream` / `group_commit` TVFs use it, and they require a backend id on the ConnectContext (only present for stream-load style HTTP requests), so they can never appear in a normal query/DELETE/UPDATE. `FileScanner::_counter.num_rows_filtered` is only accumulated in `_convert_to_output_block` (load-only path), so for `http_stream` the `NumberFilteredRows` reported to clients comes from sink validation and is unaffected by this gate. ### Test - New regression case `regression-test/suites/load_p0/insert/test_scan_filtered_rows_not_pollute_load_counter.groovy`: uses value-column predicates on an AGGREGATE KEY table (cannot be pushed down to storage, evaluated by scanner conjuncts) to deterministically reproduce; before this fix the INSERT-SELECT/DELETE/UPDATE statements fail with `0/-10`. - `test_group_commit_http_stream` semantics preserved: `insert into ... select from http_stream(...) where ...` still reports `NumberUnselectedRows`/`NumberTotalRows` (the previous `_is_load`-only attempt broke this with `expected: <6> but was: <5>`).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)