*: enhance mem tracker exclude rss_file & check limit for columnar

[yongman]

What problem does this PR solve?

Issue Number: ref #10844

Problem Summary:

1. Exclude rss_file result to query failed with memory exceeded exception.
2. For columnar read path, the memory usage can not counted for columnar hub in Rust.

What is changed and how it works?

1. Exclude rss_file part when check the memory limitation. rss_file part can be auto freed by OS.
2. Actively check the memory limitation after read block from columnar hub in Rust to avoid OOM.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

Summary by CodeRabbit

• Bug Fixes

  • More accurate memory-limit enforcement by excluding file-backed RSS and using an effective RSS measure.
  • Improved process memory monitoring in background collection to capture additional metrics.
  • Ensured jemalloc stats are refreshed before sampling for more consistent metrics.

• Chores

  • Added runtime RSS checks in storage read paths to better enforce limits.
  • Updated testing setting description to reference effective RSS.

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign ichn-hu, likidu for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS
• dbms/src/Interpreters/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: f28c017e-0109-4395-9537-33f9c9dedae9

📥 Commits

Reviewing files that changed from the base of the PR and between 6b4336a and dd94f94.

📒 Files selected for processing (2)

• dbms/src/Common/MemoryTracker.cpp
• dbms/src/Storages/StorageDisaggregatedColumnar.cpp

---

📝 Walkthrough

Walkthrough

The PR separates file-backed RSS from anonymous RSS, records rss_file via process metrics, computes effective RSS = max(real_rss - real_rss_file, 0), centralizes RSS-limit checks, integrates the check into allocation and storage read paths, and advances jemalloc epoch before metrics collection.

Changes

RSS-aware memory tracking and limit enforcement

Layer / File(s)	Summary
RSS file-backed metrics data model dbms/src/Common/MemoryTracker.h, dbms/src/Common/MemoryTracker.cpp	Global atomic real_rss_file is added alongside real_rss to separately track RSS from memory-mapped files.
Process metrics collection from background task dbms/src/Common/BackgroundTask.cpp	Background memory collection loop now calls both get_process_mem_usage() for RSS and get_process_metrics() for file-backed RSS metrics, updating real_rss and real_rss_file atomics.
Allocation path: rollback, effective_rss, fault checks dbms/src/Common/MemoryTracker.cpp	Allocation-time logic refactored to use a rollback lambda; fault-injection accuracy checks compute effective_rss = max(real_rss - real_rss_file, 0) and report rss_file/total rss.
RSS limit checking methods and declarations dbms/src/Common/MemoryTracker.h, dbms/src/Common/MemoryTracker.cpp	New private checkRssLimitImpl(...) centralizes RSS validation and throwing; public MemoryTracker::checkRssLimit() and CurrentMemoryTracker::checkRssLimit() forward the check to the current tracker.
RSS validation in storage read operations dbms/src/Storages/StorageDisaggregatedColumnar.cpp	Storage layer adds MemoryTracker include and calls CurrentMemoryTracker::checkRssLimit() immediately after proxy block read to enforce RSS limits in I/O paths.
Metrics collection and test settings updates dbms/src/Interpreters/AsynchronousMetrics.cpp, dbms/src/Interpreters/Settings.h	JEMalloc stats epoch is advanced before metric collection; memory_tracker_accuracy_diff_for_test setting description now references effective RSS (excluding file-backed pages).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested labels

approved, lgtm

Suggested reviewers

• CalvinNeo
• JinheLin

Poem

🐰 I hop where pages come and go,
Counting bytes both high and low,
I split the RSS into two,
Track the file-backed and the true,
So memory limits stay in tow!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 17.65% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main changes: excluding rss_file from memory tracking and adding memory limit checks for columnar operations.
Description check	✅ Passed	The description includes the required sections with specific problem statements and implementation details, though the commit message section is empty.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[pantheon-ai]

@yongman I've received your pull request and will start the review. I'll conduct a thorough review covering code quality, potential issues, and implementation details.

⏳ This process typically takes 10-30 minutes depending on the complexity of the changes.

_{ℹ️ Learn more details on Pantheon AI.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@dbms/src/Common/MemoryTracker.cpp`:
- Line 190: The call to checkRssLimitImpl(...) (e.g., checkRssLimitImpl(/*
require_tracked_growth */ true, will_be, size)) can throw after tracked bytes
were already incremented, so you must roll back the tracked increase on
exception; wrap the checkRssLimitImpl call in a try/catch and on catch subtract
the previously added amount (e.g., call the corresponding decrease/undo method
or subtract from tracked_bytes by size) before rethrowing; apply the same
rollback pattern to the other allocation paths referenced (the block around
lines 235–290) so any early abort always undoes the tracked increment.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: fd99563b-e41a-4731-b09c-3a308a1afd56

📥 Commits

Reviewing files that changed from the base of the PR and between f0766c2 and a60789f.

📒 Files selected for processing (6)

• dbms/src/Common/BackgroundTask.cpp
• dbms/src/Common/MemoryTracker.cpp
• dbms/src/Common/MemoryTracker.h
• dbms/src/Interpreters/AsynchronousMetrics.cpp
• dbms/src/Interpreters/Settings.h
• dbms/src/Storages/StorageDisaggregatedColumnar.cpp

[ti-chi-bot]

@yongman: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-sanitizer-asan	dd94f94	link	false	/test pull-sanitizer-asan

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.