Skip to content

support remote scan next batch#149

Merged
lokax merged 17 commits into
mainfrom
remote-scan-next-batch-cc-new
Nov 3, 2025
Merged

support remote scan next batch#149
lokax merged 17 commits into
mainfrom
remote-scan-next-batch-cc-new

Conversation

@lokax

@lokax lokax commented Sep 26, 2025

Copy link
Copy Markdown
Collaborator

Here are some reminders before you submit the pull request

  • Add tests for the change
  • Document changes
  • Reference the link of issue using fixes eloqdb/tx_service#issue_id
  • Reference the link of RFC if exists
  • Pass ./mtr --suite=mono_main,mono_multi,mono_basic

Summary by CodeRabbit

  • Refactor

    • Improved scan operation infrastructure with enhanced progress tracking and bucket-level management.
    • Optimized local and remote scanning pathways for better distinction and efficiency.
    • Restructured cache and scanner implementations for improved memory handling and performance.
  • Chores

    • Updated internal worker pool API signatures to support enhanced task processing.

@coderabbitai

coderabbitai Bot commented Sep 26, 2025

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

This pull request refactors the distributed scanning subsystem by removing local-path batch scanning APIs, restructuring scan progress tracking with explicit bucket-level state, adding direct execution paths via Execute(CcShard&) overrides, and transitioning from pointer-based scan caches to new proto-based ShardCacheMsg structures with improved locality tracking.

Changes

Cohort / File(s) Summary
Local scanning API removal
include/cc/cc_handler.h, include/cc/local_cc_handler.h
Removed ScanNextBatchLocal method declarations from both interfaces, eliminating the dedicated local batch-scanning entry point.
Execute overrides for direct execution
include/cc/cc_request.h, include/remote/remote_cc_request.h
Added Execute(CcShard&) override to ScanNextBatchCc and RemoteScanNextBatch, enabling direct CcShard-based execution with term validation and cc map delegation.
Bucket fetch locality tracking
include/cc/cc_req_misc.h, src/cc/cc_req_misc.cpp
Added is_local boolean parameter to FetchBucketDataCc::Reset to distinguish local vs. remote bucket fetches; updated error handling to branch on locality.
Scan progress proto restructuring
include/proto/cc_request.proto
Introduced RemoteTxKey, ShardCacheMsg, ShardCacheMsgMap, BucketScanProgressMsg, BucketScanProgressMap, BucketScanInfoMsg messages; refactored ScanNextRequest/Response to use explicit progress tracking and shard cache maps instead of flat scan cache.
Remote handler signature overhaul
include/remote/remote_cc_handler.h, src/remote/remote_cc_handler.cpp
Updated RemoteCcHandler::ScanNext to accept explicit table_name, start_key, end_key, bucket_ids, bucket_scan_progress parameters, replacing ScanCache* parameter.
Remote scan cache refactoring
include/tx_operation_result.h
Replaced RemoteScanCache constructor and members to use ShardCacheMsg* and per-bucket kv_scan_cache structures; introduced Merge(), RemoveLast(), and capacity-based methods.
Scan state and blocking info
include/remote/remote_cc_request.h, src/remote/remote_cc_request.cpp
Added comprehensive key, blocking, and per-core state accessors to RemoteScanNextBatch (e.g., StartKeyType(), EndKeyStr(), BlockingCceLockAddr(), WaitForFetchBucketCnt()); introduced ScanBlockingInfo struct for per-core lock tracking.
Cache structure refactoring
include/cc/ccm_scanner.h
Renamed index_chains_ to shard_caches_, restructured ScanCache with capacity_ tracking, changed key deserialization to packed form (SetPackedKey), introduced Iterator support for incremental cache traversal, renamed Merge() to MergeCompoundIndex().
Pause position reference semantics
include/tx_operation_result.h, include/tx_request.h
Changed BucketScanPlan to accept pause_position by reference instead of pointer; updated null checks to empty() and loop iteration accordingly.
DataStore scanning interface removal
include/store/data_store_handler.h, include/store/int_mem_store.h
Removed ScanForward virtual method from DataStoreHandler and IntMemoryStore classes.
CcShard FetchBucketData locality
include/cc/cc_shard.h, src/cc/cc_shard.cpp
Added is_local parameter to FetchBucketData; branched requester casting and wait-counter operations on locality flag.
Local handler scanning consolidation
src/cc/local_cc_handler.cpp
Removed ScanNextBatchLocal implementation; consolidated local and remote scan handling into single ScanNextBatch flow using BucketScanPlan-driven progress; simplified ScanOpen to mark finished immediately.
Local scan cache propagation
include/tx_operation.h
Added assignment of ccm_scanner_ to both slice_hd_result_ and hd_result_ in ScanNextOperation::UpdateScanState.
Remote stream response parsing
src/remote/cc_stream_receiver.cpp
Overhauled ScanNextResponse handling to populate ShardCacheMap from message; updated progress tracking and bucket-level cache merging; added assertions for unimplemented ScanOpenRequest/Response.
Tx execution scan integration
src/tx_execution.cpp
Changed partition type detection from IsHashPartitioned() to scanner type check; fixed function name ToTalRangeBuckets()TotalRangeBuckets(); added data store search condition gating and scanner metadata propagation.
Worker pool task signature
include/tx_worker_pool.h, src/tx_worker_pool.cpp
Updated BulkSubmitWork to accept functions taking size_t worker-id parameter instead of no-argument functions.
Miscellaneous
include/cc/cc_shard.h (rename), src/cc/cc_entry.cpp (formatting)
Renamed ToTalRangeBuckets()TotalRangeBuckets() in Sharder; added blank line in LruEntry::ClearLocks.

Sequence Diagram(s)

sequenceDiagram
    actor Client
    participant LocalCcHandler
    participant BucketScanPlan
    participant CcShard
    participant FetchBucketDataCc
    participant CcMap
    
    rect rgb(200, 220, 255)
    Note over Client,CcMap: New unified ScanNextBatch flow (local path)
    Client->>LocalCcHandler: ScanNextBatch(..., is_local=true)
    LocalCcHandler->>LocalCcHandler: Check leadership, determine is_local
    LocalCcHandler->>BucketScanPlan: new BucketScanPlan(pause_position, ...)
    BucketScanPlan->>BucketScanPlan: Initialize per-bucket progress
    LocalCcHandler->>CcShard: For each bucket, FetchBucketData(is_local=true, ...)
    CcShard->>CcShard: Cast requester to ScanNextBatchCc
    CcShard->>CcShard: IncreaseWaitForFetchBucketCnt(core_id_)
    CcShard->>FetchBucketDataCc: Reset(..., is_local=true, ...)
    FetchBucketDataCc->>CcMap: Execute (backfill)
    CcMap-->>FetchBucketDataCc: results
    rect rgb(255, 200, 200)
    Note over FetchBucketDataCc,CcMap: Error handling branches on is_local
    end
    FetchBucketDataCc->>ScanNextBatchCc: Enqueue (if is_local=true)
    end
    
    rect rgb(200, 220, 255)
    Note over Client,CcMap: New unified ScanNextBatch flow (remote path)
    Client->>LocalCcHandler: ScanNextBatch(..., is_local=false, remote_node)
    LocalCcHandler->>BucketScanPlan: new BucketScanPlan(pause_position, ...)
    LocalCcHandler->>RemoteCcHandler: ScanNext(table_name, start_key, end_key, bucket_ids, progress, ...)
    RemoteCcHandler->>RemoteCcHandler: Populate ScanNextRequest with table metadata
    RemoteCcHandler->>RemoteCcHandler: Branch on bucket_scan_progress empty/non-empty
    alt First plan (progress empty)
        RemoteCcHandler->>RemoteCcHandler: Create BucketScanInfoMsg with start_key, bucket_ids
    else Resume (progress non-empty)
        RemoteCcHandler->>RemoteCcHandler: Populate BucketScanProgressMap per-core
    end
    RemoteCcHandler->>Network: Send ScanNextRequest
    end
Loading
sequenceDiagram
    participant RemoteScanNextBatch
    participant CcShard
    participant CcMap
    
    rect rgb(200, 220, 255)
    Note over RemoteScanNextBatch,CcMap: New Execute(CcShard&) override
    RemoteScanNextBatch->>RemoteScanNextBatch: Execute(ccs) override
    RemoteScanNextBatch->>RemoteScanNextBatch: Validate term (leader vs standby)
    RemoteScanNextBatch->>RemoteScanNextBatch: GetCcm via table catalog
    alt Catalog hit
        RemoteScanNextBatch->>CcMap: GetCcmExecutor(term_check, ...)
        rect rgb(200, 255, 200)
        Note over RemoteScanNextBatch,CcMap: Delegate to cc map execution
        end
        RemoteScanNextBatch->>CcMap: Execute()
        CcMap-->>RemoteScanNextBatch: result
    else Catalog miss or drop
        rect rgb(255, 200, 200)
        Note over RemoteScanNextBatch: Return error (CATALOG_NOT_FOUND, etc.)
        end
    end
    end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Rationale: This PR spans ~25 files with substantial heterogeneous changes across the scanning subsystem—removing legacy APIs, introducing new proto message structures, refactoring cache internals with renamed containers and new iterator support, transitioning remote scan tracking from pointers to explicit bucket progress, and adding dual local/remote logic branches throughout. The interconnected nature of scanner, handler, request, and proto changes, combined with moderate logic density in Execute overrides and control flow updates, necessitates careful cross-file reasoning.

Possibly related PRs

  • Update txworker pool definition #153: Updates TxWorkerPool::BulkSubmitWork to accept std::function<void(size_t)> tasks (worker-id callbacks), directly matching the signature change in this PR's worker pool updates.
  • Refactor scan next batch for hash partition #131: Modifies the same scan/FetchBucketData/ScanNextBatch code paths (CcHandler, LocalCcHandler, CcShard, remote request/response handling), suggesting a related scanning subsystem refactoring effort.

Poem

🐰 From local paths we bid adieu,
Bucket progress tracked anew,
Shard caches dance with reference grace,
Execute now claims its place,
Scanning unified, remote and near!

Pre-merge checks and finishing touches

❌ Failed checks (2 warnings)
Check name Status Explanation Resolution
Description Check ⚠️ Warning The PR description is completely empty, containing only the template checklist with no actual content added by the author. All checkboxes remain unchecked and there is no substantive description of what the PR accomplishes, why the changes are necessary, whether tests have been added, documentation has been updated, or which issue/RFC this addresses. This is a template-only submission with no value-adding information for reviewers or future reference. Add a meaningful PR description that explains the objectives of these changes. At minimum, describe what problem is being solved, summarize the approach taken, indicate the testing status, confirm whether documentation has been updated, and provide links to relevant issues or RFCs using the format fixes eloqdb/tx_service#issue_id.
Docstring Coverage ⚠️ Warning Docstring coverage is 4.50% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (1 passed)
Check name Status Explanation
Title Check ✅ Passed The PR title "support remote scan next batch" is directly related to the main changes in the pull request. The changeset includes significant work on remote scanning capabilities, particularly evident in the enhanced RemoteCcHandler and RemoteScanNextBatch implementations, new proto message types for distributed scanning, and modifications to support bucket-based progress tracking for remote operations. While the PR also includes substantial refactoring of the local scanning path and consolidation of local/remote logic through a unified plan-based approach, the title accurately captures a primary and substantial aspect of these changes.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@lokax lokax linked an issue Sep 26, 2025 that may be closed by this pull request
@lokax lokax force-pushed the remote-scan-next-batch-cc-new branch from 7c7076b to 14ca731 Compare September 26, 2025 10:25
@lokax lokax force-pushed the remote-scan-next-batch-cc-new branch from 14ca731 to b13bb2b Compare October 10, 2025 02:54
@lokax lokax force-pushed the remote-scan-next-batch-cc-new branch from 7952363 to 6247426 Compare October 13, 2025 07:15
@liunyl

liunyl commented Oct 20, 2025

Copy link
Copy Markdown
Contributor

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 20, 2025

Copy link
Copy Markdown
✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 15

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (7)
include/cc/ccm_scanner.h (2)

655-667: Overzealous assert when KV cache missing

Asserting DataStoreHandler==nullptr on a missing KV cache is brittle. Returning nullptr is sufficient; let callers branch on it.

Apply:

-            else
-            {
-                assert(Sharder::Instance().GetDataStoreHandler() == nullptr);
-                return nullptr;
-            }
+            else
+            {
+                return nullptr;
+            }

309-316: Critical behavior mismatch: RangePartitionedCcmScanner uses SetPackedKey but DecodeKey expects Deserialize

The code at line 309 uses SetPackedKey in the generic TemplateScanCache::AddScanTuple method, which is shared by both hash and range partitions. However, RangePartitionedCcmScanner::DecodeKey (line 1325) uses Deserialize to reconstruct keys, creating a contract violation.

  • HashPartitionCcScanner: SetPackedKey in DecodeKey (line 909) + SetPackedKey in AddScanTuple (line 309) ✓ consistent
  • RangePartitionedCcmScanner: Deserialize in DecodeKey (line 1329) + SetPackedKey in AddScanTuple (line 309) ✗ inconsistent

Range-partitioned keys stored as packed via SetPackedKey cannot be properly deserialized later. RangePartitionedCcmScanner must either:

  1. Override AddScanTuple to use Deserialize for range-partitioned keys, or
  2. Store keys differently and adjust DecodeKey accordingly

Verify producers provide appropriately formatted keys (packed for hash, serialized for range) and that all code paths respect this contract.

src/remote/cc_stream_receiver.cpp (1)

1053-1063: Do not crash on ScanOpenResponse; handle gracefully.

Same availability risk: replace assert with logging + error propagation to the waiting handler to avoid remote hangs.

-        assert(false && "Unimplemented");
+        LOG(ERROR) << "ScanOpenResponse is unsupported; dropping.";
+        msg_pool_.enqueue(std::move(msg));
src/cc/cc_shard.cpp (1)

1596-1614: Unsafe static_cast downcasts confirmed; fix required.

The review is correct. Your code at lines 1624-1637 uses unsafe static_cast<ScanNextBatchCc *> and static_cast<remote::RemoteScanNextBatch *> to downcast from CcRequestBase *requester. Since CcRequestBase is polymorphic (has a virtual destructor), these downcasts silently succeed at compile time but have no runtime type checking. If an unexpected type reaches this code, you'll get undefined behavior.

The issue is confirmed:

  • CcRequestBase (include/cc/cc_req_base.h:37) is polymorphic with virtual ~CcRequestBase()
  • Both ScanNextBatchCc and RemoteScanNextBatch inherit from CcRequestBase via TemplatedCcRequest
  • IncreaseWaitForFetchBucketCnt() is non-virtual in both derived classes, not declared in the base

Either: (1) use dynamic_cast + runtime assertions to catch contract violations in debug builds, or (2) add a virtual IncreaseWaitForFetchBucketCnt() to CcRequestBase so the call can be made polymorphically without downcasting. Option 2 is preferred to avoid RTTI overhead.

src/tx_execution.cpp (1)

2553-2608: Fix infinite loop when read_local_ is enabled

Filtering out non-local buckets means seen_bucket_cnt never reaches Sharder::Instance().TotalRangeBuckets(), so the while (seen_bucket_cnt < total_bucket_cnt) loop spins forever. This wedges every read-local hash scan. Recompute total_bucket_cnt from the buckets that actually remain in grouped before entering the loop.

-    size_t plan_bucket_cnt_soft_limit = 256;
-    size_t total_bucket_cnt = Sharder::Instance().TotalRangeBuckets();
+    size_t plan_bucket_cnt_soft_limit = 256;
+    size_t total_bucket_cnt = 0;
+    for (const auto &[ignored_ng, core_map] : grouped)
+    {
+        for (const auto &[ignored_core_idx, bucket_vec] : core_map)
+        {
+            total_bucket_cnt += bucket_vec.size();
+        }
+    }
src/cc/local_cc_handler.cpp (1)

1087-1156: Guard against null Scan plan before dereferencing.

You dereference hd_res.Value().current_scan_plan_ without a null check. If a caller forgets to attach a plan, this will crash before the assert on Buckets(). Add a fast guard and surface a typed error.

Apply this diff:

-    BucketScanPlan *plan = hd_res.Value().current_scan_plan_;
+    BucketScanPlan *plan = hd_res.Value().current_scan_plan_;
+    if (UNLIKELY(plan == nullptr)) {
+        hd_res.SetError(CcErrorCode::INVALID_SCAN_PLAN);
+        return;
+    }

Also consider converting the assert on plan->Buckets().count(node_group_id) into an error at runtime in release builds.

include/remote/remote_cc_handler.h (1)

162-187: Add direct include for absl::flat_hash_map to avoid fragile transitive dependency.

The file uses absl::flat_hash_map in the public ScanNext signature but relies on a transitive include from tx_operation_result.h. Direct includes should be explicit for all types used in public interfaces. BucketScanProgress visibility is not a concern since tx_operation_result.h is already included.

Apply this diff:

 #include <memory>
 #include <vector>
+#include "absl/container/flat_hash_map.h"
🧹 Nitpick comments (14)
src/tx_worker_pool.cpp (1)

91-91: Consider accepting the parameter by rvalue reference for efficiency.

The work parameter is passed by value, which copies the entire vector and all contained function objects. Consider using std::vector<std::function<void(size_t)>>&& work to enable move semantics and avoid unnecessary allocations.

Apply this diff:

-void TxWorkerPool::BulkSubmitWork(std::vector<std::function<void(size_t)>> work)
+void TxWorkerPool::BulkSubmitWork(std::vector<std::function<void(size_t)>>&& work)
include/tx_worker_pool.h (1)

46-46: Consider accepting the parameter by rvalue reference for efficiency.

The work parameter is passed by value, which copies the entire vector and all contained function objects. Consider using std::vector<std::function<void(size_t)>>&& work to enable move semantics and avoid unnecessary allocations.

Apply this diff:

-    void BulkSubmitWork(std::vector<std::function<void(size_t)>> work);
+    void BulkSubmitWork(std::vector<std::function<void(size_t)>>&& work);
include/tx_operation_result.h (2)

246-252: Guard against overfilling memory_cache_hash_codes_

Add a capacity check to keep hash codes aligned with memory tuples.

Apply this diff:

 void AddHashCode(size_t hash_code)
 {
-    memory_cache_hash_codes_.push_back(hash_code);
+    if (memory_cache_hash_codes_.size() < capacity_) {
+        memory_cache_hash_codes_.push_back(hash_code);
+    } else {
+        assert(false && "memory_cache_hash_codes_ exceeds capacity");
+    }
     assert(memory_cache_hash_codes_.size() ==
            static_cast<size_t>(
                shard_cache_->memory_scan_cache().scan_tuple_size()));
 }

793-821: pause_position handling: copy vs reference semantics

Resuming by copying BucketScanProgress entries is fine, but confirm the caller supplies a fully-populated map (all node groups) and accepts the copy cost. If large, consider passing a const reference and moving selectively.

include/cc/ccm_scanner.h (2)

602-617: Rename ToTalSize() -> TotalSize() (and call site)

Nit but improves readability; also mark const.

Apply:

-        size_t ToTalSize()
+        size_t TotalSize() const
         {
             size_t size = 0;
             if (memory_cache_)
             {
                 size += memory_cache_->Size();
             }
             for (const auto &[bucket_id, kv_cache] : kv_caches_)
             {
                 size += kv_cache->Size();
             }
             return size;
         }

And:

-            shard_code_and_sizes->emplace_back(shard_code, cache->ToTalSize());
+            shard_code_and_sizes->emplace_back(shard_code, cache->TotalSize());

Also applies to: 880-881


1178-1179: Remove unused member cache_offset_

Not referenced; drop it.

Apply:

-    size_t cache_offset_{0};
include/sharder.h (1)

230-234: Rename to TotalRangeBuckets is fine

Public API rename is OK. If external modules still call the old name, consider a temporary inline alias to ease migration.

include/cc/cc_req_misc.h (1)

614-631: Name the locality explicitly; consider an enum instead of a bool.

Passing a bare bool is_local obscures call intent. Prefer an enum (e.g., ScanOrigin::Local/Remote) or at least document it in the signature.

Apply if you opt for an enum:

-               CcShard *ccs,
-               bool is_local,
+               CcShard *ccs,
+               ScanOrigin origin,

And map to a bool internally if needed.

src/cc/cc_req_misc.cpp (2)

905-947: Add defensive checks for requester type vs is_local_.

Reset() stores is_local_ and later statically casts requester_ based on it. If a mismatch ever occurs, this is UB.

Apply this diff to assert intent:

 void FetchBucketDataCc::Reset(
   ...
-  ccs_ = ccs;
-  is_local_ = is_local;
+  ccs_ = ccs;
+  is_local_ = is_local;
+  // Defensive: ensure requester matches locality to prevent invalid casts later.
+  if (is_local_) {
+    assert(dynamic_cast<ScanNextBatchCc*>(requester) != nullptr &&
+           "requester must be ScanNextBatchCc when is_local=true");
+  } else {
+    assert(dynamic_cast<remote::RemoteScanNextBatch*>(requester) != nullptr &&
+           "requester must be RemoteScanNextBatch when is_local=false");
+  }

If RTTI is disabled, consider a small enum tag on the requester to avoid RTTI.


971-1009: Error path looks correct; minor nit on naming and visibility.

DecreaseWaitForFetchBucketCnt/IsWaitForFetchBucket/WaitForFetchBucketCnt reads well. Consider making these clearly atomic (or document thread-affinity) to avoid confusion.

No code change required; add a short comment on thread ownership or atomicity of these counters.

src/remote/remote_cc_request.cpp (2)

1169-1210: Accessor robustness: avoid .at(core_id) on progress map.

StartKeyType() and StartKeyStr() use .at(core_id) which throws on missing key. Prefer find() and fall back to global_info when absent.

Apply this pattern:

-    start_key_case = scan_next.progress()
-                         .progress()
-                         .at(core_id)
-                         .start_key()
-                         .inner_key_case();
+    const auto& prog_map = scan_next.progress().progress();
+    auto it = prog_map.find(core_id);
+    if (it != prog_map.end()) {
+      start_key_case = it->second.start_key().inner_key_case();
+    } else {
+      // Fallback: use global_info if present, else treat as -inf
+      start_key_case = scan_next.has_global_info()
+        ? scan_next.global_info().start_key().inner_key_case()
+        : RemoteTxKey::InnerKeyCase::kNegInf;
+    }

Mirror the same for StartKeyStr().


1047-1168: Reset: good coverage of new fields; minor nits.

  • Good: carry term, node_group_id, table metadata, ckpt flag, and build shard cache map per core.
  • Nit: The assert(scan_next.direction()) will fail once backward scans are supported; convert to a runtime default (e.g., default to forward if field missing/false) until direction is fully wired.
include/remote/remote_cc_request.h (2)

503-516: Tighten atomic memory ordering for error path.

SetErrorCode()/ErrorCode() use relaxed ordering. Given finalization uses unfinished_core_cnt_ with acq_rel, pair the final read with acquire (see SetFinish fix). For SetErrorCode, relaxed is fine, but document “first-wins” semantics.

Optionally add comments:

-        err_.compare_exchange_strong(expected,
-                                     err_code,
-                                     std::memory_order_relaxed,
-                                     std::memory_order_relaxed);
+        // First error wins; relaxed is sufficient since SetFinish does an acquire load.
+        err_.compare_exchange_strong(expected, err_code,
+                                     std::memory_order_relaxed,
+                                     std::memory_order_relaxed);

And prefer acquire in ErrorCode() when used outside SetFinish:

-        return err_.load(std::memory_order_relaxed);
+        return err_.load(std::memory_order_acquire);

577-586: Const-correctness for accessors.

Make pure getters const to improve API clarity and thread-safety expectations.

Apply this diff:

-int64_t NodeGroupTerm()
+int64_t NodeGroupTerm() const
 {
     return ng_term_;
 }
-const TxKey *EndKey()
+const TxKey *EndKey() const
 {
     return &end_key_;
 }
-    CcErrorCode ErrorCode()
+    CcErrorCode ErrorCode() const
     {
-        return err_.load(std::memory_order_relaxed);
+        return err_.load(std::memory_order_acquire);
     }

Optionally:

-    bool IsWaitForFetchBucket(uint16_t core_id)
+    bool IsWaitForFetchBucket(uint16_t core_id) const
     { /* unchanged body using at(core_id) */ }

-    bool IsRequireRecs()
+    bool IsRequireRecs() const
     { return is_require_recs_; }

Also applies to: 512-516, 489-493, 641-646, 701-704

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between f02996e and 9fdca31.

📒 Files selected for processing (25)
  • include/cc/cc_handler.h (0 hunks)
  • include/cc/cc_req_misc.h (2 hunks)
  • include/cc/cc_request.h (2 hunks)
  • include/cc/cc_shard.h (1 hunks)
  • include/cc/ccm_scanner.h (27 hunks)
  • include/cc/local_cc_handler.h (0 hunks)
  • include/proto/cc_request.proto (3 hunks)
  • include/remote/remote_cc_handler.h (1 hunks)
  • include/remote/remote_cc_request.h (3 hunks)
  • include/sharder.h (1 hunks)
  • include/store/data_store_handler.h (0 hunks)
  • include/store/int_mem_store.h (0 hunks)
  • include/tx_operation.h (1 hunks)
  • include/tx_operation_result.h (6 hunks)
  • include/tx_request.h (1 hunks)
  • include/tx_worker_pool.h (1 hunks)
  • src/cc/cc_entry.cpp (1 hunks)
  • src/cc/cc_req_misc.cpp (3 hunks)
  • src/cc/cc_shard.cpp (3 hunks)
  • src/cc/local_cc_handler.cpp (3 hunks)
  • src/remote/cc_stream_receiver.cpp (3 hunks)
  • src/remote/remote_cc_handler.cpp (2 hunks)
  • src/remote/remote_cc_request.cpp (9 hunks)
  • src/tx_execution.cpp (7 hunks)
  • src/tx_worker_pool.cpp (1 hunks)
💤 Files with no reviewable changes (4)
  • include/cc/cc_handler.h
  • include/cc/local_cc_handler.h
  • include/store/data_store_handler.h
  • include/store/int_mem_store.h
🔇 Additional comments (14)
src/cc/cc_entry.cpp (1)

163-169: Cosmetic formatting improvement in ClearLocks.

The blank line insertion between read intent deletion and queued request cleanup improves readability by visually grouping related lock cleanup operations. No functional changes; the logic and control flow remain intact.

include/tx_request.h (1)

400-402: Remove the verification concern — no dangling reference issue exists.

The BucketScanPlan constructor receives pause_position as a reference parameter but does not store it. Instead, the constructor iterates through pause_position during initialization and copies/moves its data into the owned member variable current_position_. The pause_position reference is scoped to the constructor and is not retained, so there is no dangling reference risk.

include/tx_operation_result.h (3)

217-227: RemoteScanCache: constructors/capacity init look good

Ctor + reserve are consistent with default capacity; no issues.


234-239: Capacity/Full and memory cache accessor are consistent

Tuple-count-based capacity checks and mutable accessor look fine.

Also applies to: 241-245


299-327: min_key overwrite bug confirmed—applies to both memory and KV cache iteration

The bug is real: min_key is overwritten per bucket in the loop (lines 318-328) without comparison, selecting the last-iterated key instead of the actual minimum. The suggested fix correctly adds if (min_key == nullptr || last_tuple.key() < *min_key) to perform proper comparison.

The "Also applies to: 341-354" refers to usage lines within the same method where the incorrect min_key is consumed by RemoveLast() calls.

The kv_is_drained pre-population concern is also valid—the code uses .at(bucket_id) at lines 319 and 321, which throws if the key doesn't exist. The caller must initialize this map with all bucket IDs before invoking Merge(). However, the exact initialization pattern in calling code could not be located in the codebase search results.

include/cc/ccm_scanner.h (3)

74-84: ScanCache capacity_ adoption is correct

Capacity now consistently gates Status/Reset/Full. Good cleanup.

Also applies to: 105-106, 124-125, 141-143


906-911: DecodeKey uses SetPackedKey — matches AddScanTuple change

Consistent with packed-key path. See verification script above.


913-941: ShardCache iterator driving Current/MoveNext looks solid

Init skips empties; MoveNext walks memory then KV per shard then advances shards. LGTM.

Confirm no concurrent mutation of shard_caches_ while iterating.

Also applies to: 950-991

include/tx_operation.h (1)

496-501: Propagating ccm_scanner_ into hd_result_ is correct

Ensures downstream users can read scanner state without relying on slice_hd_result_. Good.

include/cc/cc_shard.h (1)

743-761: All call sites correctly updated; is_local parameter properly integrated throughout

Verification confirms: The two main FetchBucketData call sites (template_cc_map.h lines 2389, 3035) explicitly pass is_local (true and false respectively). The FetchBucketDataCc struct stores the flag in is_local_ member field, Reset() correctly assigns it, and the simpler overload carries consistency through the struct. No missing updates detected.

src/cc/local_cc_handler.cpp (2)

954-957: Good: make ScanOpen non-blocking and defer actual work to ScanNext.

Setting finished here simplifies the state machine and avoids early scanning side effects.


1159-1184: Review comment is accurate; all proposed changes are correct and necessary.

Verification confirms:

  • First ScanNext overload (line 162) lacks direction parameter and hardcodes direction=true (line 557) ✓
  • CcScanner::Direction() method exists (include/cc/ccm_scanner.h:464) ✓
  • ScanDirection::Forward enum exists (include/cc/scan.h:34) ✓
  • Call site at line 1160 correctly should pass scanner.Direction() == ScanDirection::Forward
  • Second ScanNext overload (line 188) already correctly extracts direction from ccm_scanner_ and requires no changes ✓

All three coordinated changes in the review comment are accurate and address the bug completely for the affected overload.

include/remote/remote_cc_request.h (2)

472-473: Execute(CcShard&) override: ensure no blocking I/O and bounded work.

Given this is called on request threads, ensure it never performs blocking waits or unbounded merges. If heavy merge is needed, offload to worker/bthread and return.

Can you confirm the implementation offloads long-running steps (bucket fetch, merges) and uses timeouts for remote calls?


489-493: No compilation issues exist. ng_term_ is properly declared in TemplatedCcRequest, the base class.

RemoteScanNextBatch inherits from TemplatedCcRequest<RemoteScanNextBatch, Void>, which is where ng_term_ is declared as a protected member (include/cc/cc_request.h:335). The NodeGroupTerm() method correctly accesses this inherited member. The code compiles without issues.

Comment thread include/cc/cc_req_misc.h
Comment thread include/cc/cc_request.h
Comment thread include/remote/remote_cc_request.h
Comment thread include/remote/remote_cc_request.h
Comment thread include/remote/remote_cc_request.h
Comment thread src/remote/cc_stream_receiver.cpp Outdated
Comment on lines +1124 to +1131
// Even if ScanNext operation fail, we should also move scan
// result into read set to release acquired lock.
{
uint32_t node_group_id = scan_next_res.node_group_id();
CcScanner *scanner = hd_res->Value().ccm_scanner_;
hd_res->Value().current_scan_plan_->UpdateNodeGroupTerm(
node_group_id, scan_next_res.term());
auto *ng_bucket_scan_progress =

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Comment contradicts behavior.

The comment says to move scan result even on failure, but the code only executes on success. Align the comment or implement the intended fallback.

🤖 Prompt for AI Agents
In src/remote/cc_stream_receiver.cpp around lines 1124-1131, the inline comment
claims we should move the scan result into the read set even if ScanNext fails,
but the code only does this on success; either update the comment to reflect the
current behavior or implement the intended fallback: add an else/failure path
that still extracts node_group_id, uses hd_res->Value().ccm_scanner_ and
current_scan_plan_ to call UpdateNodeGroupTerm and move the scan result into the
read set (releasing any acquired lock), while guarding null pointers and logging
the failure case; ensure the fallback mirrors the success-path cleanup so locks
are always released.

Comment thread src/remote/cc_stream_receiver.cpp Outdated
Comment thread src/remote/remote_cc_handler.cpp Outdated
Comment thread src/remote/remote_cc_handler.cpp
Comment thread src/remote/remote_cc_request.cpp
@lokax lokax force-pushed the remote-scan-next-batch-cc-new branch from c733733 to 59b71fa Compare October 21, 2025 07:49
@lokax lokax force-pushed the remote-scan-next-batch-cc-new branch from c66c35b to c388cfe Compare October 22, 2025 03:34
@lokax lokax force-pushed the remote-scan-next-batch-cc-new branch from dd9cf54 to 8a97681 Compare October 22, 2025 09:16
@lokax lokax force-pushed the remote-scan-next-batch-cc-new branch 3 times, most recently from 01366e5 to 731cb1c Compare October 28, 2025 07:36
@lokax lokax force-pushed the remote-scan-next-batch-cc-new branch from 731cb1c to 88f8b8b Compare October 31, 2025 02:36
@lokax lokax changed the base branch from hash-part-scan to main November 3, 2025 02:33
@lokax lokax changed the base branch from main to hash-part-scan November 3, 2025 02:40
@lokax lokax changed the base branch from hash-part-scan to main November 3, 2025 02:57
@lokax lokax merged commit d7cab24 into main Nov 3, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Refactor remote scan for hash partition

2 participants