Skip to content

Add new subcommands for forest-cli snasphot#6128

Merged
elmattic merged 60 commits into
mainfrom
elmattic/snapshot-export-qol
Oct 23, 2025
Merged

Add new subcommands for forest-cli snasphot#6128
elmattic merged 60 commits into
mainfrom
elmattic/snapshot-export-qol

Conversation

@elmattic

@elmattic elmattic commented Oct 3, 2025

Copy link
Copy Markdown
Contributor

Summary of changes

Changes introduced in this pull request:

  • Add forest-cli snasphot export-status
  • Add forest-cli snasphot export-cancel
$ forest-cli snapshot export-status --wait   
[00:01:37] [####################################################>----------------] 76% Exporting

Reference issue to close (if applicable)

Closes #6082

Other information and links

Change checklist

  • I have performed a self-review of my own code,
  • I have made corresponding changes to the documentation. All new code adheres to the team's documentation standards,
  • I have added tests that prove my fix is effective or that my feature works (if possible),
  • I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

Summary by CodeRabbit

  • New Features

    • Snapshot export: status and cancel commands with optional wait, live progress display, JSON/text output, and clear completion/cancellation messages.
  • APIs

    • New structured endpoints to report export progress and request cancellation.
    • V2 API now supports basic Eth RPC methods (chain ID, protocol, syncing, accounts).
  • Tests

    • Added end-to-end export start/status/cancel orchestration and updated ignored checks.
  • Documentation

    • Updated changelog (added JWT expiration gate entry; removed prior db-import entry).

@coderabbitai

coderabbitai Bot commented Oct 3, 2025

Copy link
Copy Markdown
Contributor

Walkthrough

Adds RPC endpoints and CLI commands to monitor and cancel chain snapshot exports, introduces a shared ExportStatus and progress-tracking in the chain stream, changes export RPCs to return structured ApiExportResult (Done/Cancelled), and updates CLI, scripts, and test ignore lists.

Changes

Cohort / File(s) Summary
Changelog
CHANGELOG.md
Added unreleased entries for export-status, export-cancel, JWT expiration flag, and V2 Eth RPC method support.
RPC Types
src/rpc/types/mod.rs
Added ApiExportStatus (progress, exporting, cancelled, start_time) and ApiExportResult (Done(Option<String>), Cancelled) with lotus JSON registration; imported chrono::Utc.
RPC Methods
src/rpc/methods/chain.rs
Converted ForestChainExport/ChainExport to return ApiExportResult; added cancellation token via CHAIN_EXPORT_LOCK; integrated start_export/end_export/cancel_export; implemented cancellation-aware export flow and added ForestChainExportStatus and ForestChainExportCancel RPCs.
RPC Registration
src/rpc/mod.rs
Registered ForestChainExportStatus and ForestChainExportCancel in the chain RPC list.
CLI
src/cli/subcommands/snapshot_cmd.rs
Added public Format enum and SnapshotCommands::ExportStatus { wait, format } and ExportCancel {} variants; adapted export handling to ApiExportResult; implemented status polling (--wait) with progress rendering and cancel handling.
Progress State & Stream
src/ipld/util.rs
Added public ExportStatus and CHAIN_EXPORT_STATUS (LazyLock<Mutex<...>>), start_export/end_export/cancel_export; added track_progress field and builder on ChainStream; instrumented stream to update epoch when tracking.
Chain Export Integration
src/chain/mod.rs
Enabled progress tracking by calling .track_progress(true) on the ChainStream used for export.
Tests / Tooling & Scripts
src/tool/subcommands/api_cmd/test_snapshots_ignored.txt, scripts/tests/calibnet_export_check.sh
Added Forest.ChainExportCancel and Forest.ChainExportStatus to ignored list; added orchestration script changes to background export, poll status, cancel export, and validate cancellation and subsequent export flows.

Sequence Diagram(s)

sequenceDiagram
    participant CLI
    participant RPC as Forest RPC
    participant State as CHAIN_EXPORT_STATUS
    CLI->>RPC: ForestChainExportStatus(--wait?)
    activate RPC
    loop while exporting
        RPC->>State: read ApiExportStatus
        State-->>RPC: ApiExportStatus(progress, exporting, cancelled, start_time)
        RPC-->>CLI: render progress / status
    end
    alt cancelled
        RPC-->>CLI: Export cancelled
    else done
        RPC-->>CLI: Export completed
    end
    deactivate RPC
Loading
sequenceDiagram
    participant CLI
    participant RPC as Forest RPC
    participant Lock as CHAIN_EXPORT_LOCK
    participant State as CHAIN_EXPORT_STATUS
    CLI->>RPC: ForestChainExportCancel()
    activate RPC
    RPC->>Lock: attempt cancel token
    alt token existed
        Lock-->>RPC: cancelled
        RPC->>State: set cancelled flag
        RPC-->>CLI: true / "Export cancelled."
    else none
        RPC-->>CLI: false / "No export in progress to cancel."
    end
    deactivate RPC
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested labels

RPC

Suggested reviewers

  • akaladarshi
  • LesnyRumcajs

Pre-merge checks and finishing touches

❌ Failed checks (2 warnings)
Check name Status Explanation Resolution
Out of Scope Changes Check ⚠️ Warning While the functional code changes are appropriately scoped to implement issue #6082, the CHANGELOG.md file contains entries for issues #6161, #6166, and #6171 that do not have corresponding code implementations visible in the provided summaries. Specifically, the changelog removes an entry for #6161 (forest-tool db import) and adds entries for #6166 (JWT expiration validation) and #6171 (V2 API support for Eth RPC methods), yet the code summaries show no implementations of these features beyond the #6082 snapshot export functionality. These CHANGELOG entries appear disconnected from the actual code changes in this PR. Remove the CHANGELOG entries for issues #6161, #6166, and #6171 from this PR, or provide corresponding code changes and explanations for these features. The CHANGELOG should accurately reflect only the features implemented in the current pull request. If these entries are intended for a multi-issue release, they should be managed through separate changelog organization rather than mixed into a PR focused on #6082.
Docstring Coverage ⚠️ Warning Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check ✅ Passed The pull request title "Add new subcommands for forest-cli snasphot" accurately describes the main change in the changeset: the addition of new CLI subcommands (export-status and export-cancel) for the snapshot command. The title directly relates to the primary objective from the linked issue #6082. However, there is a typographical error in the title: "snasphot" should be "snapshot," which slightly diminishes clarity though it remains understandable in context.
Linked Issues Check ✅ Passed The pull request implements all primary coding requirements from linked issue #6082. The implementation includes: progress reporting via the new ForestChainExportStatus RPC method and ApiExportStatus type; a CLI export-status subcommand with a --wait flag to poll and display progress using a progress bar; a ForestChainExportCancel RPC method and export-cancel CLI subcommand to cancel in-progress exports; comprehensive progress tracking infrastructure in src/ipld/util.rs and src/rpc/methods/chain.rs; and integration tests validating the export cancellation and status checking workflows. All required features specified in the issue are present in the code changes.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch elmattic/snapshot-export-qol

Comment @coderabbitai help to get the list of available commands and usage tips.

@elmattic elmattic marked this pull request as ready for review October 8, 2025 13:56
@elmattic elmattic requested a review from a team as a code owner October 8, 2025 13:56
@elmattic elmattic requested review from akaladarshi and hanabi1224 and removed request for a team October 8, 2025 13:56

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🧹 Nitpick comments (7)
src/rpc/types/mod.rs (1)

33-33: Add safety notes for status fields; otherwise good

  • Chrono DateTime with serde is fine; lotus_json wrapper will pass through.
  • Consider adding #[serde(rename_all = "PascalCase")] for consistency with adjacent types (optional).

Also applies to: 566-582

src/chain/mod.rs (1)

31-36: Cancellation notifier risks missed or stale signals

Using a global Notify without a per-job “generation” can race:

  • If cancel is called before the export starts awaiting, notify_waiters() does nothing and the cancel is lost (race).
  • If switched to notify_one() to avoid loss, a late cancel may leave a pending permit and cancel the next export spuriously.

Prefer a per-export token:

  • Use tokio_util::sync::CancellationToken stored for the active export (e.g., in a Mutex<Option>). On start, create/replace the token; in cancel, call token.cancel(); in the export, select on token.cancelled().
src/cli/subcommands/snapshot_cmd.rs (3)

142-147: Fix progress template: use current bytes, not total.

You're updating position but not setting a length, so {binary_total_bytes} won’t show progress. Use {binary_bytes}.

-    ProgressStyle::with_template(
-        "{spinner} {msg} {binary_total_bytes} written in {elapsed} ({binary_bytes_per_sec})",
-    )
+    ProgressStyle::with_template(
+        "{spinner} {msg} {binary_bytes} written in {elapsed} ({binary_bytes_per_sec})",
+    )

212-237: Throttle polling; cleanly clear the bar on early exit.

Polling every 10ms will hammer the RPC. Use ~500–1000ms. Also abandon/finish the bar when returning early to avoid a stale line.

-                        if !result.exporting {
-                            return Ok(());
-                        }
+                        if !result.exporting {
+                            pb.abandon();
+                            return Ok(());
+                        }
@@
-                        tokio::time::sleep(Duration::from_millis(10)).await;
+                        tokio::time::sleep(Duration::from_millis(750)).await;

253-255: Improve UX on no-op cancel.

Print a message when there’s nothing to cancel.

-                if result {
-                    println!("Export cancelled.");
-                }
+                if result {
+                    println!("Export cancelled.");
+                } else {
+                    println!("No export in progress.");
+                }
src/ipld/util.rs (2)

32-34: Prefer explicit Mutex::new for clarity.

Avoid relying on Into for Mutex construction; it’s less idiomatic and may surprise readers.

-pub static CHAIN_EXPORT_STATUS: LazyLock<Mutex<ExportStatus>> =
-    LazyLock::new(|| ExportStatus::default().into());
+pub static CHAIN_EXPORT_STATUS: LazyLock<Mutex<ExportStatus>> =
+    LazyLock::new(|| Mutex::new(ExportStatus::default()));

298-300: Use consistent epoch access.

Elsewhere this module and callers use block.epoch. Mixing block.uncached.epoch bypasses invariants and is inconsistent.

-                            update_epoch(block.uncached.epoch);
+                            update_epoch(block.epoch);
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2bafb16 and c837c8a.

📒 Files selected for processing (8)
  • CHANGELOG.md (1 hunks)
  • src/blocks/header.rs (1 hunks)
  • src/chain/mod.rs (2 hunks)
  • src/cli/subcommands/snapshot_cmd.rs (3 hunks)
  • src/ipld/util.rs (6 hunks)
  • src/rpc/methods/chain.rs (6 hunks)
  • src/rpc/mod.rs (1 hunks)
  • src/rpc/types/mod.rs (2 hunks)
🧰 Additional context used
🧬 Code graph analysis (3)
src/rpc/methods/chain.rs (2)
src/ipld/util.rs (3)
  • cancel_export (57-61)
  • end_export (52-55)
  • start_export (43-50)
src/chain/mod.rs (3)
  • export (52-52)
  • export (58-67)
  • export_v2 (71-133)
src/ipld/util.rs (2)
src/blocks/header.rs (3)
  • cid (69-71)
  • cid (304-306)
  • new (282-288)
src/blocks/tipset.rs (5)
  • cid (40-48)
  • new (242-260)
  • new (494-511)
  • epoch (306-308)
  • epoch (543-545)
src/cli/subcommands/snapshot_cmd.rs (1)
src/rpc/methods/chain.rs (18)
  • chain (375-375)
  • chain (413-413)
  • handle (145-147)
  • handle (161-173)
  • handle (187-202)
  • handle (217-231)
  • handle (246-278)
  • handle (291-299)
  • handle (312-323)
  • handle (336-439)
  • handle (452-465)
  • handle (478-487)
  • handle (500-543)
  • handle (556-580)
  • handle (596-605)
  • handle (620-625)
  • handle (640-678)
  • handle (693-709)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
  • GitHub Check: Build forest binaries on Linux AMD64
  • GitHub Check: tests-release
  • GitHub Check: tests
  • GitHub Check: Build Ubuntu
  • GitHub Check: cargo-publish-dry-run
  • GitHub Check: Build MacOS
  • GitHub Check: All lint checks
🔇 Additional comments (5)
src/blocks/header.rs (1)

239-239: Clarify necessity of public uncached field (src/blocks/header.rs:239)

  • Currently only used internally (e.g., src/ipld/util.rs:299); if it’s not consumed by external crates, restrict visibility to pub(crate) or expose via a getter.
  • Fully public access allows direct mutation, risking cached CID/signature state becoming inconsistent.
  • Please document the snapshot-export use case that requires this field to be fully public.
CHANGELOG.md (1)

38-39: Changelog entry looks good

Entry correctly reflects the new CLI subcommands under “Added”.

src/rpc/mod.rs (1)

86-88: Method registration looks correct

Both ForestChainExportStatus and ForestChainExportCancel are properly registered under the chain vertical.

src/chain/mod.rs (1)

167-169: Progress tracking enablement

Enabling track_progress(true) here is appropriate to surface export status. Ensure the tracker correctly bounds values [0,1] to avoid NaN/Inf in RPC (see related comment in chain RPC).

src/cli/subcommands/snapshot_cmd.rs (1)

119-123: No changes required: DateTime::from_timestamp(secs, nanos) is provided by chrono 0.4.x.

Comment thread src/cli/subcommands/snapshot_cmd.rs Outdated
Comment thread src/cli/subcommands/snapshot_cmd.rs
Comment thread src/ipld/util.rs
Comment thread src/rpc/methods/chain.rs
Comment thread src/rpc/methods/chain.rs

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (3)
src/cli/subcommands/snapshot_cmd.rs (3)

164-166: Previous concern remains: Duration::MAX in timeout.

As flagged in the previous review, Duration::MAX can overflow or misbehave in timer math. Consider using a large finite timeout.

Apply this diff to use a finite timeout:

-    let export_result = client
-        .call(ForestChainExport::request((params,))?.with_timeout(Duration::MAX))
-        .await?;
+    let export_result = client
+        .call(ForestChainExport::request((params,))?.with_timeout(Duration::from_secs(6 * 60 * 60)))
+        .await?;

172-178: Previous concern remains: Persist before checksum, avoid let-chains.

As flagged in the previous review:

  1. Persisting the file after writing the checksum can leave an orphaned checksum file if persist fails.
  2. The if-let chain syntax may pose MSRV concerns depending on the project's minimum Rust version.
  3. The clone() on line 172 is unnecessary since export_result is only used once more in the match statement below.

Apply this diff:

-    if !dry_run && let ApiExportResult::Done(hash_opt) = export_result.clone() {
-        if let Some(hash) = hash_opt {
-            save_checksum(&output_path, hash).await?;
-        }
-
-        temp_path.persist(output_path)?;
-    }
+    if !dry_run {
+        match &export_result {
+            ApiExportResult::Done(hash_opt) => {
+                // Persist first to avoid orphaned checksum on failure
+                temp_path.persist(&output_path)?;
+                if let Some(hash) = hash_opt {
+                    save_checksum(&output_path, hash.clone()).await?;
+                }
+            }
+            ApiExportResult::Cancelled => {
+                // No file to persist on cancel
+            }
+        }
+    }

239-244: Previous concern remains: Duration::MAX in timeout.

As flagged in the previous review, Duration::MAX can overflow or misbehave in timer math. For a simple status query, a shorter timeout is more appropriate.

Apply this diff:

-    let result = client
-        .call(ForestChainExportStatus::request(())?.with_timeout(Duration::MAX))
-        .await?;
+    let result = client
+        .call(ForestChainExportStatus::request(())?.with_timeout(Duration::from_secs(30)))
+        .await?;
🧹 Nitpick comments (1)
src/cli/subcommands/snapshot_cmd.rs (1)

246-256: Add user feedback when no export is running.

When result is false (no export is currently running), the command exits silently. Users would benefit from explicit feedback in this case.

Apply this diff:

     let result = client
         .call(
             ForestChainExportCancel::request(())?.with_timeout(Duration::from_secs(30)),
         )
         .await?;
-    if result {
+    if result {
         println!("Export cancelled.");
+    } else {
+        println!("No export is currently running.");
     }
     Ok(())
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c837c8a and 7760b22.

📒 Files selected for processing (1)
  • src/cli/subcommands/snapshot_cmd.rs (3 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
src/cli/subcommands/snapshot_cmd.rs (1)
src/rpc/methods/chain.rs (16)
  • handle (145-147)
  • handle (161-173)
  • handle (187-202)
  • handle (217-231)
  • handle (246-278)
  • handle (291-299)
  • handle (312-323)
  • handle (336-439)
  • handle (452-465)
  • handle (478-487)
  • handle (500-543)
  • handle (556-580)
  • handle (596-605)
  • handle (620-625)
  • handle (640-678)
  • handle (693-709)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
  • GitHub Check: tests-release
  • GitHub Check: tests
  • GitHub Check: Build forest binaries on Linux AMD64
  • GitHub Check: Build MacOS
  • GitHub Check: Build Ubuntu
  • GitHub Check: cargo-publish-dry-run
  • GitHub Check: All lint checks
🔇 Additional comments (3)
src/cli/subcommands/snapshot_cmd.rs (3)

10-10: LGTM!

The new import is necessary to handle the export result variants (Done/Cancelled).


46-53: LGTM!

The new command variants are well-structured and align with the PR objectives.


180-188: LGTM!

The user feedback messages for completion and cancellation are appropriate.

Comment thread src/cli/subcommands/snapshot_cmd.rs Outdated
Comment thread src/cli/subcommands/snapshot_cmd.rs

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
scripts/tests/calibnet_export_check.sh (2)

46-46: Inconsistent format specification across export commands.

Earlier in the script, exports use --format "$format" (lines 18, 21) to respect the format variable, but the new test sections either hard-code --format json (lines 46, 57, 66) or omit format entirely (line 54). This creates inconsistency and makes the test less flexible.

Apply this diff to use the format variable consistently:

-output=$($FOREST_CLI_PATH snapshot export-status --format json)
+output=$($FOREST_CLI_PATH snapshot export-status --format "$format")

Repeat for lines 57 and 66. For line 54, add the format flag:

-$FOREST_CLI_PATH snapshot export
+$FOREST_CLI_PATH snapshot export --format "$format"

Also applies to: 54-54, 57-57, 64-64, 66-66


46-51: Insufficient error handling for jq parsing failures.

If jq fails to parse the JSON response or the expected fields (.exporting, .cancelled) are missing or null, the variables will be empty or contain unexpected values. The set -e option won't catch this because command substitution doesn't propagate jq errors by default. Add error handling:

Apply this diff to add explicit error checks:

-output=$($FOREST_CLI_PATH snapshot export-status --format json)
-is_exporting=$(echo "$output" | jq -r '.exporting')
+output=$($FOREST_CLI_PATH snapshot export-status --format json) || { echo "Failed to get export status"; exit 1; }
+is_exporting=$(echo "$output" | jq -r '.exporting' 2>/dev/null) || { echo "Failed to parse exporting field"; exit 1; }
 echo "Testing that no export is in progress"
-if [ "$is_exporting" == "true" ]; then
+if [ -z "$is_exporting" ] || [ "$is_exporting" != "true" ] && [ "$is_exporting" != "false" ]; then
+  echo "Unexpected exporting value: $is_exporting"
   exit 1
 fi

Alternatively, enable set -o pipefail at the top of the script to propagate errors through pipes.

Also applies to: 57-62, 66-72

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 781569e and 747d7b3.

📒 Files selected for processing (1)
  • scripts/tests/calibnet_export_check.sh (1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
scripts/tests/calibnet_export_check.sh (2)
src/chain/mod.rs (2)
  • export (49-49)
  • export (55-64)
src/tool/subcommands/archive_cmd.rs (1)
  • export (1166-1191)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
  • GitHub Check: Build MacOS
  • GitHub Check: Build Ubuntu
  • GitHub Check: cargo-publish-dry-run
  • GitHub Check: tests-release
  • GitHub Check: tests
  • GitHub Check: All lint checks
  • GitHub Check: Build forest binaries on Linux AMD64

Comment thread scripts/tests/calibnet_export_check.sh Outdated
Comment on lines +54 to +62
$FOREST_CLI_PATH snapshot export
sleep 1

output=$($FOREST_CLI_PATH snapshot export-status --format json)
is_exporting=$(echo "$output" | jq -r '.exporting')
echo "Testing that export is in progress"
if [ "$is_exporting" == "false" ]; then
exit 1
fi

@coderabbitai coderabbitai Bot Oct 17, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major

🧩 Analysis chain

Hard-coded 1-second sleep creates race condition risk.

The test assumes the export will still be in-progress after sleep 1, but this timing is not guaranteed. If the export completes faster, the test fails; if slower, you're only testing intermediate progress by coincidence. Consider:

  • Making the wait duration configurable or polling with timeout
  • Using --wait flag (mentioned in PR objectives) if the export command supports it
  • Adding a loop with timeout instead of fixed sleep

Does the snapshot export command support a --wait flag or similar mechanism to wait for completion? If so, consider using it instead of a hard-coded sleep.


I'll verify this review comment by examining the script and checking the context around the flagged issues.


🏁 Script executed:

#!/bin/bash
# Read the full script to understand context and format variable usage
echo "=== Full script content ==="
cat scripts/tests/calibnet_export_check.sh

echo ""
echo "=== Checking for format variable usage ==="
rg -n "format" scripts/tests/calibnet_export_check.sh

echo ""
echo "=== Checking for error handling patterns ==="
rg -n "set -" scripts/tests/calibnet_export_check.sh

echo ""
echo "=== Checking for wait or polling patterns ==="
rg -n "wait|sleep|poll" scripts/tests/calibnet_export_check.sh

Length of output: 2571


🏁 Script executed:

#!/bin/bash
# Search for --wait flag in forest-cli documentation or related files
echo "=== Searching for --wait flag in codebase ==="
rg -n "\-\-wait" --type=sh --type=rs --type=toml -C2

echo ""
echo "=== Searching for snapshot export command definitions ==="
rg -n "snapshot.*export" --type=rs --type=toml -C2 | head -50

Length of output: 158



🏁 Script executed:

#!/bin/bash
# Search for --wait flag more broadly
echo "=== Searching for --wait flag ==="
rg -n "\-\-wait" -C2

echo ""
echo "=== Searching for snapshot export command implementation ==="
rg -n "snapshot.*export" -C2 | head -80

echo ""
echo "=== Checking jq error handling in other test files ==="
rg -n "jq.*pipefail|pipefail.*jq" -C2

echo ""
echo "=== Looking for export-status command documentation ==="
rg -n "export-status|export_status" -C2 | head -50

Length of output: 12130



Use consistent --format variable throughout snapshot export lifecycle tests.

The script establishes a format variable (line 8) and uses it for export calls (lines 18, 21), but the status checks hard-code --format json (lines 46, 57, 66). Additionally, line 54's snapshot export call omits the format flag entirely. For consistency and to respect the test's parameterization, use --format "$format" in all three status calls and add it to line 54's export invocation:

# Line 54
$FOREST_CLI_PATH snapshot export --format "$format"

# Lines 46, 57, 66
output=$($FOREST_CLI_PATH snapshot export-status --format "$format")

Note: The snapshot export command does not currently support a --wait flag; the 1-second sleep is the intended pattern to allow the operation to begin before checking its status.

🤖 Prompt for AI Agents
In scripts/tests/calibnet_export_check.sh around lines 54–62 (and related status
checks at ~lines 46, 57, 66), the script inconsistently uses a hard-coded
--format json and omits the format flag on the snapshot export call; update the
snapshot export invocation to include --format "$format" and change all snapshot
export-status calls to use --format "$format" so the test consistently respects
the parameterized format variable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@elmattic Thank you! Let me know if you need any assistance with the implementation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It’s not really relevant since the export task will be cancelled and no file will be written anyway. Let’s keep the code simple.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@elmattic Understood! Since the export will be cancelled before completion in that test section, the format parameter isn't material there. Thank you for the clarification.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (1)
scripts/tests/calibnet_export_check.sh (1)

44-44: Resolve the format variable inconsistency flagged in the previous review.

The script establishes a parameterized format variable at line 8 but all new snapshot export-status calls (lines 44, 55, 64) hard-code --format json, and the export call at line 52 omits the format flag entirely. This was flagged in a prior review and acknowledged as a good catch, but remains unfixed. Use --format "$format" consistently across all snapshot commands to respect the test's parameterization:

- output=$($FOREST_CLI_PATH snapshot export-status --format json)
+ output=$($FOREST_CLI_PATH snapshot export-status --format "$format")

Apply this change at lines 44, 55, and 64. Also update line 52:

- $FOREST_CLI_PATH snapshot export
+ $FOREST_CLI_PATH snapshot export --format "$format"

Also applies to: 52-52, 55-55, 64-64

🧹 Nitpick comments (1)
scripts/tests/calibnet_export_check.sh (1)

45-46: Add error handling for jq field extraction.

The jq commands at lines 45, 56, and 66 extract boolean fields without verifying the JSON structure. If the API response format changes or required fields are missing, jq may fail silently or produce unclear errors. Consider adding error checking:

is_exporting=$(echo "$output" | jq -e -r '.exporting' 2>/dev/null) || { echo "Failed to parse export status"; exit 1; }

Alternatively, use jq -e to fail on null values. This improves test robustness and debuggability.

Also applies to: 56-57, 66-67

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 747d7b3 and 9504ea1.

📒 Files selected for processing (1)
  • scripts/tests/calibnet_export_check.sh (1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
scripts/tests/calibnet_export_check.sh (3)
src/chain/mod.rs (2)
  • export (49-49)
  • export (55-64)
src/tool/subcommands/archive_cmd.rs (1)
  • export (1166-1191)
src/chain/tests.rs (1)
  • export (62-62)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
  • GitHub Check: Build Ubuntu
  • GitHub Check: cargo-publish-dry-run
  • GitHub Check: Build MacOS
  • GitHub Check: Build forest binaries on Linux AMD64
  • GitHub Check: tests
  • GitHub Check: tests-release
  • GitHub Check: All lint checks

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (2)
scripts/tests/calibnet_export_check.sh (2)

47-52: Add error handling for jq to prevent silent test failures.

The script uses set -eu but not pipefail. If jq parsing fails (line 47, 59, 75), the assignment succeeds with the error message instead of the expected JSON value, causing silent test failures. For example, if .exporting is undefined, is_exporting becomes an error string rather than "true" or "false", breaking the conditional logic.

Recommend adding set -o pipefail at the top of the script (after line 6) to catch jq failures explicitly:

 set -eu
+set -o pipefail

Alternatively, add explicit error checks after jq calls:

-output=$($FOREST_CLI_PATH snapshot export-status --format json)
-is_exporting=$(echo "$output" | jq -r '.exporting')
+output=$($FOREST_CLI_PATH snapshot export-status --format json) || exit 1
+is_exporting=$(echo "$output" | jq -r '.exporting') || { echo "Failed to parse export status"; exit 1; }

Also applies to: 59-69, 75-86


44-46: Add cleanup for test artifacts created during the test.

Line 55 redirects export output to snapshot_export.log, which is then read on failure (lines 65, 82). However, this log file is not cleaned up after the test completes. On subsequent test runs, this file may cause confusion or interfere with test reproducibility.

Recommend adding snapshot_export.log to the cleanup step (lines 44–45):

 echo "Cleaning up the last snapshot"
-rm --force --verbose ./*.{car,car.zst,sha256sum}
+rm --force --verbose ./*.{car,car.zst,sha256sum} snapshot_export.log

Also applies to: 54-56

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9504ea1 and e32f14f.

📒 Files selected for processing (1)
  • scripts/tests/calibnet_export_check.sh (1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (8)
  • GitHub Check: Build forest binaries on Linux AMD64
  • GitHub Check: Check
  • GitHub Check: cargo-publish-dry-run
  • GitHub Check: Build MacOS
  • GitHub Check: Build Ubuntu
  • GitHub Check: All lint checks
  • GitHub Check: tests-release
  • GitHub Check: tests
🔇 Additional comments (1)
scripts/tests/calibnet_export_check.sh (1)

41-42: Polling and retry logic is well-structured.

The use of configurable retries and sleep_interval (lines 41–42) with proper timeout and failure diagnostics (lines 65, 82) ensures robust detection of export state transitions. The polling loops correctly allow the export to start and reach in-progress state before verification, and properly verify cancellation state afterward.

Also applies to: 58-69, 74-86

@LesnyRumcajs

Copy link
Copy Markdown
Member

no green checkmark, no review!

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (2)
src/rpc/methods/chain.rs (2)

548-555: Cancel handler should take() the token to avoid stale-cancel races

Keeping the token in the mutex after calling cancel() lets subsequent export-cancel calls report true and, worse, risks cancelling a freshly started export if cleanup lags. Take the token atomically when cancelling.

-    async fn handle(_ctx: Ctx<impl Blockstore>, (): Self::Params) -> Result<Self::Ok, ServerError> {
-        if let Some(token) = CHAIN_EXPORT_LOCK.lock().await.as_ref() {
-            token.cancel();
-            return Ok(true);
-        }
-        Ok(false)
-    }
+    async fn handle(_ctx: Ctx<impl Blockstore>, (): Self::Params) -> Result<Self::Ok, ServerError> {
+        let mut guard = CHAIN_EXPORT_LOCK.lock().await;
+        if let Some(token) = guard.take() {
+            token.cancel();
+            return Ok(true);
+        }
+        Ok(false)
+    }

513-536: Progress semantics: return 1.0 when idle; keep clamp/is_finite guards

When no export is active, progress should be 1.0, not 0.0. Also keep clamping and finiteness checks to avoid NaN/Inf serialization.

-        let progress = if mutex.initial_epoch == 0 {
-            0.0
-        } else {
-            let p = 1.0 - ((mutex.epoch as f64) / (mutex.initial_epoch as f64));
-            if p.is_finite() {
-                p.clamp(0.0, 1.0)
-            } else {
-                0.0
-            }
-        };
+        let progress = if mutex.initial_epoch == 0 {
+            if mutex.exporting { 0.0 } else { 1.0 }
+        } else {
+            let p = 1.0 - ((mutex.epoch as f64) / (mutex.initial_epoch as f64));
+            if p.is_finite() { p.clamp(0.0, 1.0) } else { 0.0 }
+        };
🧹 Nitpick comments (1)
src/rpc/methods/chain.rs (1)

579-585: Don’t hold CHAIN_EXPORT_LOCK for the entire diff export

Using CHAIN_EXPORT_LOCK as a coarse-grained guard via try_lock() and keeping the guard alive through the whole operation blocks cancel/start of snapshot exports and can cause unrelated calls to await the lock. Prefer a dedicated diff job guard or a non-blocking state check.

-        let _locked = CHAIN_EXPORT_LOCK.try_lock();
-        if _locked.is_err() {
+        // Use a dedicated diff mutex/flag instead of the shared export lock,
+        // or drop the guard immediately after the check.
+        let locked = CHAIN_EXPORT_LOCK.try_lock();
+        if locked.is_err() {
             return Err(
                 anyhow::anyhow!("Another chain export diff job is still in progress").into(),
             );
         }
+        drop(locked); // don't hold the shared lock across the long-running operation
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b1667a7 and d2a2f69.

📒 Files selected for processing (3)
  • CHANGELOG.md (1 hunks)
  • src/chain/mod.rs (1 hunks)
  • src/rpc/methods/chain.rs (7 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/chain/mod.rs
🧰 Additional context used
🧬 Code graph analysis (1)
src/rpc/methods/chain.rs (1)
src/ipld/util.rs (3)
  • cancel_export (57-61)
  • end_export (52-55)
  • start_export (43-50)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
  • GitHub Check: All lint checks
  • GitHub Check: Build forest binaries on Linux AMD64
  • GitHub Check: cargo-publish-dry-run
  • GitHub Check: Build Ubuntu
  • GitHub Check: Build MacOS
  • GitHub Check: tests-release
  • GitHub Check: tests
🔇 Additional comments (2)
src/rpc/methods/chain.rs (1)

385-498: Confirm permissions for exporting/cancelling snapshots

Both Forest.ChainExport and Forest.ChainExportCancel use Permission::Read. Given these write to local filesystem and can disrupt running jobs, consider Permission::Admin unless there’s a strong compatibility constraint.

If changing is acceptable, update:

  • Lines 383-384 (ForestChainExport) and 542-544 (ForestChainExportCancel) to Permission::Admin.

Also applies to: 621-648

CHANGELOG.md (1)

32-33: Changelog entry LGTM

Accurately reflects new subcommands; no further action.

@elmattic elmattic requested a review from hanabi1224 October 22, 2025 13:12
@elmattic elmattic added this pull request to the merge queue Oct 23, 2025
Merged via the queue into main with commit 613e790 Oct 23, 2025
50 checks passed
@elmattic elmattic deleted the elmattic/snapshot-export-qol branch October 23, 2025 13:37
@coderabbitai coderabbitai Bot mentioned this pull request Jun 12, 2026
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Snapshot export status/stop

4 participants