Skip to content

Slim update feed storage#354

Merged
ModerRAS merged 2 commits into
masterfrom
codex/slim-update-feed
May 15, 2026
Merged

Slim update feed storage#354
ModerRAS merged 2 commits into
masterfrom
codex/slim-update-feed

Conversation

@ModerRAS
Copy link
Copy Markdown
Owner

@ModerRAS ModerRAS commented May 14, 2026

Summary

  • stop producing step packages and custom full MUP zst packages for the update feed
  • keep only cumulative MUP packages in B2, with touched-file tracking and full snapshot manifests for future cumulative planning
  • publish the user-facing full package as a normal GitHub Release zip and add absolute package/updater URLs to catalog.json
  • teach the Windows self-updater to download absolute URLs and extract zip fallback packages

Tests

  • dotnet test TelegramSearchBot.Test\TelegramSearchBot.Test.csproj -c Release --filter "FullyQualifiedNameUpdate|FullyQualifiedNameModerUpdateIntegrationTests"
  • dotnet build TelegramSearchBot.sln -c Release
  • dotnet test TelegramSearchBot.sln -c Release --no-build --verbosity normal

Summary by CodeRabbit

  • New Features

    • Added support for full release packages in update catalog with metadata storage
    • Enabled ZIP package extraction alongside existing cumulative updates
    • Implemented URL-based package download capability for update packages and updater
    • Enhanced update catalog metadata with updater URLs and full package references
  • Chores

    • Updated CI/CD workflow for improved package management and cloud storage retention policies
    • Refined documentation for build, test, and auto-update processes
    • Expanded test coverage for update builder and bootstrap functionality

Review Change Stack

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 14, 2026

Warning

Rate limit exceeded

@ModerRAS has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 45 minutes and 16 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8b7c37db-2c73-4137-9cda-c728c7258da5

📥 Commits

Reviewing files that changed from the base of the PR and between 97a3deb and e0651ba.

📒 Files selected for processing (7)
  • .github/workflows/push.yml
  • TelegramSearchBot.Common/Model/Update/UpdateCatalog.cs
  • TelegramSearchBot.Test/ModerUpdateIntegrationTests.cs
  • TelegramSearchBot.Test/Service/Update/SelfUpdateBootstrapTests.cs
  • TelegramSearchBot.Test/Service/Update/UpdateBuilderTests.cs
  • TelegramSearchBot.UpdateBuilder/Program.cs
  • TelegramSearchBot/Service/Update/SelfUpdateBootstrap.Windows.cs
📝 Walkthrough

Walkthrough

This PR extends the TelegramSearchBot update system to support optional full-package fallback metadata in the update catalog, cumulative package generation via touched-file snapshot planning, and runtime downloads from catalog-provided URLs with ZIP archive support.

Changes

Full-Package Metadata and Cumulative Snapshot-Based Planning

Layer / File(s) Summary
Update data contracts and package format definitions
TelegramSearchBot.Common/Model/Update/UpdateCatalog.cs, TelegramSearchBot.Common/Model/Update/UpdateCatalogEntry.cs, TelegramSearchBot.Common/Model/Update/UpdateManifest.cs, TelegramSearchBot.Common/Model/Update/UpdatePackageFormats.cs, external/moder-update/src/Moder.Update/Models/UpdateCatalogEntry.cs, external/moder-update/src/Moder.Update/Models/UpdateManifest.cs
New optional/required properties: UpdateCatalog gains UpdaterUrl, FullPackageUrl, FullPackageName, FullPackageChecksum, FullPackageSize; UpdateCatalogEntry gains PackageUrl and PackageFormat; UpdateManifest gains SnapshotFiles; new UpdatePackageFormats constants class defines ModerUpdateZstd and Zip identifiers.
Update builder: cumulative planning, snapshots, and full-package metadata
TelegramSearchBot.UpdateBuilder/Program.cs
Implements anchor-based touched-file planning by aggregating paths from base cumulative manifests, source manifests, anchor directory, and optional previous files, then materializes only those touched files from the current snapshot. Adds optional catalog-only full-package metadata driven by --full-package-* arguments. Snapshot-aware manifest loading prefers SnapshotFiles when available. Catalog pruning retains at most one historical cumulative entry per format/suffix. Snapshots are populated in generated UpdateManifest when provided. New CLI arguments: --full-package-url, --full-package-name, --full-package-checksum, --full-package-size, --updater-url, --package-base-url.
Workflow: full-package generation and builder integration
.github/workflows/push.yml
Adds dedicated "Package full release asset" step that creates TelegramSearchBot-win-x64-full-<version>.zip, computes SHA-512, and exports FULL_PACKAGE_* environment variables. Cumulative planning logic updates: messages now reference "cumulative planning" and "cumulative-free" terminology; base cumulative package selection filters by *-cumulative.zst and prefers PackageUrl when present; cumulative source package selection similarly filters and prefers PackageUrl. Moder.Update builder now receives --updater-url, --package-base-url, and full-package metadata when available. Catalog generation injects UpdaterUrl and includes PackageUrl in selected fields. B2 reachability pruning narrowed to packages/* paths; conditional sync of update-feed packages directory only when it exists.
Runtime bootstrap: catalog-driven downloads and ZIP extraction
TelegramSearchBot/Service/Update/SelfUpdateBootstrap.Windows.cs
Package preparation downloads from catalog PackageUrl or path, derives cache filename from entry metadata, verifies checksums, and extracts by format. ZIP detection via PackageFormat, extension, or PackageUrl; ZIP extraction uses ZipArchive with entry path normalization and directory-escape protection. Downloader refactored to accept pathOrUrl and resolve against Env.UpdateBaseUrl when needed. Updater download uses UpdaterUrl when available.
Test suite: builder integration and bootstrap ZIP support
TelegramSearchBot.Test/Service/Update/UpdateBuilderTests.cs, TelegramSearchBot.Test/Service/Update/SelfUpdateBootstrapTests.cs, TelegramSearchBot.Test/ModerUpdateIntegrationTests.cs
New UpdateBuilderTests suite runs isolated builder processes with various cumulative/base/anchor combinations, validates output package selection, manifest reverted-file inclusion, snapshot isolation, and catalog pruning. SelfUpdateBootstrapTests extended with CreateTestZipPackage helper and ZIP extraction test; planning tests updated with PackageFormat and PackageUrl assertions. Integration test updated to assert new CLI arguments and cumulative-vs-full package filtering.
Documentation: build guide and README updates
Docs/Build_and_Test_Guide.md, README.md
Build guide specifies cumulative update packages and full-package-only GitHub Releases upload with B2 pruning of expired versions. README clarifies cumulative update packages as update source and full-zip availability on Releases for manual deployment/rollback.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~65 minutes

Possibly related PRs

  • ModerRAS/TelegramSearchBot#312: Modifies .github/workflows/push.yml around publishing TelegramSearchBot-win-x64-full-<version>.zip to GitHub Releases and adjusting B2 cleanup behavior for release artifacts.
  • ModerRAS/TelegramSearchBot#348: Modifies Windows self-update bootstrap download/extraction pipeline in TelegramSearchBot/Service/Update/SelfUpdateBootstrap.Windows.cs, overlapping in the same code paths for package downloading and extraction.
  • ModerRAS/TelegramSearchBot#323: Modifies the same update pipeline code including push.yml, UpdateBuilder/Program.cs, and shared update models, extending the chain-based incremental update flow.

Poem

🐰 A rabbit hops through versions new,
With snapshots kept and touched-files true,
Full packages wait on GitHub's shore,
While cumulative paths guide us more.
ZIP zips and URLs dance with glee—
Update once, update gracefully!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 2.56% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Slim update feed storage' directly aligns with the PR's primary objective of reducing storage by eliminating step packages and custom full zst packages while retaining only cumulative packages in B2.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/slim-update-feed

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 14, 2026

PR Check Report

Summary

Test Results

Platform Status Details
Ubuntu Passed Tests passed, artifacts uploaded
Windows Passed Tests passed, artifacts uploaded

Code Quality

  • Code formatting check
  • Security vulnerability scan
  • Dependency analysis
  • Code coverage collection

Test Artifacts

  • Test results artifacts count: 2
  • Code coverage uploaded to Codecov

Links


This report is auto-generated by GitHub Actions

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (4)
TelegramSearchBot.Common/Model/Update/UpdateCatalog.cs (1)

14-14: ⚡ Quick win

Consider making FullPackageSize nullable for consistency.

All other optional full-package metadata properties (lines 10-13) are nullable, but FullPackageSize defaults to 0. This creates ambiguity: you cannot distinguish between "size not provided" and "actual 0-byte file". Making it long? would maintain consistency with the other optional fields and eliminate the ambiguity.

♻️ Proposed fix
-    public long FullPackageSize { get; init; }
+    public long? FullPackageSize { get; init; }

You'll also need to update the corresponding usage in Program.cs line 580 and the workflow integration.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@TelegramSearchBot.Common/Model/Update/UpdateCatalog.cs` at line 14, Change
the FullPackageSize property to a nullable long (long?) on the UpdateCatalog
record to match the other optional full-package metadata fields, then adjust all
usages that assume a non-null value (notably the code that reads FullPackageSize
in Program.cs and any workflow/serialization code) to handle null
explicitly—either by checking HasValue, using a null-coalescing default where
appropriate, or altering logic to treat null as "unknown" rather than zero;
update any tests or integration points expecting a default 0 accordingly.
.github/workflows/push.yml (1)

290-290: ⚡ Quick win

Clarify error messages for anchor download failures.

Lines 290 and 300 say "refusing to publish a cumulative-free update feed", but this wording is misleading. If the anchor download fails, the generated feed will still contain other entries—it simply won't include the cumulative package from this specific anchor version.

Consider rephrasing to: "refusing to publish without cumulative package from anchor {anchorVersion}" for clarity.

♻️ Proposed fix
         gh release download $anchorTag --repo ${{ github.repository }} --pattern "TelegramSearchBot-win-x64-full-*.zip" --dir artifacts\anchor-release
         if ($LASTEXITCODE -ne 0) {
-          throw "Failed to download cumulative update anchor $anchorTag; refusing to publish a cumulative-free update feed."
+          throw "Failed to download cumulative update anchor $anchorTag; refusing to publish without cumulative package from anchor."
         }
 
         $zipFile = Get-ChildItem artifacts\anchor-release -Filter "TelegramSearchBot-win-x64-full-*.zip" | Select-Object -First 1
         if ($zipFile) {
           Expand-Archive -Path $zipFile.FullName -DestinationPath .\artifacts\anchor-standalone -Force
           Add-Content -Path $env:GITHUB_ENV -Value "ANCHOR_VERSION=$anchorVersion" -Encoding utf8
           Add-Content -Path $env:GITHUB_ENV -Value "ANCHOR_STANDALONE_DIR=artifacts\anchor-standalone" -Encoding utf8
           Write-Host "Extracted cumulative update anchor from $($zipFile.Name)."
         } else {
-          throw "No anchor full zip found; refusing to publish a cumulative-free update feed."
+          throw "No anchor full zip found; refusing to publish without cumulative package from anchor."
         }

Also applies to: 300-300

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/push.yml at line 290, Update the thrown error strings that
reference the anchor download failure to make them specific about the missing
cumulative package: replace occurrences that say "refusing to publish a
cumulative-free update feed." with wording like "refusing to publish without
cumulative package from anchor {anchorTag}" (or {anchorVersion} if available) so
the message clearly indicates the cumulative package for the given anchor is
missing; adjust both the throw at the location referencing anchorTag and the
similar throw at the other occurrence to use the new phrasing while keeping the
anchor identifier interpolated.
TelegramSearchBot.UpdateBuilder/Program.cs (1)

590-590: 💤 Low value

Consider improving the usage help formatting.

The usage string on line 590 is extremely long and difficult to read. Consider breaking it into multiple lines or a bulleted list for better maintainability and user readability.

♻️ Example improved formatting
     public static void PrintUsage()
     {
         Console.WriteLine("Usage:");
-        Console.WriteLine(
-            "  TelegramSearchBot.UpdateBuilder --source-dir <dir> --output-dir <dir> --target-version <version> --min-source-version <version> [--prev-source-dir <dir> --prev-version <version>] [--existing-catalog <path>] [--anchor-version <version> --anchor-source-dir <dir>] [--base-cumulative-package <path>] [--cumulative-source-package-dir <dir>] [--full-package-url <url> --full-package-name <name> --full-package-checksum <sha512> --full-package-size <bytes>] [--updater-url <url>] [--package-base-url <url>]");
+        Console.WriteLine("  TelegramSearchBot.UpdateBuilder");
+        Console.WriteLine("    --source-dir <dir>");
+        Console.WriteLine("    --output-dir <dir>");
+        Console.WriteLine("    --target-version <version>");
+        Console.WriteLine("    --min-source-version <version>");
+        Console.WriteLine("    [--prev-source-dir <dir> --prev-version <version>]");
+        Console.WriteLine("    [--existing-catalog <path>]");
+        Console.WriteLine("    [--anchor-version <version> --anchor-source-dir <dir>]");
+        Console.WriteLine("    [--base-cumulative-package <path>]");
+        Console.WriteLine("    [--cumulative-source-package-dir <dir>]");
+        Console.WriteLine("    [--full-package-url <url> --full-package-name <name>");
+        Console.WriteLine("     --full-package-checksum <sha512> --full-package-size <bytes>]");
+        Console.WriteLine("    [--updater-url <url>]");
+        Console.WriteLine("    [--package-base-url <url>]");
     }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@TelegramSearchBot.UpdateBuilder/Program.cs` at line 590, The long usage
string in Program.cs (the literal passed as the help/usage text near the
top-level usage message) is hard to read; split it into a multi-line verbatim
string or construct it from an array of shorter lines joined with
Environment.NewLine (or use StringBuilder) so each option appears on its own
line or as a bulleted list; update the code that currently contains the single
long literal (the usage string shown in Program.cs) to use the new multi-line
format so the help output and source remain readable and maintainable.
TelegramSearchBot/Service/Update/SelfUpdateBootstrap.Windows.cs (1)

877-895: 💤 Low value

Consider validating that at least one of PackageUrl or PackagePath is provided.

The current implementation relies on a fallback at line 887 when the filename cannot be derived. While this works, explicitly validating that entry.PackageUrl or entry.PackagePath is non-empty earlier in the method would make the contract clearer and catch malformed catalog entries sooner.

♻️ Optional early validation
 private static string GetPackageCacheFileName(UpdateCatalogEntry entry)
 {
+    if (string.IsNullOrWhiteSpace(entry.PackageUrl) && string.IsNullOrWhiteSpace(entry.PackagePath))
+    {
+        throw new InvalidDataException("Catalog entry must have either PackageUrl or PackagePath.");
+    }
+
     var pathOrUrl = string.IsNullOrWhiteSpace(entry.PackageUrl)
         ? entry.PackagePath
         : entry.PackageUrl;
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@TelegramSearchBot/Service/Update/SelfUpdateBootstrap.Windows.cs` around lines
877 - 895, GetPackageCacheFileName currently proceeds when both
UpdateCatalogEntry.PackageUrl and PackagePath are empty and only falls back to a
default filename later; add an explicit early validation in
GetPackageCacheFileName to check that at least one of entry.PackageUrl or
entry.PackagePath is not null/whitespace, and if both are empty throw an
ArgumentException (or ArgumentNullException) mentioning the invalid
UpdateCatalogEntry so callers get immediate, clear feedback; keep the existing
fallback logic for filename derivation and continue to use IsZipPackage to
decide default extension if you still need it after validation.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@TelegramSearchBot.UpdateBuilder/Program.cs`:
- Around line 555-561: The code silently ignores long.TryParse on
fullPackageSizeText, so fullPackageSize can be 0 even when parsing failed;
update the parsing logic around fullPackageSizeText/fullPackageSize and
fullPackageUrl so that you check the TryParse return value and handle failures:
if fullPackageUrl is provided require a successful long.TryParse (using the
returned bool) and either log/throw an error or exit with a clear message,
otherwise handle the missing/invalid size explicitly (e.g., set an optional flag
or default only when intended). Locate the parsing call where
values.TryGetValue("--full-package-size", out var fullPackageSizeText) and the
subsequent _ = long.TryParse(...) and replace with a validated parse that reacts
to failure.

In `@TelegramSearchBot/Service/Update/SelfUpdateBootstrap.Windows.cs`:
- Around line 648-653: The IsZipPackage method risks a NullReferenceException by
calling updateEntry.PackagePath.EndsWith(...) without a null check; update the
logic in IsZipPackage to guard PackagePath (and optionally PackageUrl) with
null/empty checks before calling EndsWith, e.g., check
!string.IsNullOrEmpty(updateEntry.PackagePath) prior to EndsWith and similarly
handle updateEntry.PackageUrl (or use the null-conditional pattern already used
for PackageUrl) so the method safely returns whether the entry is a ZIP by
inspecting PackageFormat, PackagePath, and PackageUrl.

---

Nitpick comments:
In @.github/workflows/push.yml:
- Line 290: Update the thrown error strings that reference the anchor download
failure to make them specific about the missing cumulative package: replace
occurrences that say "refusing to publish a cumulative-free update feed." with
wording like "refusing to publish without cumulative package from anchor
{anchorTag}" (or {anchorVersion} if available) so the message clearly indicates
the cumulative package for the given anchor is missing; adjust both the throw at
the location referencing anchorTag and the similar throw at the other occurrence
to use the new phrasing while keeping the anchor identifier interpolated.

In `@TelegramSearchBot.Common/Model/Update/UpdateCatalog.cs`:
- Line 14: Change the FullPackageSize property to a nullable long (long?) on the
UpdateCatalog record to match the other optional full-package metadata fields,
then adjust all usages that assume a non-null value (notably the code that reads
FullPackageSize in Program.cs and any workflow/serialization code) to handle
null explicitly—either by checking HasValue, using a null-coalescing default
where appropriate, or altering logic to treat null as "unknown" rather than
zero; update any tests or integration points expecting a default 0 accordingly.

In `@TelegramSearchBot.UpdateBuilder/Program.cs`:
- Line 590: The long usage string in Program.cs (the literal passed as the
help/usage text near the top-level usage message) is hard to read; split it into
a multi-line verbatim string or construct it from an array of shorter lines
joined with Environment.NewLine (or use StringBuilder) so each option appears on
its own line or as a bulleted list; update the code that currently contains the
single long literal (the usage string shown in Program.cs) to use the new
multi-line format so the help output and source remain readable and
maintainable.

In `@TelegramSearchBot/Service/Update/SelfUpdateBootstrap.Windows.cs`:
- Around line 877-895: GetPackageCacheFileName currently proceeds when both
UpdateCatalogEntry.PackageUrl and PackagePath are empty and only falls back to a
default filename later; add an explicit early validation in
GetPackageCacheFileName to check that at least one of entry.PackageUrl or
entry.PackagePath is not null/whitespace, and if both are empty throw an
ArgumentException (or ArgumentNullException) mentioning the invalid
UpdateCatalogEntry so callers get immediate, clear feedback; keep the existing
fallback logic for filename derivation and continue to use IsZipPackage to
decide default extension if you still need it after validation.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 77b0a5c6-24a0-467a-b93a-3fd09a866cd0

📥 Commits

Reviewing files that changed from the base of the PR and between c347dd1 and 97a3deb.

📒 Files selected for processing (14)
  • .github/workflows/push.yml
  • Docs/Build_and_Test_Guide.md
  • README.md
  • TelegramSearchBot.Common/Model/Update/UpdateCatalog.cs
  • TelegramSearchBot.Common/Model/Update/UpdateCatalogEntry.cs
  • TelegramSearchBot.Common/Model/Update/UpdateManifest.cs
  • TelegramSearchBot.Common/Model/Update/UpdatePackageFormats.cs
  • TelegramSearchBot.Test/ModerUpdateIntegrationTests.cs
  • TelegramSearchBot.Test/Service/Update/SelfUpdateBootstrapTests.cs
  • TelegramSearchBot.Test/Service/Update/UpdateBuilderTests.cs
  • TelegramSearchBot.UpdateBuilder/Program.cs
  • TelegramSearchBot/Service/Update/SelfUpdateBootstrap.Windows.cs
  • external/moder-update/src/Moder.Update/Models/UpdateCatalogEntry.cs
  • external/moder-update/src/Moder.Update/Models/UpdateManifest.cs

Comment thread TelegramSearchBot.UpdateBuilder/Program.cs Outdated
Comment thread TelegramSearchBot/Service/Update/SelfUpdateBootstrap.Windows.cs
@ModerRAS
Copy link
Copy Markdown
Owner Author

Addressed the CodeRabbit findings in e0651ba: validated --full-package-size parsing, guarded zip/package path handling, made full package size nullable, clarified anchor errors, and added regression tests for the parser and zip detection cases.

@ModerRAS ModerRAS merged commit f27c681 into master May 15, 2026
5 checks passed
@ModerRAS ModerRAS deleted the codex/slim-update-feed branch May 15, 2026 08:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant