Skip to content

fix: add tests#31

Merged
lightsing merged 5 commits intofeat/walfrom
copilot/review-journal-design-tests
Feb 26, 2026
Merged

fix: add tests#31
lightsing merged 5 commits intofeat/walfrom
copilot/review-journal-design-tests

Conversation

Copy link
Copy Markdown

Copilot AI commented Feb 26, 2026

Review of the journal/WAL design revealed a data race: write_index is bumped via CAS before data is copied into the ring buffer slot, so the WAL worker can read stale data. Additionally, the busy-wait spin loop caused thread starvation on low-core machines, and all existing tests shared a single ./wal/ directory causing parallel test failures.

Data race fix: filled_index

Added a separate filled_index atomic to JournalInner. Writers reserve a slot via write_index CAS, copy data, then advance filled_index in-order with Release ordering. The WAL worker reads filled_index (not write_index) with Acquire to determine safe-to-read entries.

// After writing data to the reserved slot:
while self.inner.filled_index.compare_exchange_weak(
    current_written,
    current_written.wrapping_add(1),
    Ordering::Release,
    Ordering::Relaxed,
).is_err() {
    std::hint::spin_loop();
}

WAL worker changes

  • Removed busy-wait spin loop — worker now parks immediately when idle, woken via unpark() on commit
  • Drains all available data before parking (removed IO_BATCH_LIMIT cap) — a burst of entries is fully persisted in one flush/sync_all pass
  • Added Drop for WalInner — sets shutdown flag and unparks worker to prevent orphaned spinning threads

Test infrastructure

  • Added tempfile dev-dep; each test gets an isolated temp directory via test_journal() helper
  • All existing tests updated to use isolated dirs and explicit shutdown() for cleanup
  • Fixed bogus matches!(err, JournalBusy) assertion (was always true — matched any value into a new binding)

New WAL tests (21 tests, 31 total)

  • Segment helpers: file naming, creation, scanning (sorted, non-wal files, invalid names)
  • Recovery: empty dir, data replay into ring buffer, respects consumed checkpoint, partial last segment truncation, incomplete non-last segment detection, consumed-ahead-of-written detection, cross-segment recovery
  • Runtime: end-to-end persistence verification, segment rotation, shutdown rejects commits, shutdown idempotent, commit future resolves after persistence, persisted_index advancement
  • Cleanup: old segment truncation

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI and others added 2 commits February 26, 2026 07:53
Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>
…move spin loop per review

- Add filled_index to JournalInner to fix data race where WAL worker could read
  slots before data was written (write_index was bumped before slot fill)
- Writers now spin-wait on filled_index CAS after writing data, ensuring
  the WAL worker only reads fully-filled slots
- Remove busy-wait spin loop from WAL worker; use park/unpark only
- WAL worker now drains ALL available data before parking (no IO_BATCH_LIMIT cap)
- Add Drop impl for WalInner to signal shutdown on drop
- Add tempfile dev-dependency for test isolation
- Fix all existing tests to use isolated temp directories via test_journal() helper
- Fix incorrect matches! assertion in commit_returns_error_when_full test
- Add shutdown() calls to tests for proper WAL worker cleanup
- Add 21 comprehensive WAL tests covering:
  - Segment file naming, creation, scanning
  - Segment rotation and old segment cleanup
  - Recovery from empty dir, with data replay, with consumed checkpoint
  - Recovery of partial last segment (truncation)
  - Detection of incomplete segments and data loss
  - Shutdown behavior (rejects commits, idempotent)
  - End-to-end WAL persistence verification
  - Cross-segment recovery

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>
Copilot AI changed the title [WIP] Review current journal design and add tests for WAL implementation Fix WAL data race, remove spin loop, add comprehensive WAL tests Feb 26, 2026
@lightsing lightsing changed the title Fix WAL data race, remove spin loop, add comprehensive WAL tests fix: add tests Feb 26, 2026
@lightsing lightsing marked this pull request as ready for review February 26, 2026 09:05
@lightsing lightsing merged commit 717817f into feat/wal Feb 26, 2026
@lightsing lightsing deleted the copilot/review-journal-design-tests branch February 26, 2026 09:05
@lightsing lightsing restored the copilot/review-journal-design-tests branch February 26, 2026 09:20
@lightsing lightsing deleted the copilot/review-journal-design-tests branch February 26, 2026 09:29
lightsing added a commit that referenced this pull request Mar 4, 2026
* impl wal

* apply review

* fix: add tests (#31)

* Initial plan

* Initial plan for WAL test coverage

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* Fix WAL race condition, add filled_index, comprehensive WAL tests, remove spin loop per review

- Add filled_index to JournalInner to fix data race where WAL worker could read
  slots before data was written (write_index was bumped before slot fill)
- Writers now spin-wait on filled_index CAS after writing data, ensuring
  the WAL worker only reads fully-filled slots
- Remove busy-wait spin loop from WAL worker; use park/unpark only
- WAL worker now drains ALL available data before parking (no IO_BATCH_LIMIT cap)
- Add Drop impl for WalInner to signal shutdown on drop
- Add tempfile dev-dependency for test isolation
- Fix all existing tests to use isolated temp directories via test_journal() helper
- Fix incorrect matches! assertion in commit_returns_error_when_full test
- Add shutdown() calls to tests for proper WAL worker cleanup
- Add 21 comprehensive WAL tests covering:
  - Segment file naming, creation, scanning
  - Segment rotation and old segment cleanup
  - Recovery from empty dir, with data replay, with consumed checkpoint
  - Recovery of partial last segment (truncation)
  - Detection of incomplete segments and data loss
  - Shutdown behavior (rejects commits, idempotent)
  - End-to-end WAL persistence verification
  - Cross-segment recovery

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* some fix

* update docs

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>
Co-authored-by: lightsing <light.tsing@gmail.com>

* apply reviews

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>
lightsing added a commit that referenced this pull request Mar 4, 2026
* add submit_digest poc impl

* refactor: timestamp rewrite (#2)

* add builder

* add builder

* rewrite

* rewrite Attestation

* clippy

* return when ok

* fix

* full integration with allocator API

* fmt

* apply review

* apply review

* feat: proof builder (#9)

* complete

* fmt

* clippy

* apply review

* apply review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* typo

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* feat: add submit_digest bench (#10)

* add bench

* apply review

* feat: high performace journal mvp (#6)

* wip

* mvp

* add tests

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* update README

* add stamper

* fix wait_at_least

* fix merge

* fmt

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* feat: link journal (#11)

* feat: solidity contracts (#14)

* init foundry workspace

* add contracts crate

* apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* wip: ts lib & web (#17)

* fix ignore file

* pnpm init vue-ts

* add poc

* add APIs

* wip: stamper (#8)

* wip

* mvp

* add tests

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* update README

* add stamper

* fix wait_at_least

* fix merge

* add stamper

* fmt

* apply review

* add stamper to calendar server

* add contract

* update lock

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* feat: verifier (#21)

* embed abi

* add Ethereum UTS tag

* add Ethereum UTS tag

* add verifier

* refactor

* deploy default instance

* fmt

* feat: support upgrade (#22)

* feat: add verifier (#23)

* add catch

* fix: domain seperation inner node hash (#24)

* feat: cli crate (#25)

* move to cli crate

* add stamp command

* fix

* add upgrade

* Create README.md for UTS CLI tool

Added README.md for UTS CLI tool with usage instructions and command reference.

* feat: ts sdk (#26)

* init

* use noble

* add encoder

* add decoder

* add tests

* add sdk

* complete sdk

* fix

* apply review

* ignore lock file

* add eslint

* apply review

* update readme

* chore: setup monorepo framework: rollup SDK build, vite web app, lint/fmt/test orchestration (#27)

* Initial plan

* setup proper monorepo framework: rollup for SDK, vite for web, lint/fmt/test/build scripts

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* plan: tsc declaration-only emit + .ts import suffixes + eslint-plugin-import-x

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* tsc declaration-only emit, .ts import suffixes, add eslint-plugin-import-x

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* downgrade eslint from ^10.0.2 to ^9.38.0 to fix peer dep conflict with eslint-plugin-import-x

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* update

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>
Co-authored-by: Jack <Jack>

* feat(web): DApp for UniversalTimestamps (#28)

* Initial plan

* Install web dependencies: pinia, vueuse, tailwindcss, lucide-vue-next, date-fns

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* feat(web): build cyberpunk DApp UI with file digest, stamping, verification, and live feed

- Configure Vite with path aliases and Tailwind CSS v4
- Create cyberpunk theme (neon colors, glassmorphism, scanlines, animations)
- Add Pinia store for calendar status and recent stamps
- Add useFileDigest composable: pick file from local FS and compute SHA-256/Keccak-256
- Add useTimestampSDK composable: stamp, verify, upgrade, decode .ots files
- Add useWebSocketFeed composable: simulated real-time attestation feed
- Create base components: GlassCard, BaseButton, StatusBadge
- Create HeroTerminal with drag-and-drop file picker and hash input
- Create StampingWorkflow with step-by-step pipeline visualization
- Create VerificationResult with .ots upload and MerkleTreeViz (recursive)
- Create LiveFeed with real-time scrolling entries
- Create HomeView assembling all components with tab navigation
- Export SDK class, DEFAULT_CALENDARS, enums as runtime values from SDK index

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* webapp don't need to build sdk

* fix naming

* feat(web): ethereum-first UI with real feed data from ethRPCs

- Replace mock LiveFeed with real Ethereum Attested event polling via SDK ethRPCs
- Show Ethereum chain RPC statuses in header instead of Bitcoin calendar nodes
- Update subtitle to "anchored to Ethereum"
- Expose singleton SDK instance via getSDK() for shared access
- Bitcoin attestation verification still fully supported in Verify tab

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* feat: calendar config, real stamping progress, .ots download, upgrade polling

SDK changes:
- Add StampEvent/StampEventCallback types for real-time stamp progress
- SDK.stamp() now accepts optional onEvent callback, fires events for
  each phase: generating-nonce, building-merkle-tree, broadcasting,
  calendar-response (per calendar), building-proof, complete

Web app changes:
- Add calendar URL configuration panel (settings gear in header)
  with add/remove/reset-to-defaults, persisted to localStorage
- useTimestampSDK wires SDK callbacks to drive real pipeline phases
- Download .ots file automatically on stamp completion via Encoder
- Poll sdk.upgrade() every 15s after stamping for attestation upgrade
- StampingWorkflow shows real phases including broadcast progress,
  building-proof, upgrading, and upgraded states
- Pass fileName through stamp flow for meaningful .ots download names

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* feat: add web3 wallet provider support for browser usage

SDK changes:
- Add web3Provider (Eip1193Provider) field to SDKOptions and SDK class
- Add WELL_KNOWN_CHAINS map for wallet_switchEthereumChain
- Add getWeb3ProviderForChain() helper that tries wallet first, with
  automatic chain switching for well-known networks
- verifyEthereumUTSAttestation now tries web3Provider first (no CORS),
  then falls back to ethRPCs
- Export WELL_KNOWN_CHAINS from SDK index

Web app changes:
- Create useWallet composable: connect/disconnect MetaMask via EIP-1193,
  track address/chainId, listen for account/chain changes
- Add setWeb3Provider() to useTimestampSDK for hot-swapping wallet
- useWebSocketFeed prefers web3Provider for event polling when connected,
  falls back to ethRPCs when no wallet
- Add Connect Wallet button in header (shows address + chain when
  connected, "No Wallet" when no EIP-1193 provider detected)
- Wallet provider synced to SDK via watch on connection state

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* feat: manual .ots download, keepPending setting, attestation details, tab persistence, deduped feed

SDK changes:
- upgrade() accepts keepPending param (default false): when false, replaces
  pending attestation with upgraded one; when true, preserves both in a FORK

Web app changes:
- Don't auto-download pending timestamp; show "Download .ots" button in
  StampingWorkflow when stamping completes
- Upgraded .ots uses original upload filename (e.g. document.pdf.ots)
- Add keepPending toggle in settings panel, persisted to localStorage
- Create AttestationDetail.vue: collapsible attestation info with etherscan
  links for blocks, addresses, tx hashes across all supported chains
- VerificationResult uses AttestationDetail for each attestation
- LiveFeed entries are clickable/collapsible with chain/block/sender details
  and etherscan links
- Tabs use v-show instead of v-if to preserve content when switching
- Active tab persisted to localStorage
- Feed polls both web3Provider AND ethRPCs simultaneously, skipping chains
  already covered by wallet; deduplicates entries by id (chainId-block-index)

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* fix: correct Scroll chain IDs, delay hashing, wider feed, default keccak, directory stamp

SDK: Fix Scroll chain IDs from 54351/54352 to 534351/534352 with correct
hex values 0x8274f/0x82750 matching the Rust CLI and actual chain IDs.

Web: Delay file hashing until Stamp button is pressed (not on file select).
Add "Choose Directory" button for batch stamping entire directories.
Batch stamps download all .ots files as a zip archive.
Default file hash algorithm changed to Keccak-256.
Add internal hash algorithm setting (for Merkle tree construction).
Widen Live Feed panel from 1/3 to 2/5 grid for more horizontal space.
All settings persisted to localStorage.

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* fix: batch stamp download, correct filenames with dir structure, download button in Complete step

- Don't auto-download pending timestamps on stamp completion
- Manual "Download pending .ots" button inside the Complete step only
  (hidden once phase moves to upgrading/upgraded)
- Batch stamps download as zip preserving directory structure
  (webkitRelativePath used for path-aware filenames)
- Each .ots in zip named after its input file (e.g. mydir/file.txt.ots)
- Upgraded timestamps also download as zip for batch with correct names
- Single file stamps download as individual .ots (same behavior as before)
- Track all file names in stampFileNames array, not single stampFileName
- Remove JSZip/Encoder imports from HomeView (moved to composable)

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* feat: download button fix, fake progress, verify original file, attestation details in proof path, upgrade tab, Scroll logo, testnet warnings, clickable chain panel

StampingWorkflow: Download pending .ots button visible during both
'complete' and 'upgrading' phases (hidden once upgraded or idle).

useFileDigest: Fake progress bar estimates time based on file size
(~200 MB/s) instead of real streaming progress. Added digestFiles()
batch method.

VerificationResult: Optional original file upload to verify digest
matches the .ots header. Shows match/mismatch status with icons.

MerkleTreeViz: ATTESTATION steps now collapsible with full details
(type, chain, block height, contract, tx hash with etherscan links).

AttestationDetail/MerkleTreeViz/LiveFeed: Warning flag (⚠) for
testnet or unknown network attestations. Mainnet = Ethereum (1) and
Scroll (534352). All others show orange warning banner.

LiveFeed: Added txHash to feed entries from log.transactionHash.
Scroll logo shown for Scroll chain entries.

AttestationDetail: Scroll logo in header and expanded type row.

UpgradePanel: New tab for manual .ots upgrade with verify, upgrade
now button, and download upgraded result.

HomeView: Clickable "X/Y chains" in header opens dropdown showing
each chain with name, ID, status, latency, Scroll logo. Add custom
chain by ID, remove individual chains, reset to defaults. Persisted
to localStorage.

stores/app: addChain/removeChain/resetChains with localStorage
persistence. Public RPC fallbacks for common chains.

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* fix: chain panel z-index over content, add RPC endpoint editing per chain

- Add `relative z-50` to header so chain dropdown renders above terminal
  and live feed content
- Add customRpcs storage (localStorage persisted) for per-chain RPC URLs
- Chain panel shows current RPC URL for each chain with pencil edit button
- Click pencil to inline-edit RPC URL, Enter or check to save
- Custom RPCs take priority over SDK providers and public RPC fallbacks
- Reset defaults clears custom RPCs along with chain list

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* use scroll logo

* fix paths

* add cors header

* allow to modify rpc endpoints

* remove scroll logo from overview

* run prettier

* fix

* adjust space

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>
Co-authored-by: lightsing <light.tsing@gmail.com>

* Initial plan

* feat(web): add i18n support using @lingui/core with Vue integration

- Install @lingui/core, @lingui/cli, @lingui/vite-plugin, @lingui/format-po
- Create lingui.config.ts for locale configuration (en, zh)
- Create src/i18n.ts for i18n instance setup with locale persistence
- Create src/composables/useLingui.ts Vue composable for reactive translations
- Add .po module type declarations (src/env.d.ts)
- Create English (en.po) and Chinese (zh.po) message catalogs
- Update vite.config.ts with lingui Vite plugin
- Wrap all hardcoded UI text across all components with t()
- Add language switcher button (Languages icon) in header

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* fix(web): use generateMessageId for correct Lingui catalog key matching

- Add @lingui/message-utils dependency for generateMessageId
- Hash message IDs in useLingui composable to match compiled catalog keys
- Move locale switch handler to named function in HomeView
- Sync _locale ref with i18n on composable init

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* more strict regex

* remove unused

* fix: bmt tests forget update after domain seperation (#32)

* feat: real wal (#30)

* impl wal

* apply review

* fix: add tests (#31)

* Initial plan

* Initial plan for WAL test coverage

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* Fix WAL race condition, add filled_index, comprehensive WAL tests, remove spin loop per review

- Add filled_index to JournalInner to fix data race where WAL worker could read
  slots before data was written (write_index was bumped before slot fill)
- Writers now spin-wait on filled_index CAS after writing data, ensuring
  the WAL worker only reads fully-filled slots
- Remove busy-wait spin loop from WAL worker; use park/unpark only
- WAL worker now drains ALL available data before parking (no IO_BATCH_LIMIT cap)
- Add Drop impl for WalInner to signal shutdown on drop
- Add tempfile dev-dependency for test isolation
- Fix all existing tests to use isolated temp directories via test_journal() helper
- Fix incorrect matches! assertion in commit_returns_error_when_full test
- Add shutdown() calls to tests for proper WAL worker cleanup
- Add 21 comprehensive WAL tests covering:
  - Segment file naming, creation, scanning
  - Segment rotation and old segment cleanup
  - Recovery from empty dir, with data replay, with consumed checkpoint
  - Recovery of partial last segment (truncation)
  - Detection of incomplete segments and data loss
  - Shutdown behavior (rejects commits, idempotent)
  - End-to-end WAL persistence verification
  - Cross-segment recovery

Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

* some fix

* update docs

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>
Co-authored-by: lightsing <light.tsing@gmail.com>

* apply reviews

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lightsing <15951701+lightsing@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants