Feat: use sdk's next version#3
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR modernizes the codebase by upgrading to Synapse SDK's next version and converting the project to use ES modules. The changes include automatic wallet management for storage payments, integration with Kaggle datasets, and updates to deal creation workflow to accommodate SDK changes.
Key changes:
- Conversion to ES modules with
.jsfile extensions in imports - Upgrade to Synapse SDK v0.24.0-dev1 with updated API calls
- Addition of automatic wallet funding and service approval functionality
Reviewed Changes
Copilot reviewed 38 out of 39 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| package.json | Updates SDK version and enables ES module support |
| src/scheduler/scheduler.service.ts | Adds wallet management logic for automatic deposits and approvals |
| src/deal/deal.service.ts | Updates deal creation to use new SDK APIs and Kaggle data source |
| src/dataSource/dataSource.service.ts | Implements Kaggle dataset fetching with fallback to local datasets |
| Multiple module files | Converts all imports to use .js extensions for ES module compatibility |
Files not reviewed (1)
- pnpm-lock.yaml: Language not supported
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| } | ||
|
|
||
| return buffer; | ||
| throw new Error(`Failed to construct kaggle dataset download url for dataset ${dataset.ref}`); |
There was a problem hiding this comment.
The error message should provide more context about why the URL construction failed, such as mentioning missing required fields or invalid dataset structure.
Suggested change
| throw new Error(`Failed to construct kaggle dataset download url for dataset ${dataset.ref}`); | |
| throw new Error( | |
| `Failed to construct kaggle dataset download url. ` + | |
| `Dataset details: ref=${JSON.stringify(dataset.ref)}, ` + | |
| `url=${JSON.stringify(dataset.url)}, ` + | |
| `urlNullable=${JSON.stringify(dataset.urlNullable)}, ` + | |
| `hasUrl=${JSON.stringify(dataset.hasUrl)}. ` + | |
| `Possible reasons: missing or invalid 'ref', 'url', or 'urlNullable' fields.` | |
| ); |
This was referenced Apr 27, 2026
SgtPooki
added a commit
that referenced
this pull request
May 5, 2026
* docs(checks): close resolved TBDs in data-storage, events, README Items previously marked TBD that are now implemented in code: - data-storage.md assertions #3, #5, #6, #7 (pieceConfirmed, IPNI discoverability, retrievability, all-checks-gated) -> Yes. - data-storage.md poll intervals: replace TBD_VARIABLE refs with the concrete sources (hardcoded POLLING_INTERVAL_MS = 2.5s for SP piece status, IPNI_VERIFICATION_POLLING_MS env var with 2s default for IPNI verification; doc previously claimed 5s). - data-storage.md section 7 header drops TBD; intro disclaimer removed. - data-storage.md "TBD Summary" rewritten as "Implementation History" with code references for inline retrieval, CID integrity, per-deal timeout (AbortController -> DealStatus.FAILED), gated status, status model, onPieceConfirmed, IPFS gateway retrieval, filecoin-pin CAR. - events-and-metrics.md: pieceConfirmed -> Yes (pieceConfirmedOnChainMs histogram); ipfsRetrievalIntegrityChecked -> implemented inline via per-block sha256 verification in ipfs-block.strategy.ts (no discrete event); ipfsRetrievalFirstByte/LastByteReceived marked Partial since duration histograms exist but no discrete event; histogram-buckets TBD replaced with link to metrics-prometheus.module.ts. - README.md: name the dataset-creation job (data-set-creation) and reference its config envs. Still TBD (not changed in this commit): uploadToSpStart, ipniVerificationStart, ipfsRetrievalStart events; jobs.md PR #263 lookahead-skip; PDP_SUBGRAPH_ENDPOINT production value. * docs(checks): address review feedback on callback names and event states - data-storage.md: rename Synapse callbacks to plural form (onPiecesAdded, onPiecesConfirmed) to match deal.service.ts. - events-and-metrics.md: same rename in the event list. Clarify that dealCreated maps to DealStatus.DEAL_CREATED only after all gates pass (upload alone sets UPLOADED, not DEAL_CREATED). - events-and-metrics.md: ipfsRetrievalIntegrityChecked downgraded from Yes to Partial since no discrete event is emitted (inline check only). - events-and-metrics.md: Mermaid timeline now matches the table - ipfsRetrievalFirstByteReceived/LastByteReceived labelled as "Partial: histogram only", ipfsRetrievalIntegrityChecked labelled "Partial: inline check, no event". - README.md: refer to the canonical pg-boss job type data_set_creation (underscore) so operators can map the doc to jobType values. * docs(checks): fix unreadable Mermaid rect fill in event timeline The 'Data Storage Only' rect used rgb(50, 50, 50), which renders as a near-black block that hides the message labels and arrows inside it (both on GitHub light/dark themes). Switch to a translucent rgba(120, 120, 200, 0.15) so the highlight is visible without obscuring content. * docs(events): reframe Event List as timing markers, not emitted events The 'events' in this doc are named anchors used to define metric Timer Starts/Ends; dealbot does not necessarily emit each as a discrete Prometheus event or log line. Add an explicit note up top so readers don't expect every entry to map to an emitted event, and update rows that were marked TBD/Partial purely because no discrete event is emitted. - uploadToSpStart -> Yes (anchor: deal.uploadStartTime in deal.service.ts:255). - ipniVerificationStart -> Yes (anchor: ipniVerificationStartTime in ipni-verification.service.ts:63 - drives ipniVerifyMs). - ipfsRetrievalStart -> Yes (anchor: retrieval startTime in retrieval-addons.service.ts:227; logs 'retrieval_started'). - ipfsRetrievalFirstByteReceived -> Yes (drives ipfsRetrievalFirstByteMs). - ipfsRetrievalLastByteReceived -> Yes (drives ipfsRetrievalLastByteMs). - ipfsRetrievalIntegrityChecked -> Yes (per-block sha256 in ipfs-block.strategy.ts; inline, no discrete event). - Mermaid timeline: drop the (TBD) / (Partial: ...) annotations on these markers so the diagram and the table agree. * docs(events): drop Implemented column from Event List All rows are now Yes (each marker is anchored in code), so the column adds no signal. Anchor details folded into the Source-of-truth column. Intro note tightened. * Update docs/checks/data-storage.md Co-authored-by: Puspendra Mahariya <95584952+silent-cipher@users.noreply.github.com> * Update docs/checks/README.md Co-authored-by: Puspendra Mahariya <95584952+silent-cipher@users.noreply.github.com> --------- Co-authored-by: Puspendra Mahariya <95584952+silent-cipher@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Includes -