feat: add network column to schema for multi-network support by silent-cipher · Pull Request #463 · FilOzone/dealbot

silent-cipher · 2026-04-22T11:09:22Z

Summary

Introduces a network column to the core runtime tables so a single dealbot instance can safely operate across both calibration and mainnet without collisions between same-address providers/deals on different chains.

Step towards #297

Copilot

Pull request overview

Adds network scoping to core runtime tables so a single backend instance can operate against both calibration and mainnet without address/deal collisions.

Changes:

Introduces network columns and updates PK/FK/unique constraints to be network-aware (migration + entity updates).
Updates several upsert/conflict targets to include network for correct deduplication across chains.
Adds SUPPORTED_NETWORKS constant and updates unit tests to reflect conflict key changes.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
apps/backend/src/database/migrations/1776790420000-AddNetworkColumn.ts	Adds `network` columns, updates PK/FK/unique constraints and supporting indexes for multi-network isolation.
apps/backend/src/database/entities/storage-provider.entity.ts	Makes `StorageProvider` primary key composite `(address, network)`.
apps/backend/src/database/entities/deal.entity.ts	Adds `network` column and updates SP relation join to `(sp_address, network)`.
apps/backend/src/database/entities/job-schedule-state.entity.ts	Adds `network` column (but currently leaves unique index definition misaligned with new schema).
apps/backend/src/database/entities/data-retention-baseline.entity.ts	Makes baseline primary key composite `(provider_address, network)`.
apps/backend/src/wallet-sdk/wallet-sdk.service.ts	Updates provider upsert conflict target to include `network`.
apps/backend/src/wallet-sdk/wallet-sdk.service.spec.ts	Updates unit test to expect network-aware conflict paths.
apps/backend/src/jobs/repositories/job-schedule.repository.ts	Updates job schedule upsert conflict target to include `network`.
apps/backend/src/data-retention/data-retention.service.ts	Updates baseline upsert conflict columns to include `network`.
apps/backend/src/data-retention/data-retention.service.spec.ts	Updates unit test to expect network-aware conflict columns.
apps/backend/src/common/constants.ts	Adds `SUPPORTED_NETWORKS` constant used by the migration.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-22T11:14:20Z

        faultedPeriods: "10",
        successPeriods: "90",
        lastBlockNumber: "1200",
      },
-      ["providerAddress"],
+      ["providerAddress", "network"],


This test updates the conflict columns to include network, but it still asserts an upsert payload that omits network. To prevent cross-network baseline collisions, update the test to expect network to be set on the persisted baseline (using the configured blockchain network).

agree this needs fixed.

Copilot · 2026-04-22T11:14:20Z

      INSERT INTO job_schedule_state (job_type, sp_address, interval_seconds, next_run_at)
      VALUES ($1, $2, $3, $4)
-      ON CONFLICT (job_type, sp_address) DO UPDATE
+      ON CONFLICT (job_type, sp_address, network) DO UPDATE
      SET interval_seconds = EXCLUDED.interval_seconds,
          paused = job_schedule_state.paused,


This schedule upsert uses ON CONFLICT (job_type, sp_address, network) but the INSERT statement doesn’t provide a network value, so every row will be written under the DB default network. That prevents separate schedules per chain in multi-network mode. Include network in the INSERT columns/values and accept/pass it as a parameter (or otherwise derive it) so schedules are stored per network.

I believe this still neds fixed

Copilot · 2026-04-22T11:14:21Z

        faultedPeriods: baseline.faultedPeriods.toString(),
        successPeriods: baseline.successPeriods.toString(),
        lastBlockNumber: blockNumber.toString(),
      },
-      ["providerAddress"],
+      ["providerAddress", "network"],


persistBaseline() now upserts on (providerAddress, network) but the upsert payload does not include network. That means every baseline row will be written under the default network, and baselines from different chains will collide/overwrite. Add network to the upserted entity (from the configured blockchain network) and ensure any in-memory baseline map keys are network-scoped as well so deltas don’t mix across chains.

agree this needs fixed.

SgtPooki

Looks like there are just a few more gaps, and we need to make sure runtime reads pull from the correct network-scoped records. Pass network in the relevant DB read functions and make sure it’s handled in the appropriate repository/query logic.

SgtPooki · 2026-04-27T13:09:14Z

        faultedPeriods: "10",
        successPeriods: "90",
        lastBlockNumber: "1200",
      },
-      ["providerAddress"],
+      ["providerAddress", "network"],


agree this needs fixed.

SgtPooki · 2026-04-27T13:09:28Z

        faultedPeriods: baseline.faultedPeriods.toString(),
        successPeriods: baseline.successPeriods.toString(),
        lastBlockNumber: blockNumber.toString(),
      },
-      ["providerAddress"],
+      ["providerAddress", "network"],


agree this needs fixed.

SgtPooki · 2026-04-27T13:11:08Z

+  @PrimaryColumn({ name: "network", type: "text" })
+  network!: Network;
+


can we use an enum instead?

Suggested change

@PrimaryColumn({ name: "network", type: "text" })

network!: Network;

@PrimaryColumn({

name: "network",

type: "enum",

enum: Network,

})

network!: Network;

SgtPooki · 2026-04-27T13:11:43Z

+  @Column({ name: "network", type: "text" })
+  network: Network;


ditto for enum if possible

SgtPooki · 2026-04-27T13:12:00Z

+  @Column({ name: "network", type: "text" })
+  network!: Network;


ditto for enum if possible

SgtPooki · 2026-04-27T13:12:58Z

+  @PrimaryColumn({ name: "network", type: "text" })
+  network!: Network;


use enum if possible

SgtPooki · 2026-04-27T13:13:56Z

+ * Backfill strategy: existing rows are assigned the network declared by the
+ * operator via `DEALBOT_LEGACY_NETWORK_BACKFILL` (preferred) or the legacy
+ * `NETWORK` env var. The migration fails fast if neither is set to a supported
+ * network, so backfill is never silently wrong. Operators later expanding a
+ * single-network deployment to an additional network must update their
+ * `NETWORKS` config and re-run a providers_refresh to populate the
+ * network-scoped rows for the newly added network.


can we get a specific runbook for operators to use to handle the migration to multi-network?

SgtPooki

I couldn't see anything wrong at first glance, so I threw gemini + claude + codex at it and they uncovered potential gaps.. I'm testing locally and will get back with findings.

SgtPooki · 2026-05-06T14:32:53Z

  private mapJobPayload(row: ScheduleRow): SpJobData | ProvidersRefreshJobData | DataRetentionJobData {
    if (
      row.job_type === "deal" ||
      row.job_type === "retrieval" ||
      row.job_type === "data_set_creation" ||
      row.job_type === "piece_cleanup"
    ) {
      return { jobType: row.job_type, spAddress: row.sp_address, intervalSeconds: row.interval_seconds };
    }
    return { intervalSeconds: row.interval_seconds };
  }


we likely need network here as well

SgtPooki · 2026-05-06T14:33:19Z

      if (isSpJobType(jobType)) {
        const spData = data as SpJobData;
        if (!finalOptions.singletonKey) {
          finalOptions.singletonKey = spData.spAddress;


we likely need to update singleTonKey to spAddress + network?

silent-cipher · 2026-05-06T15:33:19Z

I'm finding it a bit tricky to define the scope of this pr and where to draw the boundary for changes. My initial thought was to split the work between this pr and #468 so we could roll out smaller prs. Would you prefer that approach, or should I instead consolidate everything from #468 into this pr and make it a single change?

SgtPooki · 2026-05-06T16:09:13Z

@silent-cipher yea, i think the idea of a schema-only PR change would be ideal, and keeping PRs small, but this PR already does some application level piping through of data. as a minimum, I think we need #510 on top of this (or similar if you recommend differently)

I don't want to disrupt you too much, so let me know what you think would be best

silent-cipher · 2026-05-06T17:02:20Z

I’m thinking of splitting this into three PRs to keep things small and backward compatible -

a schema-focused pr(this one), including changes from fix(network): close multi-network gaps before #297 expansion #510, and I’ll verify if anything else is needed there
a pr to update prometheus metrics with the network label
a final pr for env/config schema updates along with the relevant use cases and docs

feat: add network column to schema

88e1523

Copilot AI review requested due to automatic review settings April 22, 2026 11:09

FilOzzy added this to FOC Apr 22, 2026

github-project-automation Bot moved this to 📌 Triage in FOC Apr 22, 2026

Copilot started reviewing on behalf of silent-cipher April 22, 2026 11:09 View session

fix: typecheck

492caf1

Copilot AI reviewed Apr 22, 2026

View reviewed changes

rjan90 requested a review from SgtPooki April 22, 2026 11:29

rjan90 moved this from 📌 Triage to 🔎 Awaiting review in FOC Apr 22, 2026

rjan90 added this to the M4.5: GA Fast Follows milestone Apr 22, 2026

silent-cipher added 3 commits April 22, 2026 17:13

fix: align typeorm with migration

4c163ce

fix: correct migration rollback order

166a1b3

chore: fix migration comment

b4e9f0c

silent-cipher self-assigned this Apr 23, 2026

SgtPooki requested changes Apr 27, 2026

View reviewed changes

github-project-automation Bot moved this from 🔎 Awaiting review to ⌨️ In Progress in FOC Apr 27, 2026

SgtPooki mentioned this pull request Apr 28, 2026

chore: add Copilot PR review instructions #489

Merged

4 tasks

BigLep moved this from ⌨️ In Progress to 🐱 Todo in FOC May 3, 2026

chore: address pr comments

4178902

SgtPooki reviewed May 6, 2026

View reviewed changes

SgtPooki mentioned this pull request May 6, 2026

fix(network): close multi-network gaps before #297 expansion #510

Open

		@PrimaryColumn({ name: "network", type: "text" })
		network!: Network;

		@Column({ name: "network", type: "text" })
		network!: Network;

Conversation

silent-cipher commented Apr 22, 2026

Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SgtPooki left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SgtPooki left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

silent-cipher commented May 6, 2026

Uh oh!

SgtPooki commented May 6, 2026

Uh oh!

silent-cipher commented May 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants