Skip to content

Chain metadata aggregator PR3 - Loaders#1706

Draft
dioptx wants to merge 33 commits intomainfrom
chain-metadata-aggregator-pr3
Draft

Chain metadata aggregator PR3 - Loaders#1706
dioptx wants to merge 33 commits intomainfrom
chain-metadata-aggregator-pr3

Conversation

@dioptx
Copy link
Contributor

@dioptx dioptx commented Jul 22, 2025

Manual mappings and metadata:

  • Added mappings in resources/manual_chain_mappings.csv for chain ID conflicts, display names, and provider entities.

New ingestion assets:

  • Introduced daily assets (l2beat_daily, defillama_daily, dune_daily, bq_op_stack_daily, bq_goldsky_daily) in chain_metadata.py for fetching and deduplicating chain data from sources like L2Beat, DefiLlama, Dune, and BigQuery.
  • Added aggregated_daily to combine data for daily partitioning and validation.

CLI updates:

  • Improved build_metadata command in app.py for usability and logging, replacing the old command.
  • Created chains_app Typer application for chain commands.

Testing:

  • Added scripts/test_real_loaders.py for testing chain metadata with live data.

Code organization:

  • Improved imports and modularity in app.py by reorganizing modules.

Summary: Updated mappings, ingestion assets, CLI, testing, and code structure for improved data management and usability.

Architecture:

image

@dioptx dioptx self-assigned this Jul 22, 2025
Copy link
Collaborator

@gpgarner gpgarner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

think we need to look at the csv, if it needs to be added

and also just decide if a dtype check is necessary on the column

@dioptx dioptx requested a review from gpgarner August 1, 2025 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants