feat(core): use metaprogramming for tuple and fixed size array Introspect and DojoStore

[remybar]

Description

For tuple and fixed size arrays serialization with Serde, Starkware has implemented some metaprogramming mechanisms to support at most tuples/fixed size arrays of 16 elements. See:

• https://github.com/starkware-libs/cairo/blob/main/corelib/src/metaprogramming.cairo
• https://github.com/starkware-libs/cairo/blob/main/corelib/src/tuple.cairo

To have a better support of tuples serialization using DojoStore and to simplify Dojo macros code, it would be nice to reuse this metaprogramming mechanisms for DojoStore. Unfortunately, they are not visible outside the Cairo corelib.
So, this PR copy/paste metaprogrammin.cairo and tuple.cairo files from the Starkware repo and use them to manage tuple serialisation with DojoStore.

Thanks to that, Option with tuples (like Option<(u8, u16)>) are now fully supported in Dojo.

Tests

• Yes
• No, because they aren't needed
• No, because I need help

Added to documentation?

• README.md
• Dojo Book
• No documentation needed

Checklist

• I've formatted my code (scripts/rust_fmt.sh, scripts/cairo_fmt.sh)
• I've linted my code (scripts/clippy.sh, scripts/docs.sh)
• I've commented my code
• I've requested a review after addressing the comments

Summary by CodeRabbit

• New Features

  • Added support for registering and upgrading external contracts, including related events and permission checks.
  • Introduced new serialization and deserialization utilities for world storage, supporting advanced data types and layouts.
  • Added comprehensive test suites for access control, contract upgrades, event and model registration, and storage layouts.
  • Provided new testing libraries and utilities for streamlined contract and world testing.

• Improvements

  • Updated toolchain and dependencies to newer versions for improved compatibility.
  • Refactored and unified command-line workflows, metadata handling, and feature management.
  • Enhanced error messages, event handling, and test setup for greater clarity and robustness.

• Removals

  • Removed legacy macro plugins, derive macros, and attribute macros from the language plugin.
  • Deleted obsolete test data, fixtures, and configuration files.
  • Cleaned up redundant or deprecated utility functions and internal modules.

• Bug Fixes

  • Fixed compatibility and layout issues for legacy models and events.
  • Addressed edge cases in storage serialization, deserialization, and layout handling.

• Style

  • Improved code organization, import ordering, and formatting across multiple modules.

• Documentation

  • Updated and reorganized documentation and module exports for better discoverability.

[coderabbitai]

Caution

Review failed

Failed to post review comments.

Walkthrough

Ohayo sensei! This update overhauls Dojo's core and CLI by removing the entire dojo-lang plugin and language server, refactoring the Sozo CLI and related crates to use a new metadata abstraction instead of Scarb workspaces, updating core contracts for external contract management, and restructuring tests and dependencies. Major test and macro files are deleted or replaced, and new support is added for external contracts in the world contract.

Changes

File(s) / Group	Change Summary
.github/workflows/*, .tool-versions	Rust, Scarab, and tool versions updated in CI workflows and toolchain files.
Cargo.toml, bin/sozo/Cargo.toml, crates/dojo/bindgen/Cargo.toml	Workspace members and dependencies updated, renamed, or removed.
bin/dojo-language-server/*, crates/dojo/lang/*	Entire Dojo language server and plugin codebase deleted, including attribute macros, derive macros, inline macros, and plugin infrastructure.
bin/sozo/src/args.rs, bin/sozo/src/main.rs, bin/sozo/src/features.rs	CLI argument handling refactored; version flag made explicit; main function and CLI logic made async.
bin/sozo/src/commands/*	All command modules refactored to use new Metadata abstraction instead of Scarb workspace/config; many methods made async; dev and account commands removed; build, test, and migration flows simplified and updated.
bin/sozo/tests/*	Test modules, fixtures, and utilities removed or updated for new structure.
crates/dojo/core/*	Major refactor: new modules for DojoStore, metaprogramming, tuple utilities, and default values; event and model storage logic updated; support for external contracts added in world contract and resource enums; derive macros and plugin dependencies removed.
crates/dojo/core-tests/*, crates/dojo/dojo-cairo-test/*, crates/dojo/dojo-snf-test/*	Core test suites restructured, new helpers and test modules added for storage, layout, ACL, events, models, and external contracts; legacy and new storage layouts tested; test resource management updated.
crates/dojo/lang/src/attribute_macros/*, crates/dojo/lang/src/derive_macros/*, crates/dojo/lang/src/inline_macros/*, crates/dojo/lang/Scarb.toml	All attribute, derive, and inline macro implementations and their supporting files deleted.
crates/dojo/core-cairo-test/*	Entire core-cairo-test crate and its test files deleted.
crates/dojo/core/src/world/world_contract.cairo, crates/dojo/core/src/world/iworld.cairo, crates/dojo/core/src/world/resource.cairo	External contract registration and upgrade support added, with new events and resource types.
crates/dojo/core/src/model/*, crates/dojo/core/src/event/*, crates/dojo/core/src/storage/*, crates/dojo/core/src/utils/*	Model, event, and storage logic refactored for new serialization, layout, and metaprogramming support; legacy and new storage layouts handled.
crates/dojo/core/src/meta/layout.cairo	Function added to build legacy layouts from new storage layouts.
crates/dojo/core/src/meta/introspect.cairo	Option enum variant selectors updated for compatibility.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant Sozo CLI
    participant Metadata
    participant World Contract

    User->>Sozo CLI: Run command (build, test, migrate, etc.)
    Sozo CLI->>Metadata: Load project metadata from Scarb.toml
    Sozo CLI->>Sozo CLI: Parse CLI args, handle features
    Sozo CLI->>World Contract: (If applicable) Build, deploy, or interact via new metadata abstraction
    World Contract-->>Sozo CLI: Respond with results/events (including external contract support)
    Sozo CLI-->>User: Output results

Loading

sequenceDiagram
    participant User
    participant Sozo CLI
    participant Metadata
    participant External Contract

    User->>Sozo CLI: Register or upgrade external contract
    Sozo CLI->>Metadata: Load project metadata
    Sozo CLI->>World Contract: Call register_external_contract or upgrade_external_contract
    World Contract-->>Sozo CLI: Emit ExternalContractRegistered or ExternalContractUpgraded event
    Sozo CLI-->>User: Confirm registration/upgrade

Loading

Possibly related PRs

• dojoengine/dojo#2337: Adds the inline macro plugin for get_models_test_class_hashes, which is removed in this PR.
• dojoengine/dojo#3195: Adds external contract registering and upgrading features in the world contract, directly related to the new external contract support in this update.
• dojoengine/dojo#2595: Major release removing the Dojo compiler plugin and rewriting Sozo, aligning with this PR's removal of the Dojo plugin and restructuring.

Suggested labels

contributor

✨ Finishing Touches

• 📝 Generate Docstrings

🧪 Generate Unit Tests

• Create PR with Unit Tests
• Post Copyable Unit Tests in Comment
• Commit Unit Tests in branch feat-use_metaprogramming_for_tuple_serialization

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai auto-generate unit tests to generate unit tests for this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[glihm]

Awesome work @remybar, will be very nice having this tuple introspect + the fixed size array, which are actually quite common it seems.