Skip to content

Replace ObjectStoreRegistry with object_store's new ObjectStoreRegistry #15983

@criccomini

Description

@criccomini

Is your feature request related to a problem or challenge?

I'm working with the object_store folks on upstreaming DataFusion's ObjectStoreRegistry implementation. Part of the goal is that DataFusion would adopt this new registry. I want to check with y'all that the object_store version meets your needs.

The PR is here:

apache/arrow-rs-object-store#348

The implementation is largely the same, but there have been a few tweaks:

  • Replaced dashmap with an RWLock+HashMap to remove dashmap dependency
  • Remove ObjectStoreUrl
  • Add list_urls method
  • Use object_store::parse_url to create an ObjectStore on get_store cache miss (if possible)
  • Adds registry tests
  • Minor doc updates
  • get_store returns an Option now instead of Result

Would you guys be open to adopting this new version?

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions