Skip to content

should the backend for MemoryStore instances be registered? #3000

Description

@d-v-b

It would be convenient for memory stores to have a JSON representation with a URL, e.g. {"url": "memory://<id>"}. This would make memory stores structurally similar to remote-resource-based storage like the local file system or cloud storage. The connection to these storage backends generally serializes to a JSON that includes a URL field.

For zarr-python to use such a URL for memory storage, we would need to resolve the string "<id>" into a reference to a python object. IMO the easiest way to do this would be for zarr-python to maintain a registry of memorystore backends in a dict. If users want to use a registered memory store, they could initialize MemoryStore with a string, like this:

from zarr.storage import MemoryStore
from zarr.storage.memory import registry
print(registry)
# {}
memstore = MemoryStore("foo", create=True)
print(registry)
# {"foo": {}}
print(memstore.to_json())
# {"url": "memory://foo"}

memstore_2 = MemoryStore("foo", create=True) # this would attemt to create a new dict under the name "foo"
# ValueError("foo is already a registered name for a memory store")

memstore_2 = MemoryStore("foo", create=False) # this will not attempt to create a new dict

An upside of this is that multiple MemoryStore instances backed by the same mutable mapping can be created in multiple places in a program without passing around explicit references to that mutable mapping.

A downside is the the complication about the lifetime of the mutable mappings. Users who needed to completely dispose of a registered dict would need to remove it from the registry. But I think this is ultimately worth it on the basis of extending url semantics to memory stores.

cc @jbms in case you have some tensorstore-derived wisdom to share here

Metadata

Metadata

Assignees

No one assigned

    Labels

    design discussionDiscussion about the design of the library

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions