-
Notifications
You must be signed in to change notification settings - Fork 21
Docs: add obstore tutorial #527
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
aboydnw
wants to merge
11
commits into
microsoft:develop
Choose a base branch
from
aboydnw:docs-add-obstore-tutorial
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+185
−0
Open
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
5696ab2
docs: add obstore tutorial under overview
aboydnw 863040a
docs: fill in obstore notebook badge links
aboydnw 6d4c423
docs: inline notebook badge URLs and tighten copy
aboydnw bee26a1
docs: drop GitHub badge from obstore tutorial
aboydnw 0e839a7
docs: tighten obstore composability claim in tutorial
aboydnw 086d0ec
Update docs/overview/obstore.md
aboydnw 6f44f0f
Update docs/overview/obstore.md
aboydnw 0fcdbea
Update docs/overview/obstore.md
aboydnw 338133c
Update docs/overview/obstore.md
aboydnw 91254c9
docs: address obstore tutorial review feedback
aboydnw f9eb24b
Merge remote-tracking branch 'origin/docs-add-obstore-tutorial' into …
aboydnw File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,183 @@ | ||
| # Reading Planetary Computer data with obstore | ||
|
|
||
| [obstore](https://developmentseed.org/obstore/) is a Python library for reading and writing cloud object stores through a single, unified API that works the same across Azure Blob, Amazon S3, and Google Cloud Storage. Using obstore, Planetary Computer SAS tokens refresh automatically, async I/O is built in, and the same store you build for reading bytes can be handed to higher-level libraries like [async-geotiff](https://github.com/developmentseed/async-geotiff), [Lonboard](https://developmentseed.org/lonboard/), and [zarr-python](https://zarr.dev/) without re-authenticating. | ||
|
|
||
| A companion notebook walks through every step end-to-end with live timings. [Open in Planetary Computer Hub](https://pccompute.westeurope.cloudapp.azure.com/compute/hub/user-redirect/git-pull?repo=https://github.com/microsoft/PlanetaryComputerExamples&urlpath=lab/tree/PlanetaryComputerExamples/quickstarts/obstore.ipynb&branch=main) | ||
|
|
||
| ## Install obstore | ||
|
|
||
| obstore works in any Python project. To get started, install obstore alongside `pystac-client` (for searching the Planetary Computer's STAC API) and the HTTP libraries that power its [credential providers](https://developmentseed.org/obstore/latest/authentication/#credential-providers): | ||
|
|
||
| ```bash | ||
| uv add obstore pystac-client requests aiohttp aiohttp_retry | ||
| ``` | ||
|
|
||
| `requests` powers the sync credential provider; `aiohttp` and `aiohttp_retry` power the async one. Install both unless you know you only need one path. | ||
|
|
||
| ## Connect to a Planetary Computer asset | ||
|
|
||
| The most common starting point is a STAC asset returned from a search. obstore's `PlanetaryComputerCredentialProvider` reads the asset's blob URL and handles SAS token acquisition and refresh for you. | ||
|
|
||
| 1. Open the Planetary Computer STAC catalog and pick a scene to work with. | ||
|
|
||
| ```python | ||
| import pystac_client | ||
| from obstore.auth.planetary_computer import PlanetaryComputerCredentialProvider | ||
|
|
||
| catalog = pystac_client.Client.open( | ||
| "https://planetarycomputer.microsoft.com/api/stac/v1" | ||
| ) | ||
| item = next(catalog.search(collections=["naip"], max_items=1).items()) | ||
| asset = item.assets["image"] | ||
| ``` | ||
|
|
||
| 2. Build a credential provider from the asset. | ||
|
|
||
| ```python | ||
| provider = PlanetaryComputerCredentialProvider.from_asset(asset) | ||
| ``` | ||
|
|
||
| 3. Build a store using that provider. | ||
|
|
||
| ```python | ||
| from obstore.store import AzureStore | ||
|
|
||
| store = AzureStore(credential_provider=provider) | ||
| ``` | ||
|
|
||
| The credentials obstore fetches grant read access to the *entire container*, not just this one file. `from_asset` simply points the store's prefix at the asset's blob path — which is why the reads below pass an empty path (`""`). To read across the container, see [Open the whole container](#open-the-whole-container) below. | ||
|
|
||
| ## Read bytes from the store | ||
|
|
||
| Once you have a working store, obstore exposes three read methods. Call them directly on the store. | ||
|
|
||
| 1. **Read a byte range.** Useful when you only need part of the file. For example, the first ~16 KB of a Cloud Optimized GeoTIFF. | ||
|
|
||
| ```python | ||
| header = store.get_range("", start=0, end=16384) | ||
| ``` | ||
|
|
||
| 2. **Read multiple byte ranges in a single request.** Cuts round-trip latency when you need several non-contiguous slices of the same file (e.g. multiple COG tiles). obstore coalesces adjacent ranges into a single network request for you. | ||
|
|
||
| ```python | ||
| ranges = store.get_ranges("", starts=[0, 65536], ends=[16384, 81920]) | ||
| ``` | ||
|
|
||
| 3. **Read the entire file.** `store.get` returns a result you can iterate to stream the body in chunks; call `.bytes()` to collect it into one buffer. Avoid collecting large rasters — range reads and async (below) exist for that. | ||
|
|
||
| ```python | ||
| buf = store.get("").bytes() | ||
| ``` | ||
|
|
||
| ## Open the whole container | ||
|
|
||
| `from_asset` is the quickest path for a single scene. When you want to read or list many objects, build the store against the container root instead, then pass full blob paths. | ||
|
|
||
| ```python | ||
| container_store = AzureStore( | ||
| account_name="naipeuwest", | ||
| container_name="naip", | ||
| credential_provider=PlanetaryComputerCredentialProvider( | ||
| "https://naipeuwest.blob.core.windows.net/naip/" | ||
| ), | ||
| ) | ||
|
|
||
| buf = container_store.get("v002/mt/2023/40086/m_4008601_se_12_060_20230621.tif").bytes() | ||
| ``` | ||
|
|
||
| ## Run reads in parallel | ||
|
|
||
| For multi-file workloads — building a mosaic, or fetching every band across every scene in an AOI — running reads concurrently is much faster than one at a time. obstore exposes async equivalents of every read method (`get_async`, `get_range_async`, `get_ranges_async`), which you compose with `asyncio.gather`. | ||
|
|
||
| Async needs its own credential provider, `PlanetaryComputerAsyncCredentialProvider`, backed by `aiohttp` instead of `requests`. It takes the same `from_asset()` constructor. | ||
|
|
||
| ```python | ||
| import asyncio | ||
| from obstore.auth.planetary_computer import PlanetaryComputerAsyncCredentialProvider | ||
|
|
||
| items = list(catalog.search(collections=["naip"], max_items=8).items()) | ||
| stores = [ | ||
| AzureStore( | ||
| credential_provider=PlanetaryComputerAsyncCredentialProvider.from_asset( | ||
| item.assets["image"] | ||
| ) | ||
| ) | ||
| for item in items | ||
| ] | ||
|
|
||
| headers = await asyncio.gather( | ||
| *[store.get_range_async("", start=0, end=16384) for store in stores] | ||
| ) | ||
| ``` | ||
|
|
||
| To read many ranges *within a single file*, don't fan out one request per range. Use `get_ranges_async`, which coalesces adjacent ranges into a single network request under the hood: | ||
|
|
||
| ```python | ||
| tiles = await stores[0].get_ranges_async( | ||
| "", starts=[0, 65536, 131072], ends=[65536, 131072, 196608] | ||
| ) | ||
| ``` | ||
|
|
||
| ## List objects across a container | ||
|
|
||
| To enumerate objects under a prefix ("show me every NAIP scene in Montana in 2023"), call `list` on the `container_store` from above. | ||
|
|
||
| ```python | ||
| for batch in container_store.list(prefix="v002/mt/2023/"): | ||
| for entry in batch: | ||
| print(entry["path"], entry["size"]) | ||
| ``` | ||
|
|
||
| ## Hand the store to other libraries | ||
|
|
||
| Any library that accepts an [obspec](https://github.com/developmentseed/obspec)-compatible store reads through your authenticated connection without re-doing auth. Open the same NAIP scene as a Cloud Optimized GeoTIFF using [async-geotiff](https://github.com/developmentseed/async-geotiff): | ||
|
|
||
| ```python | ||
| from async_geotiff import GeoTIFF | ||
|
|
||
| async_store = AzureStore( | ||
| credential_provider=PlanetaryComputerAsyncCredentialProvider.from_asset(asset) | ||
| ) | ||
| geotiff = await GeoTIFF.open("", store=async_store) | ||
| print(geotiff.transform, geotiff.crs.name) | ||
| ``` | ||
|
|
||
| [zarr-python](https://zarr.dev/) works through a thin adapter (`zarr.storage.ObjectStore` wraps your obstore store). See the [obstore Zarr example](https://developmentseed.org/obstore/latest/examples/zarr/) for a Planetary Computer Daymet walkthrough. | ||
|
|
||
| ## Migrate from `planetary_computer.sign()` + fsspec | ||
|
|
||
| If you're updating an existing project, here's the side-by-side. The old pattern: | ||
|
|
||
| ```python | ||
| import planetary_computer | ||
| import fsspec | ||
|
|
||
| signed = planetary_computer.sign(asset.href) | ||
| with fsspec.open(signed) as f: | ||
| data = f.read() | ||
| ``` | ||
|
|
||
| The obstore equivalent: | ||
|
|
||
| ```python | ||
| from obstore.auth.planetary_computer import PlanetaryComputerCredentialProvider | ||
| from obstore.store import AzureStore | ||
|
|
||
| provider = PlanetaryComputerCredentialProvider.from_asset(asset) | ||
| store = AzureStore(credential_provider=provider) | ||
| data = store.get("").bytes() | ||
| ``` | ||
|
|
||
| obstore handles re-signing on expiry, talks to Azure Blob Storage directly instead of routing through HTTP via fsspec, and exposes async I/O for parallel reads — all without changing your auth code per request. | ||
|
|
||
| ## Use the same code against other clouds | ||
|
|
||
| obstore implements the [obspec](https://github.com/developmentseed/obspec) protocol, so the same read and write calls work against S3 or GCS. Any library built on obspec inherits this portability automatically. | ||
|
|
||
| ```python | ||
| from obstore.store import S3Store | ||
|
|
||
| s3_store = S3Store(bucket="my-bucket", region="us-west-2") | ||
| buf = s3_store.get("path/to/object").bytes() | ||
| ``` | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This makes me realize that
from_assetis a bit annoying if you want to work with a collection instead of an item.I see that the NAIP Collection JSON defines
so we could potentially have a
from_collectionconstructor too.Or maybe
from_assetshould really be renamed tofrom_stac, and support bothItemandCollection? Thoughts?