Skip to content

Commit 6b32866

Browse files
authored
Add dynamic_zarr_store module (#57)
* Add dynamic_zarr_store module * Minor fixes for dzs * add a readme * Add the python notebook demo * add the reqs file * Add the tests and fixtures. Some of the parquet fixtures still point to private GCS buckets. * Add license to test file * dynamicgribchunking.ipynb
1 parent 9d76010 commit 6b32866

File tree

99 files changed

+163395
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

99 files changed

+163395
-0
lines changed

grib_index_aggregation/README.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Prototype kerchunk grib aggregation & zarr IO for fast machine learning
2+
3+
This is prototype code shared by Camus Energy to demonstrate possible optimizations for kerchunk and zarr based on
4+
our experience with machine learning in GCP.
5+
6+
We hope to move much of the dynamic zarr store code into Kerchunk in the summer of 2024 with help from the community
7+
and an [IOOS GSOC intern](https://github.com/ioos/gsoc/issues/42).
8+
9+
The zarr parallelization is more difficult to see a path to integration in zarr python 2, but hopefully will be a
10+
benchmark for performance in zarr python 3.
11+
12+
## Setup
13+
14+
```console
15+
mkdir venv
16+
python -m venv venv
17+
source venv/bin/activate
18+
pip install -r requirements.txt
19+
20+
jupyter-lab build
21+
22+
jupyter-lab --port=${CAMUS_JUPYTER_PORT} --log_level=INFO --ip=0.0.0.0 --NotebookApp.notebook_dir=$(pwd -P)
23+
24+
python -m unittest
25+
```
26+
27+
28+
29+

0 commit comments

Comments
 (0)