Skip to content
Open
Changes from 1 commit
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
c9812c7
add big file test
valeriupredoi Feb 19, 2024
4afb750
run big file test
valeriupredoi Feb 19, 2024
2b0c7c4
use timeit see memory
valeriupredoi Feb 19, 2024
935bcaf
one single process Vasili
valeriupredoi Feb 19, 2024
40f6e6e
back to all Pythons
valeriupredoi Feb 19, 2024
f8f930f
toss some timings
valeriupredoi Feb 20, 2024
b1a1aeb
save file and toss some timings
valeriupredoi Feb 20, 2024
6030040
fail a test and add v0 test
valeriupredoi Feb 20, 2024
6423a14
GA workflow new test names
valeriupredoi Feb 20, 2024
c2d426c
turn off storage caching
valeriupredoi Feb 22, 2024
67ce586
kluge to get a Group
valeriupredoi Feb 23, 2024
748c266
kluge to get a Group
valeriupredoi Feb 23, 2024
16e5656
fix module
valeriupredoi Feb 26, 2024
fb5c5d2
add code change suggested for Kerchunk
valeriupredoi Feb 26, 2024
256cb65
allow for cases
valeriupredoi Feb 26, 2024
c3775cb
allow for cases
valeriupredoi Feb 26, 2024
e3184a7
correct file url for kerchunk and pass storage options
valeriupredoi Feb 27, 2024
6dd3a32
add function that readds the correct compressor to the reductionist
valeriupredoi Feb 28, 2024
0677ca4
just print chunk selections to look at them
valeriupredoi Feb 29, 2024
f3dc818
finally made it work all through
valeriupredoi Feb 29, 2024
d20cdc2
streamline
valeriupredoi Feb 29, 2024
0425e3c
cleanup prints
valeriupredoi Feb 29, 2024
f52693c
handle exception
valeriupredoi Feb 29, 2024
e9a8409
revert to JSON tempfile before I die of not realizing WTF was going o…
valeriupredoi Feb 29, 2024
ecc203d
added comments
valeriupredoi Mar 1, 2024
8f92ae7
unfail test
valeriupredoi Mar 1, 2024
dbae564
add info about PytestUnraisableError
valeriupredoi Mar 1, 2024
ddd38fa
Merge branch 'main' into test_optimal_kerchunk
valeriupredoi Mar 1, 2024
40f110b
cleanup GA flow
valeriupredoi Mar 1, 2024
91d3f57
pin kerchunk
valeriupredoi Mar 6, 2024
8cbf1f0
pin kerchunk
valeriupredoi Mar 6, 2024
fcefd23
use correct attributes compression and shuffle
valeriupredoi Mar 7, 2024
a4aadce
Revert "use correct attributes compression and shuffle"
valeriupredoi Mar 7, 2024
8f9e9ab
fix case filters is None
valeriupredoi Mar 7, 2024
ea20e16
use two procs for pytest
valeriupredoi Mar 7, 2024
f6b1189
Merge branch 'main' into test_optimal_kerchunk
valeriupredoi Mar 7, 2024
151284d
replace s3 test file because I am a pillock and I overwrote the other
valeriupredoi Mar 7, 2024
8c82fb8
fix SegFault
valeriupredoi Mar 7, 2024
aa30b0b
add extension to truncate to the other kerchunking block and lots of …
valeriupredoi Mar 8, 2024
0a6bd1d
add valid storage options for file opening in Kerchunk
valeriupredoi Mar 8, 2024
2f75873
reinstance variable in test
valeriupredoi Mar 8, 2024
6a4206d
del pointer
valeriupredoi Mar 8, 2024
23cc9d5
separate calls to Active
valeriupredoi Mar 8, 2024
86a22d8
reduce test
valeriupredoi Mar 8, 2024
3d50db8
reduce test
valeriupredoi Mar 8, 2024
2582d96
don't run exploratory tests
valeriupredoi Mar 8, 2024
662a69f
cleanup test
valeriupredoi Mar 8, 2024
3ee8732
reinstate test; we need to figure out the segfaulting not brushit und…
valeriupredoi Mar 9, 2024
48df2cd
cleanup and restore test
valeriupredoi Mar 9, 2024
30d435f
reinstate full workflow
valeriupredoi Mar 9, 2024
110b96a
try create new variable
valeriupredoi Mar 9, 2024
8b9c890
same var name again
valeriupredoi Mar 9, 2024
0079a8b
try returning after each conditional
valeriupredoi Mar 9, 2024
96c9ba4
comment out Dataset/group selection when preset S3 options are used
valeriupredoi Mar 9, 2024
03c11b3
add note
valeriupredoi Mar 10, 2024
d064319
comment out heavy printing to stdout
valeriupredoi Mar 18, 2024
730f6ec
turn off some printing to stdout
valeriupredoi Mar 18, 2024
d82b916
Merge branch 'main' into test_optimal_kerchunk
valeriupredoi Jun 24, 2024
d88cd99
Merge branch 'main' into test_optimal_kerchunk
valeriupredoi Jul 15, 2024
3292612
Merge branch 'main' into test_optimal_kerchunk
valeriupredoi Jul 22, 2024
a46c59a
Merge branch 'main' into test_optimal_kerchunk
valeriupredoi Aug 5, 2024
66a566e
Merge branch 'main' into test_optimal_kerchunk
valeriupredoi Aug 12, 2024
c21d02d
Merge branch 'main' into test_optimal_kerchunk
valeriupredoi Oct 24, 2024
319220a
Merge branch 'main' into test_optimal_kerchunk
valeriupredoi Jan 20, 2025
a1f54c4
Merge branch 'main' into test_optimal_kerchunk
valeriupredoi Feb 11, 2025
9429fae
Merge branch 'main' into test_optimal_kerchunk
valeriupredoi Mar 6, 2025
726ebe0
Merge branch 'main' into test_optimal_kerchunk
valeriupredoi Mar 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
same var name again
  • Loading branch information
valeriupredoi committed Mar 9, 2024
commit 8b9c890dd5a8846c7d91fe16436047441bb53444
6 changes: 3 additions & 3 deletions activestorage/netcdf_to_zarr.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,15 +74,15 @@ def gen_json(file_url, varname, outf, storage_type, storage_options):
print("Looking only at a single Dataset", s3file_r_1[varname])
s3file_w_1.create_group(varname + " ")
s3file_w_1[varname + " "][varname] = s3file_w_1[varname]
s3file_1 = s3file_w_1[varname + " "]
s3file = s3file_w_1[varname + " "]
elif isinstance(s3file_r_1[varname], h5py.Group):
print("Looking only at a single Group", s3file_r_1[varname])
s3file_1 = s3file_r_1[varname]
s3file = s3file_r_1[varname]

storage_options = {"key": S3_ACCESS_KEY,
"secret": S3_SECRET_KEY,
"client_kwargs": {'endpoint_url': S3_URL}}
h5chunks = SingleHdf5ToZarr(s3file_1, file_url,
h5chunks = SingleHdf5ToZarr(s3file, file_url,
inline_threshold=0,
storage_options=storage_options)

Expand Down