Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
9f7783d
ignore s3 exploratory tests in normal runs
valeriupredoi Jun 9, 2023
8069cef
add s3 test run GA workflow
valeriupredoi Jun 9, 2023
d854997
change GA test feature branch
valeriupredoi Jun 9, 2023
ea2cf98
add s3 exploratory test repo
valeriupredoi Jun 9, 2023
e84f21b
correct path to minio scripts
valeriupredoi Jun 9, 2023
fc5766a
run s3 AS container
valeriupredoi Jun 9, 2023
37ca643
run Active component too
valeriupredoi Jun 9, 2023
cccc8d2
temp deactivate check on physical file in active class
valeriupredoi Jun 9, 2023
d0b9805
shove in use s3
valeriupredoi Jun 9, 2023
1c10fb8
run with s3 file
valeriupredoi Jun 9, 2023
405b5eb
try load netCDF from s3
valeriupredoi Jun 9, 2023
60e1cb3
pass correct s3 uri
valeriupredoi Jun 9, 2023
bf280ab
of course I forgot import
valeriupredoi Jun 9, 2023
68a7777
try without s3 prefit for 3 file path
valeriupredoi Jun 9, 2023
0fa23f2
possible correct s3 path
valeriupredoi Jun 9, 2023
66dce92
possible correct s3 path
valeriupredoi Jun 9, 2023
de1939d
try with context manager
valeriupredoi Jun 9, 2023
6c78fce
use s3fs directly
valeriupredoi Jun 9, 2023
fee58b3
add client server arg
valeriupredoi Jun 9, 2023
a1d3ac0
try without localhost
valeriupredoi Jun 9, 2023
f076c6c
add xarray for test
valeriupredoi Jun 9, 2023
1efe1e7
add xarray for test
valeriupredoi Jun 9, 2023
6981740
use xarray for test
valeriupredoi Jun 9, 2023
482ffb5
wrong engine
valeriupredoi Jun 9, 2023
f17d08f
try mfdataset
valeriupredoi Jun 9, 2023
ab0674b
forget about xarray
valeriupredoi Jun 9, 2023
77bfef9
forget about xarray
valeriupredoi Jun 9, 2023
5b4f64e
hat does that s3 file like look like
valeriupredoi Jun 9, 2023
6003b74
how does that s3 file look like
valeriupredoi Jun 9, 2023
71051e4
asdd h5netcdf to env
valeriupredoi Jun 12, 2023
624c849
asdd h5netcdf to env
valeriupredoi Jun 12, 2023
30c33d3
try h5netcdf layer
valeriupredoi Jun 12, 2023
514b1b6
dont call netcdf4
valeriupredoi Jun 12, 2023
9dcb03d
skeip filers xtraction
valeriupredoi Jun 12, 2023
1e0ee8d
fix test
valeriupredoi Jun 12, 2023
83e3e85
tweak call
valeriupredoi Jun 12, 2023
04ee231
switch to s3fs
valeriupredoi Jun 12, 2023
b8f2761
give it a slash free name
valeriupredoi Jun 12, 2023
4af7c4c
bleghh
valeriupredoi Jun 12, 2023
c7b4fef
pytested test
valeriupredoi Jun 12, 2023
f2a1478
refine a bit
valeriupredoi Jun 12, 2023
1a43bf1
corrected check for file
valeriupredoi Jun 12, 2023
20a806e
posix to str paths
valeriupredoi Jun 12, 2023
4b5c2d8
made an s3 loader function
valeriupredoi Jun 12, 2023
770a9d0
blegh colon dude
valeriupredoi Jun 12, 2023
621f85d
return something too
valeriupredoi Jun 12, 2023
945b32f
load s3 connection params from config
valeriupredoi Jun 12, 2023
0d3492f
add test data file
valeriupredoi Jun 14, 2023
ea91080
add netCDF real file test
valeriupredoi Jun 14, 2023
d1fa7fb
missing import
valeriupredoi Jun 14, 2023
5272d7d
pass correct var
valeriupredoi Jun 14, 2023
4c5b311
fail to visualize results
valeriupredoi Jun 14, 2023
5ae2666
fixed test
valeriupredoi Jun 14, 2023
b0dc572
correct version of test
valeriupredoi Jun 14, 2023
4ef10cb
add to local s3 unit test
valeriupredoi Jun 14, 2023
2fb70fd
add mention in test
valeriupredoi Jun 14, 2023
273f034
correct file passed eh
valeriupredoi Jun 14, 2023
8bf1454
simple raiser test
valeriupredoi Jun 14, 2023
04f29a4
remove unnecessary test
valeriupredoi Jun 15, 2023
5c7f86b
start a unit test for storage types
valeriupredoi Jun 15, 2023
f401deb
Merge branch 'main' into real_world_s3_tests
valeriupredoi Jun 15, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
corrected check for file
  • Loading branch information
valeriupredoi committed Jun 12, 2023
commit 1a43bf14285e61157e9addd77488b4fb1f5aa209
2 changes: 1 addition & 1 deletion activestorage/netcdf_to_zarr.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ def gen_json(file_url, fs, fs2, varname, **so):
"""Generate a json file that contains the kerchunk-ed data for Zarr."""
# set some name for the output json file
fname = os.path.splitext(file_url)[0]
if os.sep in fname:
if "s3:" in fname:
fname = os.path.basename(fname)
outf = f'{fname}_{varname}.json' # vanilla file name

Expand Down