Currently, fx_file has the call signature (filename) -> (filename). Thus, files need to be copied if they need to be modified (overwriting input files is a very bad idea), which is very expensive. Usually, the netCDF4 library is used to perform these kind of modifications.
A possible alternative that has often come up is xarray. At the moment, there is a method DataArray.to_iris() which allows converting a DataArray into a Cube (no idea how efficient this is, though). In addition, the iris devs are working on an improved xarray-iris bridge.
Thus, it would be very nice to allow an efficient usage of xarray in fix_file. My question is now: how would that look like in practice? Possible solutions I could think of are:
- Allow
fx_file to return xarray objects (currently only DataArray?) and load to read these kind of objects.
- Allow
fx_file to return cubes and load to read cubes.
- Rewrite
fx_file in such a way that it is a callback in load (with one of the call signatures above).
The reason I am asking this now is that I want to get started with reading the ERA5 GRIB files that are available on Levante (see #1991). However, iris-grib cannot open them. On the other hand, xarray (in combination with ECMWF's cfgrib) works perfectly fine. Thus, an option to include xarray into our pipeline would be super helpful for me.
@ESMValGroup/technical-lead-development-team opinions?
Currently,
fx_filehas the call signature(filename) -> (filename). Thus, files need to be copied if they need to be modified (overwriting input files is a very bad idea), which is very expensive. Usually, thenetCDF4library is used to perform these kind of modifications.A possible alternative that has often come up is xarray. At the moment, there is a method
DataArray.to_iris()which allows converting aDataArrayinto aCube(no idea how efficient this is, though). In addition, the iris devs are working on an improved xarray-iris bridge.Thus, it would be very nice to allow an efficient usage of xarray in
fix_file. My question is now: how would that look like in practice? Possible solutions I could think of are:fx_fileto return xarray objects (currently onlyDataArray?) andloadto read these kind of objects.fx_fileto return cubes andloadto read cubes.fx_filein such a way that it is a callback inload(with one of the call signatures above).The reason I am asking this now is that I want to get started with reading the ERA5 GRIB files that are available on Levante (see #1991). However, iris-grib cannot open them. On the other hand, xarray (in combination with ECMWF's cfgrib) works perfectly fine. Thus, an option to include xarray into our pipeline would be super helpful for me.
@ESMValGroup/technical-lead-development-team opinions?