Time preprocessor uses numpy array instead of dask array

[ledm]

The line is:

ESMValCore/esmvalcore/preprocessor/_time.py

Line 178 in bcb384b

ones = np.ones_like(cube.data)

It is:

    ones = np.ones_like(cube.data)

but it should be:

    ones = da.ones(cube.data.shape)

Minor change, but this change helps me run a heavy recipe on jasmin sci1.

[ledm]

This fix is not working yet, unfortunately. The array needs to be computed - otherwise cube.collapsed fails with a type error. The computation of the dask array causes it be created in memory, which crashes the process sci1 (but not on the high memory machine, sci3).

Any ideas from our dask experts? @bouweandela @jvegasbsc @mattiarighi?

[valeriupredoi]

try using the weighted aggregator that now is lazy (thanks to @zklaus ) and use the weights dask array for weights https://scitools.org.uk/iris/docs/latest/iris/iris/analysis.html#iris.analysis.WeightedAggregator

[zklaus]

This whole routine looks a bit fishy. Imho it should be something more along the lines of

def get_time_weights(cube):
    """
    Compute the weighting of the time axis.
    Parameters
    ----------
    cube: iris.cube.Cube
        input cube.
    Returns
    -------
    numpy.array
        Array of time weights for averaging.
    """
    time = cube.coord('time')
    time_thickness = time.bounds[..., 1] - time.bounds[..., 0]
    time_weights = np.broadcast_to(time_thickness, cube.shape)
    return time_weights

Perhaps it is necessary to provide an explicit mapping using eg iris.util.broadcast_to_shape, but the data should most likely not be copied.

If for some strange reason the original approach must be maintained, perhaps something like

ones = da.ones_like(cube.lazy_data())

works better.

[mattiarighi]

@ledm would you be willing to implement @zlkaus' suggestion?