Multi-index indexing

[benbovy]

Follows #767.

This is incomplete (it still needs some tests and documentation updates), but it is working for both Dataset and DataArray objects. I also don't know if it is fully compatible with lazy indexing (Dask).

Using the example from #767:

In [4]: da.sel(band_wavenumber={'band': 'foo'})
Out[4]:
<xarray.DataArray (wavenumber: 2)>
array([ 0.00017,  0.00014])
Coordinates:
  * wavenumber  (wavenumber) float64 4.05e+03 4.05e+03

As shown in this example, similarily to pandas, it automatically renames the dimension and assigns a new coordinate when the selection doesn't return a pd.MultiIndex (here it returns a pd.FloatIndex).

In some cases this behavior may be unwanted (??), so I added a drop_level keyword argument (if False it keeps the multi-index and doesn't change the dimension/coordinate names):

In [5]: da.sel(band_wavenumber={'band': 'foo'}, drop_level=False)
Out[5]:
<xarray.DataArray (band_wavenumber: 2)>
array([ 0.00017,  0.00014])
Coordinates:
  * band_wavenumber  (band_wavenumber) object ('foo', 4050.2) ('foo', 4050.3)

Note that it also works with DataArray.loc, but (for now) in that case it always returns the multi-index:

In [6]: da.loc[{'band_wavenumber': {'band': 'foo'}}]
Out[6]:
<xarray.DataArray (band_wavenumber: 2)>
array([ 0.00017,  0.00014])
Coordinates:
  * band_wavenumber  (band_wavenumber) object ('foo', 4050.2) ('foo', 4050.3)

This is however inconsistent with Dataset.sel and Dataset.loc that both apply drop_level=True by default, due to their different implementation. Two solutions: (1) make DataArray.loc apply drop_level by default, or (2) use drop_level=False by default everywhere.

[shoyer]

This looks very nice! I would probably opt for making drop_level=True the default for .loc -- it's hard to see when that wouldn't be desirable. In fact, I'm not even sure that it's worth having the option to set drop_level=False.

[benbovy]

I followed your suggestions.

Two more comments (not critical issues I think) :

• Multi-index level drop only works when a dict is provided for a dimension, i.e., da.sel(band_wavenumber='foo') still returns the full pandas.MultiIndex. This would require for indexing.convert_label_indexer to return an updated index also when label is not dict-like, which is not that straightforward I think.
• There is a potential conflict when providing a dictionary to DataArray.loc, it may be interpreted either as a mapping of dimensions / labels or a mapping of index levels / labels for the 1st dimension. For now, the second option is not handled, e.g., da.loc[{'band': 'foo'}] raises a KeyError.

In summary, da.loc[{'band_wavenumber': {'band': 'foo'}}] returns a DataArray with updated index and coordinate / dimension name, da.loc['foo'] returns a DataArray with the full multi-index, and da.loc[{'band': 'foo'}] raises a KeyError. Three different results for the same operation, although I think that the first one is more explicit and better.

[shoyer]

Multi-index level drop only works when a dict is provided for a dimension, i.e., da.sel(band_wavenumber='foo') still returns the full pandas.MultiIndex. This would require for indexing.convert_label_indexer to return an updated index also when label is not dict-like, which is not that straightforward I think.

Indeed. This would require an another data structure somewhere keeping track of level names -- and ideally also ensuring that they are always unique (like dimensions). This seems fine to me for now.

There is a potential conflict when providing a dictionary to DataArray.loc, it may be interpreted either as a mapping of dimensions / labels or a mapping of index levels / labels for the 1st dimension. For now, the second option is not handled, e.g., da.loc[{'band': 'foo'}] raises a KeyError.

I agree -- better to require the user to be explicit. I also don't see many use cases for specifying the coordinate value and level name but not the dimension name. What happens if you type da.loc[{'band': 'foo'}, :] or da.loc[{'band': 'foo'}, ...]? In principle these are not ambiguous, but I could only guess at what happens without looking at the code.

[benbovy]

After refactoring a bit, da.loc[{'band': 'foo'}, :] and da.loc[{'band': 'foo'}, ...] now work as expected.

da.loc[{'band': 'foo'}] still raises a KeyError. Although it is possible to catch this exception, it still remains ambiguous so it is probably better to do nothing.

[shoyer]

Dictionaries not hashable, so we might be able to detect this case by
calling hash() and raising a KeyError with alternative suggestions. Just an
idea -- not sure if there's a clean way to implement that.

On Fri, Mar 25, 2016 at 10:10 AM, Benoit Bovy notifications@github.com
wrote:

After refactoring a bit, da.loc[{'band': 'foo'}, :] and da.loc[{'band':
'foo'}, ...] now work as expected.

da.loc[{'band': 'foo'}] still raises a KeyError. Although it is possible
to catch this exception, it still remains ambiguous so it is probably
better to do nothing.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#802 (comment)

[benbovy]

Unless you see any other issues, I think that this feature doesn't need more development for now.

I'll be back next week to finish this PR (write some tests and doc).

[tippetts]

This will be a great feature. I for one am really looking forward to using it.

Will this work also allow saving to/reading from hdf5 and netcdf files with a MultiIndex? If not, can you give a sketch outline of the approach you (Stephan or Benoit) would take? I assume it would involve saving the information about the MultiIndex structure in some transformed way that fits into an hdf5 file, then reconstructing it on the read. I might need to hack together something for that before MultiIndex serialization makes it into xarray, but I'd like to make sure I don't veer too far off from the real solution that will ultimately come out.

[benbovy]

I hope to have some time next week to work again on this PR.

@tippetts You can see in #719 a few comments about saving/reading xarray data objects with multi-index to/from netCDF. I also looking forward to see this feature implemented - actually I need it for another project that uses xarray - so maybe I'll find some time in the next couple of weeks to start a new PR on this.

[benbovy]

I finally managed to add some tests and docs.

Two more comments:

• Index replacement: if the multi-index has unnamed levels, the chosen name for the replaced index (if any) is <old_dim>_unnamed_level (old_dim is the name of the replaced multi-index dimension).
  Not sure whether or not it is really necessary to track the index level number. Single indexes returned by get_loc_level don't keep that information. Anyway, it is always better to set explicit names for the index levels.
• Indexing on multi-index now also works with nested tuples (multi-index panel to xrarray dataarray index selection #850).

@shoyer I think that it is ready for review. I successfully run the tests on my local branch. Currently, CI tests seem broken for some reason I don't know.

[shoyer]

This is an implementation detail -- probably best to leave it out of the public docs.

[shoyer]

If idx.name != k above, then this could be constructing an invalid dataset.

I think we should create Coordinate(k, idx) and then remap back to the original names below, if necessary.

[shoyer]

Just went through and gave another full review -- this is looking quite nice, just a few more things to clean up!

[benbovy]

Thanks for your second review @shoyer ! It actually helped me to better understand the indexing logic of pandas (never too late)!

I made some updates according to your comments.

I think we're getting closer to a working feature!

[shoyer]

It might make sense simply to drop this paragraph instead -- do we really need to explicitly call out MultiIndex if it's supported?

[benbovy]

No I don't think we need it. However, it might be good to put a sentence somewhere in the docs to recommend users to set names for multi-index levels before creating data arrays or datasets. What do you think?

[shoyer]

I did a little bit of testing. Here is one case I found where things don't work as I expected:

In [32]: idx = pd.MultiIndex.from_product([['a', 'b'], [0, 1, 2]], names=['lev0', 'lev1'])

In [33]: data = xr.DataArray(range(6), [('x', idx)])

In [34]: data.sel(x=('a',))
Out[34]:
<xarray.DataArray (lev1: 3)>
array([0, 1, 2])
Coordinates:
  * lev1     (lev1) int64 0 1 2

In [35]: data.sel(x='a')
Out[35]:
<xarray.DataArray (x: 3)>
array([0, 1, 2])
Coordinates:
  * x        (x) object ('a', 0) ('a', 1) ('a', 2)

I would expect an index drop in the last case, too. I guess we need to check for scalars.

[shoyer]

this is where scalars end up -- probably need to add a clause here to handle MultiIndex

[tippetts]

Mind if I ask if this will get merged into master? It looks like a lot of work went into the pull request, and the discussion + passed checks lead me to believe it could be close to going in. Is there anything a third party can do to push it across the finish line?

[shoyer]

@tippetts Thanks for checking in. I actually hadn't noticed the last round of changes here, but I think this is good to go in now! Big thanks to @benbovy!!!

[benbovy]

Thanks again @shoyer for your reviews and comments!

Merging this PR broke the tests added in #881. I fixed that in #903.