V2 filtering_settings

[chriddyp]

a placeholder issue to discuss the next version of the filtering_settings property. In particular:

• Its name
• Its structure: is it a string that the user has to parse or a more complex object?
• What does the grammar look like? country eq("Canada") or country is Canada or country == Canada or country = Canada, etc

Related issues
#252
#301

[chriddyp]

How can we make it easier for backend authors to write queries? num() and string casting (") is more complex to write and so in the docs, we don't support them. This makes the front-end filtering different than the back-end filtering, which is confusing.

For example, in the docs, front-end filtering only supports:

• Enter eq "Asia" in the "continent" column
• Enter > num(5000) in the "gdpPercap" column
• Enter < num(80) in the lifeExp column

whereas backend filtering only supports:

• Enter eq Asia in the "continent" column
• Enter > 5000 in the "gdpPercap" column
• Enter < 80 in the lifeExp column

[jonmmease]

Could we provide some way to automatically convert these expressions into expressions in the NumExpr DSL (https://numexpr.readthedocs.io/en/latest/user_guide.html#casting-rules)?

This DSL is directly support by pandas in the pd.query operation (https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.query.html), making backend filtering an efficient one-liner. It also opens up the possibility to filter on expression involving multiple columns.

Update:
For that matter, any reason to not adopt the expression language itself for the front-end as well?

[chriddyp]

That's a really great idea 👍

[chriddyp]

In addition to the NumExpr DSL, I'd like for us to consider how this would scale for additional datatypes beyond str and numbers, including:

• dates
• times
• IP addresses
• categories vs freeform strings
• numbers that could be considered strings (like zip codes)

Also, we should consider what the default "operator" is in the UI. Users shouldn't have to always specify an operator like contains or eq or ==: they should be able to just type some text (e.g. canada or 15) and a reasonable filtering operation should occur.

Here's a proposal that I worked on previously. Perhaps this can be updated to be more compatible with the NumExpr DSL

---

With the introduction of datatypes, the following changes could be made:

• string and enum
  • Default operator is still "contains"
  • >, >=, <, <= will filter alphanumerically
  • Regex will be supported
• number
  • Default operator is =, contains is not available
  • >, >=, <, <= will filter numerically
  • Decimal places will be supported
• date
  • Default operator is "contains" and will filter all dates at a lower resolution. For example:
    • Searching 2015 will filter all values with the year 2015, regardless of the month, day, or time
    • Searching 2015-10-05 will filter all values on the day 2015-10-05, regardless of the time
    • In this mode, it will not be possible to search just "January" across all years
    • >, >=, <, <= will filter by date, starting at the lowest date (e.g. > 2015 is equivalent to > 2015-01-01 00:00:00)
    • = won't be supported. The user can search a specific date with the default and implicit "contains" operator

[eliasdabbas]

I've used RStudio's implementation of DataTable, and I think they have a very nice solution for filtering visually, especially for a non-tech audience

Here's quickly how it works.

I think it's a great idea to provide filters for different data types, especially dates.

If this solution makes sense, then it might probably boil down to utilizing dash-core-components elements within the filtering cells somehow, and that would make a beautiful easy-to-use solution.

By automatically detecting the data type of the column, the appropriate filter would be provided.

Or maybe even give the developer the option to determine which filter they want to add per column.

dcc.RangeSlider: for numerics
dcc.Checklist OR dcc.Dropdown with multi=True: for categorical values when there is a small number of unique options
dcc.DatePickerRange: for dates

[jonmmease]

It doesn't sound like the exact NumExpr/df.query DSLs are going to cover exactly what you're talking about here, especially when considering the idea of a default operator. So it probably does make sense to still define a dash table DSL, but it would be nice to do it in such a way that the combination of all filtering operations can be transformed into a single pandas query string.

Here are a few example expressions that I was playing around with:

For the numerical column a

df.query('a == 1')
df.query('a > 0')
df.query('0 < a and a < 2')

For the string or categorical column c

df.query('c.str.contains("Tw")')
df.query('c.str.startswith("T")')
df.query('c.str.match("\w*o")')
df.query('c.str.count("([a-g])") > 1')

For the datetime column d

df.query('d.dt.year == 2018')
df.query('d >= "2018-04"')
df.query('d.dt.day_name() == "Monday"')
df.query('d.dt.is_month_start')

So my thought is that if a table has filter expressions on columns a, b, and c then the combination of expressions gets transformed into a query string like

df.query('a > 0 and c.str.match("\w*o") and d >= "2018-04"')

[chriddyp]

Another thing to consider is how this relates to conditional formatting. For conditional formatting, we use the same filtering DSL (see https://dash.plot.ly/datatable/style). It's not clear to me if pandas uses the NumExpr/df.query DSL for their conditional formatting, their docs mostly show an imperative workflow (https://pandas.pydata.org/pandas-docs/stable/style.html)

[gaoy400]

@eliasdabbas I am also looking forward to have a filter with RangeSlider and/or dropdown. The datatable video in your comment is of great interest to me. One thing that I am looking for beyond the video is a backend filtering option even when using dropdown/RangeSlider filter.

Really looking forward to have this feature in the dash table in the future!

[eliasdabbas]

@gaoy400 That would also be very nice.
This might be related somehow to #213.

[rbolado]

Hello,

I recently started using the new 'DataTable' function. I am trying to implement a simple conditional formatting applied to a column. My issue is with the 'style_data_conditional' under the 'filter': option. It looks like the num() function rounds to the nearest integer not allowing us to applied <> to decimal numbers in the column. Is this correct? Any help is very much appreciated.

Many thanks