Loosen the version constraint of dash-core-components to avoid dependency conflicts

[NeolithEra]

Hi, dash locked the version constraint of dash-core-components as dash-core-components ==1.6.0, which leads a troubling scenario to its direct downstream project beeflow which has dependency dash-core-components.

What makes the situation worse is that the downstream projects [skijumpdesign, django-plotly-dash, plotly-scientific-plots] of dash are also have dependency dash-core-components.

Could you please loosen the version constraint of dash-core-components?
Benefit of this is that users using both of dash and dash-core-components can upgrade their third party libraries in a timely manner to reduce technical debts.

Solution

The dependency trees of your project and affected downstream projects are shown as follows.
Taking the version constraints of upstream and downstream projects into comprehensive consideration, you can

1. Loosen click to be dash-core-components >=1.6.0.

2. Try to add an upper bound for dash-core-components’s version constraint, according to your compatibility.

@chriddyp Please let me know your choice. I can submit a PR to fix this issue.

Thanks for your attention.
Best,
Neolith

[NeolithEra]

Dependency tree-----

dash – 1.7.0
| +- dash-core-components(install version: 1.6.0 version range:==1.6.0)
| +- dash-html-components(install version: 1.0.2 version range:==1.0.2)
| +- dash-renderer(install version: 1.2.2 version range:==1.2.2)
| +- dash-table(install version:4.5.1 version range:==4.5.1)
| +- flask(install version:1.1.1 version range:>=1.0.2)
| | +- click(install version:7.0 version range:>=5.1)
| | +- itsdangerous(install version:1.1.0 version range:>=0.24)
| | +- jinja2(install version:2.10.3 version range:>=2.10.1)
| | | +- markupsafe(install version:1.1.1 version range:>=0.23)
| | +- werkzeug(install version:0.16.0 version range:>=0.15)
| +- flask-compress(install version:1.4.0 version range:*)
| +- plotly(install version:4.3.0 version range:*)
| | +- retrying(install version:1.3.3 version range:>=1.3.3)
| | | +- six(install version:1.13.0 version range:>=1.7.0)
| | +- six(install version:1.13.0 version range:*)

beeflow - 0.0.2
| +- dash(install version:1.7.0version range:*)
| +- dash-core-components(install version:1.6.0 version range:*)
| +- dash-html-components(install version:1.0.2 version range:*)
| +- dash-resumable-upload(install version:0.0.3 version range:*)
| +- dash-table(install version:4.5.1 version range:*)
| +- pandas(install version:0.25.3 version range:*)
| +- plotly(install version:4.3.0 version range:*)
| | +- retrying(install version:1.3.3 version range:>=1.3.3)
| | | +- six(install version:1.13.0 version range:>=1.7.0)
| | +- six(install version:1.13.0 version range:*) 

skijumpdesign - 1.3.0
| +- cython(install version:0.29.14 version range:*)
| +- dash(install version:1.7.0 version range:*)
| +- dash-core-components(install version:1.6.0 version range:*)
| +- dash-html-components(install version:1.0.2 version range:*)
| +- dash-renderer(install version:1.2.2 version range:*)
| +- fastcache(install version:1.1.0 version range:*)
| +- flask(install version:1.1.1 version range:*)
| | +- click(install version:7.0 version range:>=5.1)
| | +- itsdangerous(install version:1.1.0 version range:>=0.24)
| | +- jinja2(install version:2.10.3 version range:>=2.10.1)
| | | +- markupsafe(install version:1.1.1 version range:>=0.23)
| | +- werkzeug(install version:0.16.0 version range:>=0.15)
| +- matplotlib(install version:3.1.2 version range:*)
| +- numpy(install version:1.17.4 version range:>=0.13.0)
| +- plotly(install version:4.3.0 version range:*)
| | +- retrying(install version:1.3.3 version range:>=1.3.3)
| | | +- six(install version:1.13.0 version range:>=1.7.0)
| | +- six(install version:1.13.0 version range:*)
| +- scipy(install version:1.3.3 version range:>=1.0)
| +- setuptools(install version:42.0.1 version range:*)
| +- sympy(install version:1.4 version range:*) 

django-plotly-dash - 1.1.0
| +- dash(install version:1.7.0 version range:*)
| +- dash-core-components(install version:1.6.0 version range:*)
| +- dash-html-components(install version:1.0.2 version range:*)
| +- dash-renderer(install version:1.2.2 version range:*)
| +- django(install version:3.0rc1 version range:>=2)
| | +- asgiref(install version:3.2.3 version range:*)
| | +- pytz(install version:2019.3 version range:*)
| | +- sqlparse(install version:0.3.0 version range:>=0.2.2)
| +- dpd-components(install version:0.1.0 version range:*)
| +- flask(install version:1.1.1 version range:>=1.0.2)
| | +- click(install version:7.0 version range:>=5.1)
| | +- itsdangerous(install version:1.1.0 version range:>=0.24)
| | +- jinja2(install version:2.10.3 version range:>=2.10.1)
| | | +- markupsafe(install version:1.1.1 version range:>=0.23)
| | +- werkzeug(install version:0.16.0 version range:>=0.15)
| +- plotly(install version:4.3.0 version range:*)
| | +- retrying(install version:1.3.3 version range:>=1.3.3)
| | | +- six(install version:1.13.0 version range:>=1.7.0)
| | +- six(install version:1.13.0 version range:*)

plotly-scientific-plots - 0.1.0.6
| +- colorlover(install version:0.3.0 version range:*)
| +- dash(install version:1.7.0 version range:*)
| +- dash-core-components(install version:1.6.0 version range:*)
| +- dash-html-components(install version:1.0.2 version range:*)
| +- numpy(install version:1.17.4 version range:*)
| +- plotly(install version:4.3.0 version range:*)
| | +- retrying(install version:1.3.3 version range:>=1.3.3)
| | | +- six(install version:1.13.0 version range:>=1.7.0)
| | +- six(install version:1.13.0 version range:*)
| +- scipy(install version:1.3.3 version range:*)
| +- sklearn(install version:0.0 version range:*)
| | +- scikit-learn(install version:0.22rc2.post1 version range:*)

[alexcjohnson]

Thanks for bringing this up @NeolithEra

Back in v0.37 we version-locked all of the Dash core packages - everything that installs with dash. This allows us to update these packages knowing they will have access to new features from dash and dash-renderer (and we don't need to spend time testing and fixing the full matrix of possible version combinations), and it ensures users only need to update dash itself to get these latest versions.

What that means is that no other packages should require these locked dependencies directly.
If a downstream package requires a certain version of dash-core-components, it should require dash at the corresponding version, and dash-core-components will come along with it.

I apologize for not making this clearer to the community when we introduced version locking - but the solution is to update those downstream dependencies to not include dash-core-components, dash-html-components, dash-table, or dash-renderer at all in their own requirements.

#1063, #1064, and #1065 are all the same issue - I'm going to close these as duplicates and we can finish the discussion here. From your comments throughout these issues it seems like the projects impacted are at least:

• beeflow
• skijumpdesign
• django-plotly-dash
• plotly-scientific-plots
• janggu
• wiley-dash-boilerplate2
• metacell
• clsify

[moorepants]

It isn't clear why these dash dependencies need to be locked at specific versions. Are you changing public API at every minor version change? It would really work better on the downstream packaging if dash's dependencies were >= instead of ==. Then the dependency min versions only need to be bumped if their public API is changed or a new feature is needed. I've been managing the conda-forge packaging of dash and it is so much more of a pain to handle that all other python packages that I maintain there.

[alexcjohnson]

We're not changing the user-facing API, no. But these packages do sometimes interact with each other in ways that the user generally doesn't need to know anything about. The most recent example of this is async loading - dash-core-components and dash-table created async chunks immediately when this feature was introduced in dash. If we were to require dash-core-components >= in dash, then it would be possible to get an older version of dash that doesn't support async chunks with a newer version of dash-core-components that needs them. It's possible that would actually work... but if we don't have to think about this kind of issue, we can put that effort into new features instead.

Plotly-created dash packages outside this core group, such as dash-bio, have the reverse dependency, requiring dash >= - ie when dash-bio added async, it bumped its dash dependency to the first version to support that feature. @moorepants that's the kind of relationship you're talking about. But in the core packages we specified the dependencies the other way around, really to help users install and upgrade more efficiently and robustly - just install/upgrade dash and the rest of the core packages come along for the ride. In that sense these requirements are more about feature delivery than compatibility per se.

In the PyPI ecosystem, once we get the message out about how to require these packages, this seems to me pretty easy to manage. I confess I don't know conda well enough to understand why this makes things difficult there.

[moorepants]

I'm not sure I follow your reasoning above on a first pass. It almost sounds like you have (or want to have) circular dependencies among these "core" packages. (in fact doesn't dash-table depend on dash and dash depend on dash-table?)

If dash is hard pinning these other packages, why just merge them all into this repository? What the point is keeping them in all the separate packages? If these packages can't really do anything useful independently it isn't clear why they are separate.

conda has a full blown dependency solver (unlike pip) and it tries to find compatible sets of dependencies for a given set of packages. This means that the looser packages make their dependencies the better for all.

[alexcjohnson]

It almost sounds like you have (or want to have) circular dependencies among these "core" packages.

I guess there are two issues getting mixed up here - we're using the installation of dash as a way to deliver dash-core-components, dash-html-components, and dash-table, but dash doesn't actually depend on any of these three, instead they all depend on dash.

why not just merge them all into this repository?

Heh yeah, that's been suggested before. And we did merge dash-renderer into this repository though we didn't merge the two PyPI packages. (though perhaps we should ditch the dash-renderer PyPI package at this point. It was kept separate to allow it to be used with multiple back ends, but in practice our one other back end - Dash for R - doesn't need that)

We kept them separate really just for symmetry with other non-core component packages - it seemed confusing to have to import some components as sub-packages of dash (import dash.core_components as dcc) while others are top-level packages (import dash_bio as dashbio). Also so you need to explicitly import a component package to use it, as importing is what causes its JS to be included in the page.

Anyway in principle we could merge the repos but keep separate PyPI packages... or even one package that installs multiple modules (as we do for example with the plotly package, which also installs plotlywidget, _plotly_utils, and _plotly_future_). We talked about those options in the past but hadn't seen the benefit to justify the effort.

[moorepants]

I guess there are two issues getting mixed up here - we're using the installation of dash as a way to deliver dash-core-components, dash-html-components, and dash-table, but dash doesn't actually depend on any of these three, instead they all depend on dash.

Ah, that's the issue! Very confusing. This does not follow typical packaging practices in Python. The dependency specification should actually specify the dependencies in the correct direction.

Anyway in principle we could merge the repos but keep separate PyPI packages.

It is the PyPi packages that are the issue, not this repo, per se. The setup.py install_requires should be defined for each package with the correct dependencies. Then the conda packages can follow suit. I would suggest removing any dependencies from dash and then setting dash as a >= dependency to each of the other packages. You can always create a "meta-package" that installs them all, i.e. empty package that only contains the dependency specs for a subset of dash packages that you want to group.

[moorepants]

One other note you may not be aware of. A python distribution can contain more than one package. So you can have all the dash- things in one github repo, each dash- thing is in its on top level directory (a package), you can have a single setup.py for that group of packages and version them together. This allows you to keep the packages separate from a python importing perspective but version and distribute all of them together.

[Marc-Andre-Rivet]

As previously mentioned, our usage probably isn't standard and is caused by our trying to juggle many constraints at once. Below is a state of things as discussed above, with some additions, followed by a suggested approach.

Currently:

1. The dash component plugins are not useful on their own
2. dash is used both as the Python server package and as the delivery mechanism for the renderer and selected component plugins
3. Both dash and the component plugins are mostly JS code delivery mechanisms
4. The current versioning scheme takes into account the public facing Python API
5. The current versioning scheme does not take into account the internal (JavaScript) implementation details (renderer, 3rd party libraries, plugins, components interaction with the JS environment)
6. Non-core component plugins (not packaged as part of dash) define a minimal compatible version of dash (e.g. dash>=1.7.0) and thus are in contradiction with the core plugins
7. Non-core component plugins have an independent lifecycle vs. core component plugins

Constraints:

1. We want to keep version locking core component plugins to a specific version of dash whether this is done explicitly or implicitly -- the cost and complexity of maintaining a matrix of tests is high versus the potential gains
2. We do not want to expose internal implementation details (JS) in the packages semver

At the moment, I would like to suggest a twofold approach:

1. Minimally, while 2 is being discussed, we should clarify usage documentation and expectations when using Dash as a third-party library in another project. Current beeflow (skijumpdesign is similar) usage is:

install_requires=['pandas', 'plotly',
                      'dash', 'dash_table',
                      'dash_core_components',
                      'dash_html_components',
                      'dash_resumable_upload'],

Starting with dash>=0.37 the correct usage is to define the desired version of dash only.

2. We should simplify package publication / usage

One assumption that we've had so far is that we do not want to treat core component plugins differently from other component plugins -- for that reason we objected to sub-packaging the core component plugins so as to make it obvious that one could create his own components). This dates back to the inception of Dash ~2017. As there are numerous examples of 3rd party components now, this assumption can probably be revisited & removed.

Releasing dash + html + dcc + table as a single package with sub-packages would greatly clarify the relationship between all of the above and make the version locking requirements moot. We should continue supporting the current import syntax (e.g. import dash_table) and possibly supplement it with a new syntax (e.g. from dash import table)

Technically we can generate this single package whether the core and various plugins are in a single repo or not. The JS publication remains unaffected.