Dash Multi-Page App URLs do not always execute in the same order

[mjclawar]

Link/Location/pathname + callbacks issues

It appears that the callbacks driven by pathname from a Location component do not always execute in the same order, leading to behavior like on the https://plot.ly/dash/urls page. Refreshing the urls page quickly and repeatedly leads to one of three cases:

1. The home Plotly Dash page is displayed
2. The home Plotly Dash page is flickered and then the URLs page is displayed
3. The URLs page is displayed.

As a result, trying to have multi-page apps driven by the URL does not work, since callbacks either

1. Receive the pathname as None (assuming that would map to case 1 above), or
2. Do not update at all and pathname keeps the old value (assuming that would also map to case 1 above), or
3. It updates after other callbacks have fired, leading to more requests than necessary (assuming that would map to case 2 above), or
4. Execute as expected, but less than all of the time.

Example

The Multi-Page App URLs demo app confirms this behavior, as the initial page load keeps a '404' on the page regardless if I directly go to /apps/app1 in the URL bar in Chrome.

I updated the display_page callback in the example to

@app.callback(Output('page-content', 'children'),
              [Input('url', 'pathname')])
def display_page(pathname):
    print(pathname)
    
    if pathname == '/pages/app1':
        return app1.layout
    elif pathname == '/pages/app2':
        return 'hi!'
    else:
        return '404'

This creates this stack trace:

 * Running on http://127.0.0.1:8889/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 246-948-826
127.0.0.1 - - [13/Sep/2017 15:14:49] "GET /apps/app1 HTTP/1.1" 200 -
127.0.0.1 - - [13/Sep/2017 15:14:50] "GET /_dash-layout HTTP/1.1" 200 -
127.0.0.1 - - [13/Sep/2017 15:14:50] "GET /_dash-dependencies HTTP/1.1" 200 -
127.0.0.1 - - [13/Sep/2017 15:14:50] "GET /favicon.ico HTTP/1.1" 200 -
None
127.0.0.1 - - [13/Sep/2017 15:14:50] "POST /_dash-update-component HTTP/1.1" 200 -
/apps/app1
127.0.0.1 - - [13/Sep/2017 15:14:50] "POST /_dash-update-component HTTP/1.1" 200 -

The page just renders 404. When I include a dcc.Link('CHANGE URL', href='/pages/app1') component on the page, and click on that, then it changes to the app1 content.

Am I doing something wrong? I copied the exact project structure for this demo, am also seeing it in other apps I'm working on, and there seems to be a clear issue with the https://plot.ly/dash/urls page.

[cemal01e]

This PR might be of use to you #70. There may be cases where it makes sense to have each url supported by a different dash object. At the moment I don't think having a one dash instance supporting multiple urls works so well.

[mjclawar]

Thanks! Stumbled across that last week, and it definitely is a solution to the multi-page URL issue.

I believe that there still is an issue with the fact that pathname is None on page load, which causes the issue shown at https://plot.ly/dash/urls in my initial issue. The two options that I've seen around this (unless I'm completely missing something obvious) are:

1. Throw an error when pathname is None, preventing unnecessary computation before the pathname variable updates with the actual pathname from the URL on page load or
2. Create a default pathname, which prevents errors, but leads to terrible performance if there are many components tied to the URL.

I've gone with option 1, but this creates a mess of 500 (INTERNAL SERVER ERROR) messages in the console, and unnecessarily (however slightly) increases the burden on the server, especially if there are many callbacks dependent on pathname.

The core issue seems to be that pathname is not consistently correct on page load.

[chriddyp]

@mjclawar Thanks for opening the issue! Here's what's going on here:
1 - Unspecified properties in the app.layout are set to None
2 - When the app loads, it renders the layout and then it fires off all of the callbacks that have inputs and outputs that are visible in the app. In this case, dcc.Location.pathname is None, so that gets fired off.
3 - When the dcc.Location component is rendered, its pathname property is updated to match the URL https://github.com/plotly/dash-core-components/blob/2c7c3c08319c10639eea3b8c00ae12d399eff4c7/src/components/Location.react.js#L13-L19. In doing this, it fires off a callback with it's new pathname property.

We can't actually set the pathname to a string in the app.layout because the app could be loaded from any url. If we set it to /page-2, then the initial callback would be fired with /page-2 even if the user loaded /page-3.

The solution that was taken in the https://plot.ly/dash/urls page and in the dash-docs (https://github.com/plotly/dash-docs/blob/c2c74e718a492217502b03739e39190c359af029/tutorial/run.py#L286-L287) was to load the index page if the pathname was None and "hope" that the user indeed loaded that URL. As you mention, this causes the page to flicker if the URL was wrong (the index content will be loaded on the first callback when pathname=None and the correct page's content will be loaded when dcc.Location component is rendered).

The correct solution would probably be to load either a loading message or an empty string on None:

@app.callback(Output('content', 'children'), [Input('location', 'pathname')])
def update_children(pathname):
    if pathname is None:
        return 'Loading...'
    else:
        return my_content[pathname]

In fact, I'm going to update the dash-docs to do that right now.

At the moment I don't think having a one dash instance supporting multiple urls works so well.

By updating the content in-place with use of the dcc.Link component, Dash apps are able to provide a multi-page app experience without refreshing the entire page - it's extremely fast! Going forward, using dcc.Location and dcc.Link is the method that I "officially recommend".

[mjclawar]

@chriddyp Appreciate the detailed response.

I understand why pathname is initially None, but it still seems like this should be classified as a bug because the behavior is not consistent. The page does not merely flicker to the index page all the time, in the case of the Dash URLs page. Sometimes it flickers to the index page, then to the URLs page. Sometimes it goes directly to the URLs page. But sometimes (and this is completely undesired behavior), it goes to the index page and stays there. You can still see all of these outcomes it if you refresh the URLs page enough times, although now it has your loading message before all three outcomes 🎉 !

I have used the dcc.Link component and it's phenomenal (in fact this entire project is phenomenal), but only after you get past the initial page load hiccups raised in this issue.

[chriddyp]

But sometimes (and this is completely undesired behavior), it goes to the index page and stays there.

Hm yeah, this is definitely wrong. I've added a commit to remove the "loading index page no matter what before responding to callback" for the docs in https://github.com/plotly/dash-docs/commit/53f7b51f2ee2c0da788d75f72bb4f4f7d83adc67. We'll see if the issue persists after deployment.

Thanks for being so thorough on this one! We'll get it straightened out 👍

[mjclawar]

... and it now appears that your update fixed everything in the above comment. Cool!

So is the best way to approach the pathname being None to just have handlers for initial load that handle a None and return properly-formatted empty data? I'm trying to think about how we scale this without throwing "number of callbacks" errors for callbacks that depend on pathname (what I'm doing right now), since the other alternative is to submit a bunch of unnecessary queries. It just seems like a ton of traffic on page load for a responsive app after pathname updates, but not seeing a way around it given your explanation above.

[chriddyp]

Yikes, that didn't really help. Sometimes it flickers back to nothing. OK, I see what you mean now with things getting called "out of order". None should only be passed first and only once. It looks like it's getting passed in multiple times.

Looks like this is going to require a deeper investigation. I'll look into it and update this thread with any progress. Thanks again for reporting!

[mjclawar]

Crossed comment paths. Just found that bug. Glad we're seeing the same thing! Thanks for checking into it.

[chriddyp]

It just seems like a ton of traffic on page load for a responsive app after pathname updates, but not seeing a way around it given your explanation above.

If this works the way it is supposed to, then there should only be 1 extra / unnecessary API call: the first callback with pathname=None. In the grand scheme of things, this shouldn't be a issue given that the requests and responses JSON payloads are going to be extremely tiny (the request is something like {payload: {inputs: [{id: 'location', 'prop': 'pathname', 'value': null}], outputs: [{id: 'content', 'prop': 'children'}]} and the response is just something like {'id': 'location', 'prop': 'pathname', 'value': ''}) . However, it is certainly confusing and idiosyncratic. I don't see a good way around it given the current architecture.

For now, I'll look into getting this to work the way it is supposed to 😅

[chriddyp]

@mjclawar - I'm looking into fixing this today although I can't seem to reproduce it locally.

Here is the example that I'm using:

import dash
from dash.dependencies import Input, Output
import dash_core_components as dcc
import dash_html_components as html

import time

app = dash.Dash()
server = app.server

app.layout = html.Div([
    dcc.Location(id='location'),
    html.Div(id='content')
])


@app.callback(
    Output('content', 'children'),
    [Input('location', 'pathname')])
def display(path):
    print(path)
    if path is None:
        return ''
    elif path == '/':
        return 'index'
    else:
        return html.H3(path)


if __name__ == '__main__':
    app.run_server(debug=True, threaded=True)

Every time that I refresh the page, the callback gets called with: None and then page-1 (or w/e the pathname is). This is "as expected". I can't replicate the issue where it goes from None to page-1 and then back to None.

I have also tried running the docs locally and making the change from #133 (comment) locally and I still can't replicate it.

Are you able to reliably replicate the bug in your examples?

Thanks for the help!

[chriddyp]

Nevermind, I'm able to reproduce the issue now:

import dash
from dash.dependencies import Input, Output
import dash_core_components as dcc
import dash_html_components as html

import time

app = dash.Dash()
server = app.server
server.secret_key = 'test'


app.layout = html.Div([
    dcc.Location(id='location'),
    html.Div(id='content')
])


@app.callback(
    Output('content', 'children'),
    [Input('location', 'pathname')])
def display(path):
    print(path)
    if path is None:
        time.sleep(3)
        return ''
    elif path == '/':
        return 'index'
    else:
        return html.H3(path)


if __name__ == '__main__':
    app.run_server(debug=True, threaded=True)

The issue occurs when the intial request with pathname=None takes longer than the follow up request with pathname='/page-1'. While dash initiates the requests in order, it looks like we aren't doing a good job of dealing with request responses that come back out of order

[mjclawar]

So this is a deeper problem than just a bug. Although arguably one that Dash is not really responsible for handling.
It would be nice if there were a fix for the dcc.Location component where it could be initialized on the client side as something other than None based on, e.g., the url from the page if there were not a default pathname passed in on the server side. That (selfishly) effectively patches all the inconsistency problems I am having with it! 😄