Routing API/DSL

[briancavalier]

We've discussed several times how it would be nice to setup url routing in wire specs. There's also more discussion and backstory in this gg post and in this gist. I just want to open this up for discussion here since gist notifications seem to have gone by the wayside, and gg isn't the best place for pasting code.

[skiadas]

Ok here's some thoughts:

Routing/History protocol

Basic routing format

Basic routing mechanism: A "routes" facet you can add to any object. E.g:

routes = {
    'aroute': 'acomponentmethod',
    ....
}

Accepted routes

Routes follow the following patterns (this deviates a bit from the regexp approach):

1. Literal Match: 'object/edit'
2. Colon-prefixed parameters match between hashes: 'object/:id'. Multiple such parameters allowed
3. Star-prefixed parameters match rest of expression, or empty: 'object/*rest'
4. Star itself matches rest of expression, but does not store in special property. E.g. '*' will match all routes.

Question 1: Are there other conditions that would make sense? For example specifying that a match needs to be an integer. Perhaps allow semi-arbitrary regexs with a format like: ':id:regex'. Or perhaps better to keep it simple.

Question 1': Perhaps add a way for optional parameters, and a way to match query strings? Or perhaps matching query strings should be automatic?

Question 2: Should these obey a "first to match activates" rule, or should we let all that match run? If the former, what guarantees do we have on different javascript environments that the order in which the routes are specified is the same as the order in which they will be accessed through a "for in" loop? If the latter, do we mind the fact that there is no easy way to specify a "default" action, only to be ran if others don't match? And what sense would "default action" make, if developers are allowed to attach routes facets on different components, and have different routes rules on each? Would we be talking about "one rule per component"? If not, how do we specify priority amongst components? If we go with "one rule per component", perhaps we can allow a '$default' route option, to run only if no other routes in that component matched.

Question 3: How much do we care about trying to offer a behavior that exactly matches what popular packages that offer routing currently do?

Question 4: Should a route be activated the first time a page loads, if it matches the current URI? Or only on subsequent changes to the URI? Should this be a configurable option?

Question 5: What support should we offer for programmatically adjusting the routes?

Method signatures

Question 6: How should the methods handling routes receive the matched route information? Some possibilities:

1. First argument is an array of the matches, with the first element being the entire route matched, and subsequent elements matching any other parameters in order. Parameter names are lost. Not my favorite approach.
2. First argument is an object literal of the matched parameters only. Entire route is lost.
3. First argument is the entire route, second argument is an object literal of the matched parameters.
4. First argument is an object literal of the matched parameters. A special key, say $route in it designates the full route.
5. One argument for each parameter in order. Possibly starting with the full route match. This is what Backbone does, without the full route first.

Option 4 is probably what I would lean towards.

Question 7: Regardless, it is worth thinking as to whether extra optional arguments should be allowed, one matching the "previous/current route", a "from" field so to speak, and possibly another matching the state one could have stored via pushState, and gotten back via popState. But more on that later.

History.pushState

Opt-in or Opt-out

Question 8: Should using the HTML5 History API be opt-in or opt-out? This should likely be a configuration option on the plugin, something akin to:

$plugins: {
    { module: 'wire/route', legacy: false }
}

where legacy: false means that we do NOT try to use the new stuff if it is available. Or perhaps forceHash: true instead?

To me it makes sense to try to use the new stuff if it is available. An issue with that might be someone using a "new url", without hashes, that they got from a new browser, on an older browser. The new stuff sort of also implies your server is in a position to support direct requests to the corresponding URIs. Perhaps then it should be an opt-in thing, i.e. forceHash defaults to true (or legacy defaults to false). Kinda torn on this. I guess it depends on who we expect would be using this.

One possibility: Keep the history stuff as a separate plugin. Let 'wire/route' do its thing as usual, using hashes. But if 'wire/history' is also loaded, it tries to use the HTML5 API. I rather think a single module makes more sense.

Here's how I envision the history plugin might act:

1. Whenever it has to deal with a uri hash, it checks to see if there is an anchor named with that hash, in which case it just lets it through and doesn't trigger any routing.
2. All other hashes it turns into absolute URIs. This includes hashes that might have arisen on an old browser and then copy pasted, that were starting somewhere deeper in the application then also had a hash to them. For example www.foo.com/bar#baz would turn into www.foo.com/baz (if there is no anchor named 'baz' that is).
3. It registers itself to listen to all hashchange events as well as popState events. It turns the hashchange event into a pushState call, and calls appropriate routes. It similarly calls those routes on popState.
4. It provides a route! reference resolver meant to be used on its own, returning an object with a set method which either tries to change the hash or uses History.pushState depending on the setup, and still triggers the necessary routes. Use for programmatically setting the route. It also offers a replace method for when you want to change the hash/location/trigger routes without creating a new history item.
5. Alternately, it could instead provide setRoute! and replaceRoute! resolvers that directly return functions.

[scothis]

I like to try to follow the URI Template spec (rfc6570) as much as possible for defining routes. It's not perfect for parsing URIs, but a subset of the spec could be useful. I'm looking towards supporting this format in rest.js.

http://tools.ietf.org/html/rfc6570

[scothis]

For question 2

Deterministic matching of routes is important. First to match is dangerous as I'm not sure how to define an order. I'd expect modules of an application to be able to contribute routes.

Defining specificity rules would be an interesting approach. Much like CSS specificity rules, we can have a clear, deterministic mechanism for matching URLs to handlers.

Something to keep in mind is how this behavior would work on a server as opposed to a client. There are many factors that can affect mapping on a server beyond the URL, such as: headers, query params and method. It's probably best to not get bogged down in how these come together, but it's important to keep in mind that it's a direction we will likely want to move in the future.

[skiadas]

@scothis Good idea regarding the template spec. Seems to be solving the reverse problem in a way however. Am I correct, that by default a variable count may contain slashes and describe a longer path? I feel for routing it would be important to match slash components on their own.

Let's take a simple example. Say we want to match all things of the form: /docs/foo/bar, where 'foo' and 'bar' could be anything (but not containing slashes). I suspect in terms of the RFC this would be done with something like: /docs{/foo,bar}. How would the parser know not to have "foo" match the whole thing and set "bar" to be undefined? And would it also be matching /docs/foo? How can we specify that we don't want it to?

In other words, there's two things about variables that the rfc does not seem to specify a format for: whether they should be matching across slashes or not, and whether they could be absent or not. Unless I misread it. I feel both of those would be useful to have however.

I'm ok with trying to follow the rfc, I'm just a bit apprehensive that it might complicate things and that we'll have to extend it in an ad hoc way anyway. I'll try to rewrite the examples though to try to be closer to it, see how it looks.

[skiadas]

I made a gist so we can have a place to edit the description while we're discussing this.

https://gist.github.com/skiadas/6023597

[skiadas]

@scothis Re question 2: I agree, I don't like the lack of determinism that keeping them in an object has. One alternative is to provide them as an array, but it feels silly to add an array around single key-value pairs. We would end up with:

routes = [
   { 'route1': 'method1' },
   { 'route2': 'method2' }
]

Feels like unneeded extra syntax.

Forcing some sort of CSS-style specificity is interesting. We might still run into the problem that CSS has however, where multiple routes might have the same specificity. CSS resolves it by using the order rules appear in the file, but I doubt we'd want to go that way.

It is still not entirely clear to me that we'd want to force only one handler to match. I could probably get on board with having one match per wirespec component that has a routes section. In that case, a reasonable compromise I think would be to use a CSS-style specificity but warn that in the event of two matching routes with the same specificity the result of which handler runs cannot be guaranteed.

[briancavalier]

Sorry, just now getting to respond to this. Thanks for the great discussion so far, guys. A few overall thoughts:

My original thinking for wire routing was that it should be as utterly simple as possible. Some really smart people have put a lot of thought and energy into creating really nice routing packages. So, my thinking is that designing a relatively flexible DSL that would allow integrating other routing packages is the most important thing.

I do think we should provide a basic routing plugin that covers most simple cases. We could also integrate other routing packages (such as Crossroads, Backbone routing, etc. via their own wire plugins) ... if we've done it right, they can all use the same (or at least very similar) DSL. This can help with adoption, as well, since people might already be using a routing package (e.g. Backbone).

Ok, on to some specifics

pushState vs. hash

I think we should default to the newer pushState tech. We can potentially offer hash support as a fallback or opt-out ... or, we could point people to the more sophisticated plugins (which would, of course, need to exist!)

Firing the current URL route on startup

Good question. Seems like we should pick a default, and offer an option for people who want it the other way. What do other routing packages do?

Multiple matches

I think all matches should be triggered. But, we should probably look to what other routers do. Anyone know?

DSL

Something like what you proposed, @skiadas, seems pretty reasonable, and is similar to what I was thinking:

myComponent: {
   create: //...
   routes: {
      '<route1>': '<any standard wire connection pipeline>'
      '<route2>': '<any standard wire connection pipeline>'
      // etc.
   }
}

The one problem that creates is wanting to invoke 2 pipelines for the same route within the same routes facet. I'm not sure how often that would cause a headache. Thoughts?

[scothis]

@briancavalier that's a really good point. If we can leverage an existing router and wrap it in a wire.js DSL that's a huge win with little work.

[briancavalier]

@scothis Yep!

@skiadas Since you've been working with Backbone, what do you think about the idea of trying to build a routing plugin around the Backbone router? I think it could be a great way to flesh out details of the DSL in the context of adapting a real routing package--which hopefully would prove the idea that we can do it for multiple routing packages.

[scothis]

At the very least it will be a good PoC

[skiadas]

Yeah getting backbone's routing thing wrapped shouldn't be too bad, and it does I believe offer a way to use pushState for example. The problem is that each of the routing packages seems to be doing things their own way, and finding a common denominator can be hard.

For instance, Backbone's router uses an object literal exactly as we want it, but utilizes the order in which the routes are written, using underscore's _.keys, to force an order on them, and only activates the first that matches. The idea being that more generic routes can appear further down in the list and will only trigger if the more specific ones don't.

Crossroads on the other hand expects the routes to be added via an addRoute method, and specifies the priority in that order. It again will by default only trigger the first that matches, but it does offer a way to specify that a route would be "greedy", i.e. that it should always try to be matched, and it would contain enough information for the handler to know if it is not the first one that matched. It also seems to have a clear separation between the routes and the handlers. You can "add routes" without handlers or with handlers, and later on you can add handlers to the Crossroads.routed signal that triggers on every parse.

So the first issue that needs to be resolved for any kind of implementation is this, that most current routing systems default to one matched route executed, and base the order on the order in which they appeared (or a custom priority number). So we need to decide how much we want to follow them on that.
One way around it might be to default to a first match ran, but to allow them to write routes in an object literal, but expect them to write a bit more if they want to "fix" an order or specify a route that is meant to be more of a catchall. For example it might look something like this

routes: {
    'foo/bar/p{num}': handler1,     // Matches foo/bar/p123 say
    'foo/bar/{others}': {
        handler: handler2,
        priority: -2              // Ensure it runs at lower priority (0 being default priority)
    },
    'foo/{anypath}?' :  handler3,      // Will match anything starting with a foo/ 
    '' : handler4,             // Empty string. Catchall for other routes. Only runs if the previous don't match
    '' : {                        // Catchall for logging purposes
        handler: handler5,
        greedy: true               // Will try to match every time.
                                   // Greedy will default to priority -100 unless otherwise specified
    }
}

I think, as most routes match uniquely, that for the most part the order does not matter and it would not be too bad to expect the user to write one-two more lines of code for their more generic catchall routes. So relying on the keys order in a "for .. in" loop, much as I hate it, might be the way around it. But it definitely is the case that most of these packages default to only one route matched.

The other issue is that of syntax, in that different frameworks follow very different syntax. We can start with Backbone's syntax if you like, either way we'll have to convert it to other packages if and when we want to support them. Backbone allows variable matches via colon in front ":foo", allows matching multiple components across slashes via "*foo", and allows optional components by including them in parentheses. so it allows for something like: "docs/:section(/:subsection)" to match both docs/foo and docs/foo/bar. It seems simple enough and probably a good place to start, but it is very different from the URI specification. Crossroads follows its own syntax as well. One option might be to offer some sort of transformation functions that go back and forth between different routing systems? So that one could specify the routes in whatever system they prefer, then mark the format, perhaps in the plugin's loading parameters? Something like:

$plugins: [
    { module: 'wire/route', format: 'backbone' }
]

Thoughts?

I'll try to wrap a thin layer around Backbone's router for now. But if we can resolve those two design issues it would be helpful.

[briancavalier]

Sorry for taking a break on this, @skiadas. I like the direction in your latest comment, and I'm actually thinking that the best way to answer some of the open questions (primarily around formats and converting/transforming between them) would be to implement a proof-of-concept. Have you had any luck wrapping Backbone's router? Run into any problems or questions?

One simple way to handle format differences initially would simply be to punt :) That is, allow the routing string to be an opaque thing that is passed through directly to whatever router plugin is in play. That makes switching routing systems harder, but I think we could tackle the transformations later, like you said, possibly by supporting conversions/transformations.

[skiadas]

Hi @briancavalier actually I'm the one who's been on hiatus for a week or two ;). I've got a couple of deadlines at the moment, but I hope to get something going by the weekend.

[briancavalier]

No worries. I'll probably be slow to respond over the next two weeks, but I'll do my best. Def looking forward to seeing what you've come up with.

[skiadas]

@briancavalier I'm actually not sure I'll be very reliable for the foreseeable future, a lot of things piled up on me atm. I do plan to work on this but can't promise any timetable, so depending on how pressing it is you might have to move forward without me.