Only count a key as an ancestor if there is a separator#141
Only count a key as an ancestor if there is a separator#141
Conversation
Stebalien
force-pushed
the
fix/ancestor-sep
branch
from
d47cece to
954d9e2
Compare
Stebalien
force-pushed
the
fix/ancestor-sep
branch
from
954d9e2 to
7a1443d
Compare
| } | ||
|
|
||
| if strings.HasPrefix(p, key.String()) { | ||
| if m.Prefix.IsDescendantOf(key) { |
There was a problem hiding this comment.
I'm now using proper key functions.
There was a problem hiding this comment.
Are the comments above this function still accurate? What should happen with /ao/e/ here?
There was a problem hiding this comment.
I'm not really sure what that comment is trying to say (what's A, B?). It would return the /ao/e/ mount point, and / as "rest". I've fixed the documentation and added a test.
| // We've found an ancestor (or equal) key. We might have | ||
| // more general datastores, but they won't contain keys | ||
| // with this prefix so there's no point in searching them. | ||
| break |
There was a problem hiding this comment.
This is #140 (comment). Thinking back on this, I think it was a bug.
Given the mounts ["/foo", "/"], lookupAll("/foo/bar") would return both (there's now a test for this). However, it should have only returned "/foo" because no keys under "/foo/bar" would end up in "/".
There was a problem hiding this comment.
Note: this may have been impacting the performance of GC.
There was a problem hiding this comment.
seems reasonable, this is a contract change but if it increases performance/reliability and no one is utilizing the obscure case then we may as well go for it.
Are we pretty confident that we're not doing queries for things like /blocks even though /blocks is mounted over /?
There was a problem hiding this comment.
We are, but that should be fine. That is, we're querying for blocks in /blocks, not for blocks in /. As of this change, lookupAll("/blocks"), should return only the datastore mounted at /blocks.
|
|
||
| if mountPts[i].Equal(prefix) || suffix.String() != "/" { | ||
| return nil | ||
| } |
There was a problem hiding this comment.
The fix in lookupAll should make this unnecessary.
| if prefix != "/" { | ||
| prefix += "/" | ||
| } | ||
| qr = NaiveFilter(qr, FilterKeyPrefix{prefix}) |
There was a problem hiding this comment.
We weren't cleaning/fixing keys.
…arator
Also make sure to clean and normalize keys before using them as prefixes.
BREAKING CHANGES:
* `myds.Query(Query{Prefix:"/foo"})` will no longer match "/foobar" (or even
"/foo"). This is usually what the user expects, we had a tendency to normalize
"/foo/" to "/foo" (when we clean keys), and many datastores can't efficiently
search for prefixes that aren't on path-boundaries anyways.
* Given a mount datastore with mounts `["/foo", "/"]`, `myds.Put("/foo", "bar")`
will put the value to the datastore mounted at "/", not "/foo", as the key "/"
and "" usually doesn't make sense.
While technically breaking, these changes are much more likely to fix bugs than
they are to break things.
Stebalien
force-pushed
the
fix/ancestor-sep
branch
from
7a1443d to
5598edf
Compare
Note: if we don't do this, we need to remember to fix the key transform datastore. |
|
@whyrusleeping says
|
|
Yeah, seems fine to me. We can add more docs around our key model to compensate for the potential weirdness |
aschmahmann
left a comment
aschmahmann
left a comment
There was a problem hiding this comment.
Overall LGTM. Left a few comments and requests for clarification
query/query_test.go
Outdated
| @@ -88,14 +88,12 @@ func TestNaiveQueryApply(t *testing.T) { | |||
|
|
|||
| q = Query{ | |||
| Limit: 3, | |||
There was a problem hiding this comment.
Limit isn't being tested anymore since we only have one result
There was a problem hiding this comment.
I've added some more keys to improve the tests.
| } | ||
|
|
||
| if strings.HasPrefix(p, key.String()) { | ||
| if m.Prefix.IsDescendantOf(key) { |
There was a problem hiding this comment.
Are the comments above this function still accurate? What should happen with /ao/e/ here?
| // We've found an ancestor (or equal) key. We might have | ||
| // more general datastores, but they won't contain keys | ||
| // with this prefix so there's no point in searching them. | ||
| break |
There was a problem hiding this comment.
seems reasonable, this is a contract change but if it increases performance/reliability and no one is utilizing the obscure case then we may as well go for it.
Are we pretty confident that we're not doing queries for things like /blocks even though /blocks is mounted over /?
| func (d *Datastore) lookup(key ds.Key) (ds.Datastore, ds.Key, ds.Key) { | ||
| for _, m := range d.mounts { | ||
| if m.Prefix.Equal(key) || m.Prefix.IsAncestorOf(key) { | ||
| if m.Prefix.IsAncestorOf(key) { |
There was a problem hiding this comment.
To clarify, is the intent here that if I've mounted ["/", "/foo"] and then I do a put to /foo that the key will end up in the / mount instead of the /foo mount?
Is there any reason to support this or would we be safer just throwing an error instead?
This should have already been committed...
3 participants
Also make sure to clean and normalize keys before using them as prefixes.
BREAKING CHANGES:
myds.Query(Query{Prefix:"/foo"})will no longer match "/foobar" (or even "/foo"). This is usually what the user expects, we had a tendency to normalize "/foo/" to "/foo" (when we clean keys), and many datastores can't efficiently search for prefixes that aren't on path-boundaries anyways.["/foo", "/"],myds.Put("/foo", "bar")will put the value to the datastore mounted at "/", not "/foo", as the key "/" and "" usually doesn't make sense.While technically breaking, these changes are much more likely to fix bugs than they are to break things.