Client incorrectly rejects keys, buckets that are of the unicode datatype

[atl]

This is problematic because Python's JSON libraries return unicode objects. I ran into this issue when trying to marshall objects derived from a Link-only M/R job, but I imagine this will show up elsewhere as well. My attempt at a patch is in a9fea18 .

[atl]

Made some further corrections to my patch in 9140472 .

[roidrage]

Could you add a test for this as well?

[atl]

I would like to, but it's not going to happen in a timely manner: I'm in the midst of my own code push, and I'm unfamiliar with the project's test conventions. I might be more able to circle around and help in 2 weeks.

[roidrage]

Looking into turning this into a patch, so no worries.

[atl]

Okay, even this is non-ideal behavior. What's the real goal? URL-safe keys? My approach (for myself) might be to utf-7 encode/decode keys both on the way in and out.

[roidrage]

Yes, it's not perfect altogether. The real goal would be to have URL-safe keys indeed, to simply accept and encode/decode Unicode keys, just like you said. There's an open issue in our official bug tracker on this as well: https://issues.basho.com/show_bug.cgi?id=649

The bottom line we came down to after some more discussion is to always decode/encode keys from and to Unicode on store and fetch.

[atl]

Thanks, mattmatt. I actually was trying to address it at the client layer (consistency of URL-safe keys generated, encode/decode when crossing the boundary of the Python library), so my flip "solutions" are pretty trivial within that scope. Being an official Basho library, I suppose you do have to worry about behavior across all clients.

[danostrowski]

Thanks for patching this, what's the long term solution if this is a temporary one?

[seancribbs]

Closed https://issues.basho.com/show_bug.cgi?id=649 in favor of continuing discussion here.

[danostrowski]

Without the fix from Soren, you basically can't use Riak search, if my memory serves me right. Is there an objection to using his fix? People who already manually encode won't have a problem because he does a type check (though they could easily pass non-URL safe characters and that's not trivial to check for, I don't believe.)

[evanmcc]

@danostrowski do you know how to construct a failing test for this one? There've been some unicode fixes lately, and I'd like to figure out if this is still open or not.

[dowski]

Running into an issue with the temp-fix that was merged in above. We have keys that are opaque bytestrings - like '\x8b\xb06'. They aren't encoded data of any sort. When trying to use those keys, the Python client raises a TypeError and tells me Unicode keys are not supported..

According to the Riak docs on keys, "Keys in Riak are simply binary values (or strings) used to identify Objects." Seems like that should carry through to the client behavior as well.

Traceback (most recent call last):
  File "clear_test_bucket.py", line 14, in <module>
    new_optrs = optrs.new_binary('\x8b\xb06', data='\n 8bb0363933ab47a8b425d2d610574c93\x12*\n(3da541559918a808c2402bba5012f6c60b27661c')
  File "/usr/local/lib/python2.7/dist-packages/riak/bucket.py", line 304, in new_binary
    obj = RiakObject(self._client, self, key)
  File "/usr/local/lib/python2.7/dist-packages/riak/riak_object.py", line 48, in __init__
    raise TypeError('Unicode keys are not supported.')
TypeError: Unicode keys are not supported.