Choose Content-type for Update Requests

[uholzer]

As far as I know, SPARQL update requests are always sent with Content-type application/x-www-form-urlencoded. It would be nice if one could choose to send such requests as application/sparql-update instead. This is useful if one has a SPARQL endpoint that does not work with one of these content types.

Personally, I would like to have this feature for the following reason: RDFLib's SPARQLStore currently has its own implementation of SPARQL Updates and allows the user to choose the content type. I like replace the custom implementation with SPARQLWrapper without breaking the API.

[indeyets]

@uholzer do you have an idea how that would look API-wise?

My long-term idea was to create a base SPARQLWrapper which would be inherited by endpoint-specific classes (VirtuosoSPARQLWrapper, StardogSPARQLWrapper, 4StoreSparqlWrapper, etc.). Base class strictly follows the specification, while descendants cover implementation quirks, non-standard extensions, etc.

[uholzer]

How about a method setContentType or setUpdateContentType accepting only the values application/x-www-form-urlencode and application/sparql-update? (Or better introduce constants like MEDIA_TYPE_FORM and MEDIA_TYPE_UPDATE.)

I think it would be best if the base class SPARQLWrapper would be configurable to cover all choices outlined by the specification. Choosing among the two Content-types mentioned above is such an example. I think this would also make it easier to implement subclasses: A FusekiSPARQLWrapper (Fuseki not supporting one of the content-types) could override the setContentType method to be a noop without reimplementing the _createRequest method.

[wikier]

My preference would be, rather to expose the content type in the API, allow the update method to use according: http://www.w3.org/TR/2013/REC-sparql11-protocol-20130321/#update-operation

Would be that a solution to this issue?

[uholzer]

Sure, being able to choose between URL encoded post and direct post instead of setting the content-type is fine. Along the lines setUpdateEncoding(URL_ENCODED) and setUpdateEncoding(DIRECTLY_ENCODED). Alternatively, a method that takes a boolean setUseUpdateURLEncoding(true). A third option that comes to my mind would be to add an optional argument to setMethod: setMethod(POST, url_encoded=False).

[wikier]

Then, I proposed the following API addition:

SPARQLWrapper.setUpdateMethod(method)

where method is either URL-encoded (SPARQLWrapper.URLENCODED) or POST directly (SPARQLWrapper.POSTDIRECTLY) according SPARQL 1.1 Protocol.

To be consistent with the current behavior, SPARQLWrapper.URLENCODED will be used by default.

[wikier]

An early implementation is now available on master. Any testing will be welcomed.

[wikier]

@uholzer could you please validate the new api inclusions?

[uholzer]

There is a problem with SPARQL updates that contain non-ASCII characters. For example:

INSERT DATA { <urn:michel> <urn:says> "é" } 

causes an exeption of the form

  [...]
  File "/home/urs/.local/lib/python2.7/site-packages/SPARQLWrapper/Wrapper.py", line 502, in query
    return QueryResult(self._query())
  File "/home/urs/.local/lib/python2.7/site-packages/SPARQLWrapper/Wrapper.py", line 472, in _query
    response = urllib2.urlopen(request)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1181, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 833, in _send_output
    self.send(message_body)
  File "/usr/lib/python2.7/httplib.py", line 805, in send
    self.sock.sendall(data)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 61: ordinal not in range(128)

The following change fixed the problem for me:

diff --git a/SPARQLWrapper/Wrapper.py b/SPARQLWrapper/Wrapper.py
index 2efdc8e..cc155da 100644
--- a/SPARQLWrapper/Wrapper.py
+++ b/SPARQLWrapper/Wrapper.py
@@ -435,7 +435,7 @@ class SPARQLWrapper(object):
             request = urllib2.Request(uri)
             if self.updateMethod == POSTDIRECTLY:
                 request.add_header("Content-Type", "application/sparql-update")
-                request.add_data(self.queryString)          
+                request.add_data(self.queryString.encode('UTF-8'))          
             else: # URL-encoded                
                 request.add_header("Content-Type", "application/x-www-form-urlencoded")
                 request.add_data(encodedParameters)

Otherwise it works, thanks.

[indeyets]

@wikier I didn't do formal testing, just read the code. I think there is an issue: additional query parameters are lost in POSTDIRECTLY case. they should be moved into url if we can't send them in request body