Table: Need to encode special characters when serializing entity to XML

[joostdenijs]

Dev Estimate: 5
Test Estimate: 1

If you try to insert an entity with a "bad" character, decimal values 0-8,11,12,14-31, the XML is malformed and gets rejected by the server.

Comparing what this code emits with what the .NET Table Convienence layer emits, we see:

1.Troublesome characters are encoded with &#x_; encoding, so char(0x0E) becomes &#xE;.
2. Strings with whitespace characters (like 0x0B) have the xmlns:space="preserve" attribute added to their property:

<m:properties>
  <d:Email xmlns:space="preserve">&#xB;</d:Email>
 ...

[gcheng]

http://blogs.msdn.com/b/gongcheng/archive/2010/03/30/how-to-encode-non-ascii-characters-in-xml.aspx

[gcheng]

It is actually harder than it looks like to implement :(

[gcheng]

fix ready!

[jcookems]

Found that the fix incorrectly converts characters outside of the "bad" character set: 9, 10, 13. These should be allowed to flow through unconverted.

[jcookems]

See https://github.com/WindowsAzure/azure-sdk-for-java/pull/143/files#r1296044

The code is too aggressive:

if (charArray[index] < 0x20 || charArray[index] > 0x7f)

The following excerpt from http://en.wikipedia.org/wiki/Valid_characters_in_XML explains that

Unicode code points in the following ranges are valid in XML 1.0 documents:[1]
U+0009, U+000A, U+000D: these are the only C0 controls accepted in XML 1.0;

[jcookems]

This also breaks GB18030.

[christav]

The current codebase cheats. It encodes, for example, the byte 0x05 in the XML as . Note the leading '&' character is double-encoded.

This is problematic on a couple levels. First, the corresponding reader code doesn't decode the string, so you put in the Java literal string "\u0005", and what you end up with is ; it doesn't round trip correctly. The second issue is that there's no guarantee that other SDKs will do the same thing and interpret the resulting strings correctly.

Having said that, having the string � in an XML document is illegal; the Java stuff will throw an exception when it sees it; it doesn't even have to be straight binary, just that particular code point.

For now I'll leave it in place, and just restrict the range of code points to be escaped. We should consider doing the correct unescaping either as part of this WI or as a new one.

[jcookems]

The code now works for a majority of the characters required for GB-18030, but still fails for Chinese Extension-B characters (which are represented with surrogate characters in UTF-16). I updated the new test with this:

Entity entity = new Entity().setPartitionKey("001").setRowKey("insertEntityEscapeCharactersWorks")
...
     .setProperty("test8", EdmType.STRING, "surrogate pair \uD840\uDC00");
...
String actualTest8 = (String) result.getEntity().getProperty("test8").getValue();
assertEquals("surrogate pair \uD840\uDC00", actualTest8);