Fix the handling of supplementary characters (characters > U+FFFF)#66
Merged
Conversation
JNI uses a modified UTF-8 encoding. For supplementary characters, an invalid UTF-8 sequence was written to the database, which resulted in interoperability problems. The solution is to avoid UTF-8 in the native code and use the UTF-16 functions of SQLite (where possible). SQLite will then convert the UTF-16 to standards-compliant, unmodified UTF-8. This also fixes related bugs in JDBC3PreparedStatement and improves the "out of memory" handling in the native code. Fixed Issues: - https://bitbucket.org/xerial/sqlite-jdbc/issues/200/wrong-utf-8-decoding-of-unicode-code , same as #61 - https://bitbucket.org/xerial/sqlite-jdbc/issues/144/nativedbexec-throws-an-exception-without - https://bitbucket.org/xerial/sqlite-jdbc/issues/84/bug-in-nativedbc-bind_1blob - https://bitbucket.org/xerial/sqlite-jdbc/issues/70/setting-a-blob-in-prepstmt
Contributor
Author
|
I think that the continuous integration system has not compiled the native library, but used an old version of the native library instead. A new unit test that I have written fails because the old native library has been used. |
Owner
|
OK. I will prepare native binaries rebuilt with this fix. |
Owner
|
Thanks for the proper error handling and the improvement of the query execution by using new API of SQLite. |
Contributor
Author
|
Thank you for merging! |
Contributor
|
Just ran into this bug, glad it's already been fixed. However there doesn't seem to be a 3.9.1-SNAPSHOT version on sonatype, I only found 3.9.0. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JNI uses a modified UTF-8 encoding. For supplementary characters, an invalid UTF-8 sequence was written to the database, which resulted in interoperability problems. The solution is to avoid UTF-8 in the native code and use the UTF-16 functions of SQLite (where possible). SQLite will then convert the UTF-16 to standards-compliant, unmodified UTF-8.
This also fixes related bugs in JDBC3PreparedStatement and improves the "out of memory" handling in the native code.
Fixed Issues: