Skip to content

[fix](serde) fix split_by_delimiter missing backslash escape handling #61995

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
eldenmoon merged 1 commit into from
Apr 29, 2026
Merged

[fix](serde) fix split_by_delimiter missing backslash escape handling #61995

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion be/src/core/data_type_serde/complex_type_deserialize_util.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,9 @@ struct ComplexTypeDeserializeUtil {
std::vector<SplitResult> elements;
for (int pos = 0; pos < str.size; ++pos) {
char c = str.data[pos];
if (c == '"' || c == '\'') {
if (c == '\\' && pos + 1 < static_cast<int>(str.size)) {

@github-actions github-actions Bot Apr 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This helper is only used by the *_serde::from_string() implementations. The Stream Load JSON path described in the PR body still goes through NewJsonReader::_simdjson_write_data_to_column() and then DataTypeMapSerDe::deserialize_one_cell_from_json() when _is_load is true (be/src/format/json/new_json_reader.cpp:1084-1121). That map JSON path already has its own \ handling at be/src/core/data_type_serde/data_type_map_serde.cpp:244-245, and it never calls split_by_delimiter(). So this change does not appear to fix the reported Stream Load regression; it fixes the from_string() path instead. Can you either add a reproducer on the real load path and patch that path if needed, or narrow the PR description/tests to the string or JSONB-to-map conversion flow that this code actually changes?

@csun5285 csun5285 Apr 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new_json_reader doesn't parse the input directly into a map — it parses it into a string, and then casts that string into a map. The fix is exactly on the path that casts a string into a map.

++pos; // skip escaped character
} else if (c == '"' || c == '\'') {
if (!has_quote) {
quote_char = c;
has_quote = !has_quote;
Expand Down
57 changes: 57 additions & 0 deletions be/test/core/data_type_serde/data_type_serde_map_test.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@
#include "core/data_type/data_type_nullable.h"
#include "core/data_type/data_type_string.h"
#include "core/data_type/define_primitive_type.h"
#include "core/data_type_serde/complex_type_deserialize_util.h"
#include "core/field.h"
#include "core/types.h"
#include "storage/olap_common.h"
Expand Down Expand Up @@ -178,4 +179,60 @@ TEST_F(DataTypeMapSerDeTest, ArrowMemNotAligned) {
EXPECT_TRUE(st.ok());
}

// Stream Load JSON stores Map as String via to_json_string, then converts back
// via from_string → split_by_delimiter. The splitter must handle '\' escapes
// so that '\"' inside a value doesn't flip quote state and expose inner ':'/','.
TEST_F(DataTypeMapSerDeTest, SplitByDelimiterHandlesBackslashEscape) {
DataTypeSerDe::FormatOptions opts;
opts.map_key_delim = ':';
opts.collection_delim = ',';

auto make_map_type = []() {
auto str = std::make_shared<DataTypeNullable>(std::make_shared<DataTypeString>());
return std::make_shared<DataTypeMap>(str, str);
};

// split_by_delimiter: '\"' must not toggle quote state
// Input (after stripping outer {}): "k":"[{\"a\":\"b\\nc:"
// Expected: 2 elements — key "k" and value "[{\"a\":\"b\\nc:"
{
std::string inner = "\"k\":\"[{\\\"a\\\":\\\"b\\\\nc:\"";
StringRef str(inner.data(), inner.size());
auto result = ComplexTypeDeserializeUtil::split_by_delimiter(
str, [&](char c) { return c == opts.map_key_delim || c == opts.collection_delim; });
EXPECT_EQ(result.size(), 2u);
}

// from_string: value ending with ':' (map_key_delim) must not cause split error
// Simulates to_json_string output: {"k":"[{\"a\":\"b\\nc:"}
{
auto map_type = make_map_type();
auto col = map_type->create_column();
std::string map_str = "{\"k\":\"[{\\\"a\\\":\\\"b\\\\nc:\"}";
StringRef ref(map_str.data(), map_str.size());
EXPECT_TRUE(map_type->get_serde()->from_string(ref, *col, opts).ok());
EXPECT_EQ(col->size(), 1u);
}

// from_string: value ending with ',' (collection_delim) — same class of bug
{
auto map_type = make_map_type();
auto col = map_type->create_column();
std::string map_str = "{\"k\":\"[{\\\"a\\\":\\\"b\\\\nc,\"}";
StringRef ref(map_str.data(), map_str.size());
EXPECT_TRUE(map_type->get_serde()->from_string(ref, *col, opts).ok());
EXPECT_EQ(col->size(), 1u);
}

// Control: value ending with ')' (not a delimiter) — always worked
{
auto map_type = make_map_type();
auto col = map_type->create_column();
std::string map_str = "{\"k\":\"[{\\\"a\\\":\\\"b\\\\nc)\"}";
StringRef ref(map_str.data(), map_str.size());
EXPECT_TRUE(map_type->get_serde()->from_string(ref, *col, opts).ok());
EXPECT_EQ(col->size(), 1u);
}
}

} // namespace doris
2 changes: 1 addition & 1 deletion regression-test/data/jsonb_p0/test_jsonb_cast.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
1 \N
2 ['{\'x\':\'{"y":1}\', \'t\':\'{"y":2}\'}', '{"x":1}']
3 ['foo\'bar', 'foo"bar', 'foo\\'bar', 'foo\'\'bar']
3 ['foo\'bar', 'foo"bar', 'foo\'bar', 'foo\'\'bar']
4 ['\/some\/cool\/url', '/some/cool/url', 'a\\_\\c\\l\\i\\c\\k\\h\\o\\u\\s\\e']
6 changes: 3 additions & 3 deletions regression-test/data/jsonb_p0/test_jsonb_cast.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@
-- !select_1 --
1 \N
2 ["{'x':'{"y":1}', 't':'{"y":2}'}", "{"x":1}"]
3 ["foo'bar', 'foo"bar', 'foo\\'bar', 'foo''bar"]
3 ["foo'bar", "foo"bar", "foo'bar", "foo''bar"]
4 ["/some/cool/url", "/some/cool/url", "a\\_\\c\\l\\i\\c\\k\\h\\o\\u\\s\\e"]

-- !select_2 --
1 \N
2 ["{'x':'{"y":1}', 't':'{"y":2}'}", "{"x":1}"]
3 ["foo'bar', 'foo"bar', 'foo\\'bar', 'foo''bar"]
3 ["foo'bar", "foo"bar", "foo'bar", "foo''bar"]
4 ["/some/cool/url", "/some/cool/url", "a\\_\\c\\l\\i\\c\\k\\h\\o\\u\\s\\e"]
27 ["{"k1":"v1", "k2":200}"]
28 ["{"a.b.c":{"k1.a1":"v31", "k2":300},"a":"niu"}"]
Expand All @@ -18,7 +18,7 @@
-- !select_json --
1 \N
2 ["{'x':'{\\"y\\":1}', 't':'{\\"y\\":2}'}","{\\"x\\":1}"]
3 ["foo'bar', 'foo\\"bar', 'foo\\\\'bar', 'foo''bar"]
3 ["foo'bar","foo\\"bar","foo'bar","foo''bar"]
4 ["/some/cool/url","/some/cool/url","a\\\\_\\\\c\\\\l\\\\i\\\\c\\\\k\\\\h\\\\o\\\\u\\\\s\\\\e"]
27 ["{\\"k1\\":\\"v1\\", \\"k2\\":200}"]
28 ["{\\"a.b.c\\":{\\"k1.a1\\":\\"v31\\", \\"k2\\":300},\\"a\\":\\"niu\\"}"]
Expand Down
2 changes: 1 addition & 1 deletion regression-test/data/jsonb_p0/test_jsonb_unescaped.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
1 \N
2 ['{\'x\' : \'{"y" : 1}\', \'t\' : \'{"y" : 2}\'}', '{"x" : 1}']
3 ['foo\'bar', 'foo"bar', 'foo\\'bar', 'foo\'\'bar']
3 ['foo\'bar', 'foo"bar', 'foo\'bar', 'foo\'\'bar']
4 ['\/some\/cool\/url', '/some/cool/url', 'a\\_\\c\\l\\i\\c\\k\\h\\o\\u\\s\\e']
5 ["\"双引号\"", "反斜\\线"]
4 changes: 2 additions & 2 deletions regression-test/data/jsonb_p0/test_jsonb_with_unescaped_string.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,14 @@
-- !select_csv --
1 \N
2 ["{'x' : '{"y" : 1}', 't' : '{"y" : 2}'}", "{"x" : 1}"]
3 ["foo'bar', 'foo"bar', 'foo\\'bar', 'foo''bar"]
3 ["foo'bar", "foo"bar", "foo'bar", "foo''bar"]
4 ["/some/cool/url", "/some/cool/url", "a\\_\\c\\l\\i\\c\\k\\h\\o\\u\\s\\e"]
5 [""双引号"", "反斜\\线"]

-- !select_json --
1 \N
2 ["{'x' : '{"y" : 1}', 't' : '{"y" : 2}'}", "'{"x" : 1}'"]
3 ["foo'bar', 'foo"bar', 'foo\\'bar', 'foo''bar"]
3 ["foo'bar", "foo"bar", "foo\\'bar", "foo''bar"]
4 ["/some/cool/url", "/some/cool/url", "a\\_\\c\\l\\i\\c\\k\\h\\o\\u\\s\\e"]
5 [""双引号"", "反斜\\线"]

Loading