[Bug] Chinese characters garbled to ??? when synchronizing MySQL data with auto table creation via multi-table CDC mode of CREATE STREAMING JOB

[brantyou]

Search before asking

• I had searched in the issues and found no similar issues.

Version

4.1.1

What's Wrong?

When using the multi-table CDC mode of CREATE STREAMING JOB in Doris 4.1.1 to synchronize tables and data from a specified MySQL database with auto table creation enabled, all Chinese characters in the synchronized data are garbled and displayed as ???.
For the identical MySQL table, if we adopt the CREATE STREAMING JOB TVF mode: manually create a Doris primary key model table first, then perform data synchronization, the Chinese characters display normally without garbling.
We executed SHOW CREATE TABLE to check the DDL statements generated by Doris under the two modes, and found no obvious differences between them.
In the CDC mode synchronization task, the following parameters have already been appended to the MySQL jdbc_url:
?useUnicode=true&characterEncoding=utf-8&serverTimezone=Asia/Shanghai&zeroDateTimeBehavior=convertToNull
Yet Chinese content still turns into garbled ???.
The schema configuration of the corresponding MySQL table is as follows:
ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci ROW_FORMAT=DYNAMIC

What You Expected?

When synchronizing MySQL tables and data using Doris 4.1.1 multi-table CDC streaming job with auto table creation, Chinese text should be parsed and stored correctly, displaying normal Chinese characters instead of garbled ???.
The Chinese display effect should be consistent with the TVF streaming job mode (manually created Doris primary key tables work fine for Chinese content).

How to Reproduce?

Environment: Apache Doris 4.1.1, source MySQL table with charset utf8mb4
MySQL table config: ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci ROW_FORMAT=DYNAMIC
Create a multi-table CDC streaming job with auto table creation enabled to sync all tables from a specified MySQL database.
Append charset & timezone parameters to MySQL jdbc_url:
?useUnicode=true&characterEncoding=utf-8&serverTimezone=Asia/Shanghai&zeroDateTimeBehavior=convertToNull
Insert or read Chinese data in MySQL source table, wait for CDC synchronization to complete.
Query synced data in Doris, all Chinese characters show as ???.

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_bec3a9c3488a
This content is generated by AI for reference only.

Initial triage: this looks like a real Doris bug in the multi-table CDC write path, not a MySQL table charset problem and probably not an auto-created Doris DDL problem.

I checked the 4.1.1 code path against the local 4.1.1-rc01 tag. The auto-create path builds target columns from JDBC metadata in StreamingJobUtils.generateCreateTableCmds(), and getColumns() already expands MySQL varchar/char length by 3 for UTF-8 byte width. That matches the report that SHOW CREATE TABLE does not show a meaningful DDL difference.

The important difference is after deserialization:

• TVF mode: PipelineCoordinator.buildStreamRecords() writes each JSON record with record.getBytes(StandardCharsets.UTF_8).
• Multi-table CDC mode: PipelineCoordinator.writeRecords() sends records to DorisBatchStreamLoad with record.getBytes().

String.getBytes() without an explicit charset uses the CDC client JVM default charset. If that JVM default is not UTF-8, Chinese characters can be replaced by literal ? bytes before Stream Load receives the payload. This exactly matches the symptom where Chinese text becomes ???, and also explains why adding useUnicode=true&characterEncoding=utf-8 to the MySQL jdbc_url does not help: the loss happens later, while converting the deserialized JSON String to bytes for the multi-table Stream Load path. The Stream Load content type is already application/json;charset=UTF-8, but by that point the bytes may already contain ?.

The same default-charset call still exists on the local upstream/master ref, so this does not look limited to only the 4.1.1 release branch.

Suggested fix:

1. Change the multi-table CDC write path to encode records explicitly with UTF-8, for example:

batchStreamLoad.writeRecord(targetDb, dorisTable, record.getBytes(StandardCharsets.UTF_8));

2. Add a focused cdc-client test that writes a JSON record containing Chinese text through the multi-table writeRecords/DorisBatchStreamLoad path, preferably with a non-UTF-8 JVM default charset or by directly asserting the produced bytes.

Useful confirmation data from the reporter, if maintainers want to verify the environment:

• BE/cdc-client logs for the affected streaming job and task.
• The exact CREATE STREAMING JOB statement with credentials removed.
• MySQL SHOW VARIABLES LIKE 'character_set%'; and SHOW VARIABLES LIKE 'collation%';.
• A source-side sample such as SELECT col, HEX(col) ... and the corresponding Doris SELECT col, HEX(col) ....
• The CDC client JVM/default locale information if available, especially file.encoding, LANG, and LC_ALL.

Temporary mitigation, if a patch cannot be applied immediately: ensure the cdc-client JVM runs with UTF-8 as its default charset, but the code should still be fixed to avoid depending on process locale.

[brantyou]

TVF mode sql：

CREATE DATABASE IF NOT EXISTS lake_ods_test;
CREATE TABLE IF NOT EXISTS `lake_ods_test`.`ods_dkt_table` (
  `id` char(32) NOT NULL COMMENT '主键ID',
  `order_code` char(15) NOT NULL COMMENT '单号',
  `type` varchar(50) NOT NULL COMMENT '类型，中文',
  `remark` varchar(500) DEFAULT NULL COMMENT '补充说明',
  `created_on` datetime(3) NOT NULL COMMENT '创建时间',
  `updated_on` datetime(3) NOT NULL COMMENT '更新时间',
  `created_by` varchar(30) NOT NULL COMMENT '创建人',
  `updated_by` varchar(30) NOT NULL COMMENT '更新人',
  `version` int(11) NOT NULL DEFAULT '1' COMMENT '版本号'
)
DUPLICATE KEY(id, order_code,)
DISTRIBUTED BY HASH(id) BUCKETS 10
PROPERTIES( "replication_allocation" = "tag.location.default: 1")
;
CREATE JOB mysql_sync_ods_dkt_table
ON STREAMING
DO
INSERT INTO `lake_ods_test`.`ods_dkt_table`
SELECT 
  id,
  order_code,
  type,
  remark,
  created_on,
  updated_on,
  created_by,
  updated_by,
  version
FROM cdc_stream(
        "type"         = "mysql",
        "jdbc_url"     = "jdbc:mysql://127.0.0.1:3306",
        "driver_url"   = "https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.28/mysql-connector-java-8.0.28.jar",
        "driver_class" = "com.mysql.cj.jdbc.Driver",
        "user"         = "username",
        "password"     = "password",
        "database"     = "test",
        "table"        = "test_table",
        "offset"       = "initial"
)
;

CDC mode sql:

CREATE DATABASE IF NOT EXISTS lake_ods_test2;
CREATE JOB sync_db_test
ON STREAMING
FROM MYSQL(
        "jdbc_url"     = "jdbc:mysql://127.0.0.1:3306?useUnicode=true&characterEncoding=utf-8&serverTimezone=Asia/Shanghai&zeroDateTimeBehavior=convertToNull",
        "driver_url"   = "https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.28/mysql-connector-java-8.0.28.jar",
        "driver_class" = "com.mysql.cj.jdbc.Driver",
        "user"         = "username",
        "password"     = "password",
        "database"     = "test",
        "include_tables" = "test_table",
        "offset"       = "initial"
)
TO DATABASE lake_ods_test2 (
    "table.create.properties.replication_num" = "1"
)
;

in doris do:

SHOW VARIABLES LIKE 'character_set%';

result:

character_set_client	utf8mb4	utf8mb4	0
character_set_connection	utf8mb4	utf8mb4	0
character_set_results	utf8mb4	utf8mb4	0
character_set_server	utf8mb4	utf8mb4	0

do:

SHOW VARIABLES LIKE 'collation%';

result:

collation_connection	utf8mb4_0900_bin	utf8mb4_0900_bin	0
collation_database	utf8mb4_0900_bin	utf8mb4_0900_bin	0
collation_server	utf8mb4_0900_bin	utf8mb4_0900_bin	0

A source-side sample such as SELECT col, HEX(col) ... and the corresponding Doris SELECT col, HEX(col) ....
in mysql:

SELECT type, HEX(type) FROM test_table WHERE id='1';

result:

挖隧道、修道路	E68C96E99AA7E98193E38081E4BFAEE98193E8B7AF

in doris CDC mode:

SELECT type, HEX(type) FROM lake_ods_test2.test_table WHERE id='1';

result:

???????	3F3F3F3F3F3F3F

in doris CVT mode:

SELECT type, HEX(type) FROM lake_ods_test.test_table WHERE id='1';

result:

挖隧道、修道路	E68C96E99AA7E98193E38081E4BFAEE98193E8B7AF