Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
542 commits
Select commit Hold shift + click to select a range
c83515d
Merge branch 'hotfix_1.10_4.0.x_33303' into 1.10_release_4.0.x
Dec 22, 2020
07b9550
Merge branch 'hotfix_1.8_3.10.x_33223' into 1.8_release_3.10.x
Dec 23, 2020
24f96b5
Merge branch '1.8_release_3.10.x' into 1.8_release_4.0.x
Dec 23, 2020
494a33b
Merge branch '1.8_release_4.0.x' into 1.10_release_4.0.x
Dec 23, 2020
5ed12d5
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Dec 23, 2020
4ec99ee
[fix-33227][LogMiner]statement关闭失败不要抛出异常
kanata163 Dec 23, 2020
cdbf65e
Merge remote-tracking branch 'origin/hotfix_1.8_release_4.0.x_31891' …
yanghuaiGit Dec 23, 2020
00560f3
Merge remote-tracking branch 'origin/1.8_release_4.0.x' into temp_1.1…
yanghuaiGit Dec 23, 2020
5047185
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Dec 23, 2020
20de296
[fix-33442][LogMiner]LogMiner日志、SQL优化
kanata163 Dec 24, 2020
23ebba1
[feat]【metadata-phoneix] 修改命名为phoenix5
lvyanquan Dec 25, 2020
24c82ee
[fix-33442][LogMiner]jsqlparser版本从1.2升级至3.2,增加SQL中字段值为',"时,字段长度的校验
kanata163 Dec 25, 2020
bf1ba35
[feat][metadata-vertica] 修改vertica的pom依赖
lvyanquan Dec 25, 2020
95d25b2
[feat][metadataphoenix] 修改类加载方式
lvyanquan Dec 26, 2020
e4c7ba8
[feat][metadata hbase] config中存放zookeeper地址
lvyanquan Dec 28, 2020
237c66f
[fix-33442][LogMiner] 增加线程重试,解决SQL存在\导致的解析出错,优化错误提示
kanata163 Dec 28, 2020
d4f2f3c
[fix-33442][LogMiner] 1、解决redoSql为空导致空指针问题;2、增加preScn,查询时冗余日志组
kanata163 Dec 29, 2020
18fdd25
[feat][socket] 实现功能
lvyanquan Dec 29, 2020
2b7c587
Merge branch 'feat_1.10_socket' into feat_1.10_4.1.x_mergeSocket
lvyanquan Dec 29, 2020
f0a15c2
Merge branch '1.10_release_4.1.x' into 1.10_test_4.1.x
kanata163 Dec 30, 2020
c3d745b
[feat][socket] 代码优化
lvyanquan Dec 30, 2020
6b7a41b
[feat][socket] 代码优化
lvyanquan Dec 30, 2020
7e7e480
[fix-34152][logminer]1.8_3.10支持重试功能
yanghuaiGit Dec 30, 2020
09a073f
[feat][socket] 增加单元测试
lvyanquan Dec 30, 2020
19b03e4
[feat][socket] 增加单元测试
lvyanquan Dec 31, 2020
a6cefad
Merge branch '1.8_release_3.10.x' into 1.8_release_4.0.x
Dec 31, 2020
95975bd
Merge branch '1.8_release_4.0.x' into 1.10_release_4.0.x
Dec 31, 2020
1e34b03
Merge remote-tracking branch 'origin/1.10_release_4.0.x' into temp_1.…
yanghuaiGit Dec 31, 2020
d37ce29
[opt-34152][logminer]代码合并冲突解决
yanghuaiGit Dec 31, 2020
30471fc
[feat][metadataphoenix] 增加列族信息
lvyanquan Jan 3, 2021
33437f4
[feat][metadatahbase] 修改pom shade命名
lvyanquan Jan 4, 2021
2555d76
[feat][metadatavertica] 代码优化及单元测试
lvyanquan Jan 4, 2021
b48465b
[feat][metadathbase] zookeeperhelper 设置为非单例模式
lvyanquan Jan 4, 2021
d333dd0
[fix][core][kafka] remove org.codehaus.jackson.map.ObjectMapper.
FlechazoW Jan 4, 2021
0079015
[feat][metadatahbase] 解决guava类冲突问题
lvyanquan Jan 4, 2021
533639c
[feat][socket] 增加encoding参数,修改PlainDecoder为TextDecoder
lvyanquan Jan 4, 2021
8c648b9
Merge branch 'feat_1.10_4.1.x_mergeSocket' into '1.10_test_4.1.x'
kanata163 Jan 4, 2021
c2cf292
[feat][metadatahbase] shade core改为metadatahbase
lvyanquan Jan 4, 2021
cce2111
[feat][metadatahbase] 使用shade过后的pom依赖
lvyanquan Jan 4, 2021
dfe88fe
[feat][metadatahbase] hbase版本修改为1.2.1
lvyanquan Jan 4, 2021
ce6601a
[feat][metadatahbase] guava 改成12.0.1
lvyanquan Jan 4, 2021
7510089
[feat][hbase] 修改hbase shade方式
lvyanquan Jan 4, 2021
711e624
[feat][hbase] guava依赖改为1.12.1
lvyanquan Jan 4, 2021
0f732ac
[feat][metadatahbase] 代码优化及增加zk单元测试
lvyanquan Jan 4, 2021
3825ebd
[feat][metadatahbase] 修改json键值对中的key值
lvyanquan Jan 5, 2021
b4ae087
[feat][metadataphoenix] 修改常量名
lvyanquan Jan 5, 2021
ad879f9
[feat][socket] socket reader 增加setEncoding方法
lvyanquan Jan 5, 2021
365aed9
Merge branch 'feat_1.10_4.1.x_mergeSocket' into '1.10_test_4.1.x'
kanata163 Jan 5, 2021
09ba4ce
[feat][metadatahbase] 修改默认path值
lvyanquan Jan 5, 2021
b9ee644
[fix-34164]phoenix数据源JDBC URL支持不填写zk端口
Jan 5, 2021
9644908
[feat-33094][kafka]kafka reader支持多通道消费不同分区,支持从指定offset、timestamp消费,支持…
kanata163 Jan 5, 2021
6c503d8
Merge branch '1.10_test_4.1.x' into feat_1.10_kafkaMutilChannel
kanata163 Jan 5, 2021
4d5c90b
[feat-33094][kafka]merge
kanata163 Jan 5, 2021
e031d4a
[feat][metadatahbase] 增加单元测试
lvyanquan Jan 5, 2021
cbd8fd3
[feat][hbase] 去掉不必要的修改
lvyanquan Jan 5, 2021
15be54b
[feat][metadatahbase] 修改hbase pom版本为1.3.1
lvyanquan Jan 5, 2021
facc952
[feat][restapi] revert shade 打包
lvyanquan Jan 5, 2021
7bc85f8
[hotfix][core] 解决读插件抛出的异常没有被抛出的问题
lvyanquan Jan 6, 2021
c5cf6b3
[feat][metadata phoenix、hbase] 代码优化与增加注释
lvyanquan Jan 6, 2021
f995fdd
[feat][metadata] 为test类增加licence
lvyanquan Jan 6, 2021
3e61aa9
[feat][metadatahbase] zookeeper在finally中关闭
lvyanquan Jan 6, 2021
389aa7c
Merge branch 'feat_1.10_tmpPhoenix' into '1.10_test_4.1.x'
kanata163 Jan 6, 2021
ef20ad3
[feat-33094][kafka]更新文档,startupMode参数修改,plain修改为text
kanata163 Jan 6, 2021
cac0220
[hotfix][core] 解决编译问题,去掉重复定义的常量
lvyanquan Jan 6, 2021
8e0f0d9
[feat-33094][kafka]更新文档格式
kanata163 Jan 6, 2021
4837351
Merge branch 'tmp_1.10_mergeTest' into '1.10_test_4.1.x'
kanata163 Jan 6, 2021
0bf483d
[fix-34363][core] fix local模式运行失败,类找不到
FlechazoW Jan 6, 2021
a3b31e9
[feat-33094][kafka]更新文档格式
kanata163 Jan 6, 2021
1fe9308
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Jan 6, 2021
f3f368c
Merge branch '1.10_release_4.0.x' into hotfix_1.10_4.0.x_33442
kanata163 Jan 6, 2021
3f87816
[fix-33094][logMiner]解决SQL解析异常,数据丢失等问题
kanata163 Jan 6, 2021
36221ee
[feat-33094][kafka]删除不必要的校验
kanata163 Jan 6, 2021
bc0df43
Merge branch '1.10_test_4.1.x' into feat_1.10_kafkaMutilChannel
kanata163 Jan 6, 2021
0faef26
Merge branch 'feat_1.10_kafkaMutilChannel' into '1.10_test_4.1.x'
Jan 6, 2021
117a5b5
【fix-34160】[ftp]解决ftp大文件读取导致 completePendingCommand 超时失败,增加超时时间soTimeout
yanghuaiGit Jan 7, 2021
0597c03
[fix][hbase表未找到][34385]
Jan 7, 2021
eb13029
Merge branch 'hotfix_1.10_4.1.x_34385' into '1.10_test_4.1.x'
kanata163 Jan 7, 2021
88c0eec
[hotfix-34395][metadataphoenix] 解决phoenix表创建时间为null的问题
lvyanquan Jan 7, 2021
94402b4
[fix-34392] [metadata-hbase-reader] namespace 报错修复
Jan 7, 2021
60a6cc8
Merge branch 'hotfix_1.10_4.1.x_34395' into '1.10_test_4.1.x'
kanata163 Jan 7, 2021
f2dfbbc
Merge branch 'hotfix_1.10_4.1.x_34392' into '1.10_test_4.1.x'
kanata163 Jan 7, 2021
79bab35
[fix-34439][hdfs] timestamp类型同步到hive string类型,精度精确到毫秒
yanghuaiGit Jan 7, 2021
954bac7
Merge branch 'hotfix_1.8_3.10.x_34164' of D:\idea_workspace\git_works…
Jan 7, 2021
8bece7d
Merge remote-tracking branch 'origin/1.8_release_3.10.x' into hotfix_…
Jan 7, 2021
377b4f5
[opt-34164]删除给url赋值的多余参数,使得Phoenix、Phoenix5离线采集插件支持不配置zk端口
Jan 7, 2021
5c1bfba
Merge branch '1.10_test_4.1.x' into 1.10_release_4.1.x
Jan 7, 2021
53cc5bc
Merge remote-tracking branch 'origin/1.10_release_4.0.x' into temp_1.…
yanghuaiGit Jan 8, 2021
87470e8
[fix-34445][logminer]加载日志文件到logminer里,出现文件加载多次问题,将查询日志文件sql语句的group b…
Jan 8, 2021
fc04e62
[fix-34445][logminer] logminer 报错 加载日志文件重复,将日志查找sql group by first_ch…
yanghuaiGit Jan 8, 2021
877a883
[fix-34516][core]解决kafka序列化json失败问题
kanata163 Jan 8, 2021
77f41cc
[fix-34521][stream]解决脚本配置restore为true时,空指针异常
kanata163 Jan 8, 2021
cee9875
[fix-34537][core]写入写出字节数统计指标优化
kanata163 Jan 8, 2021
96c8b0c
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Jan 8, 2021
fbfe5a4
[fix-34475][hbase reader]修复int字段类型读取错误
Jan 8, 2021
4bbae40
Merge branch '1.8_release_3.10.x' into 1.8_release_4.0.x
Jan 8, 2021
844fc15
Merge branch '1.8_release_4.0.x' into temp_1.10_release_4.0.x_merge
kanata163 Jan 8, 2021
ab28e48
Merge branch 'hotfix_1.8_3.10_34439' into 1.8_release_3.10.x
Jan 8, 2021
836976d
Merge branch '1.8_release_3.10.x' into 1.8_release_4.0.x
Jan 8, 2021
ec1bece
Merge branch 'hotfix_1.8_4.0.x_34516' into 1.8_release_4.0.x
Jan 10, 2021
c3becd9
[fix-34392][metadata hbase reader]增加对schema的判断
Jan 11, 2021
37967ad
Merge branch 'hotfix_1.10_4.1.x_34392' into '1.10_test_4.1.x'
kanata163 Jan 11, 2021
9af7c99
Merge branch '1.8_release_4.0.x' into 1.10_release_4.0.x
Jan 11, 2021
4c5bf12
[hotfix-34356][core] 解决任务日志抛出异常的问题
lvyanquan Jan 11, 2021
e0a24d0
[fix-34194][hdfs][hive]对hadoop的高可用配置增加校验 由参数isHa控制是否效验,默认为true
yanghuaiGit Jan 12, 2021
7cd90bb
[fix][phoenix metadata reader]schema为空默认default
Jan 12, 2021
3766870
Merge branch 'hotfix_1.8_3.10.x_34356' into 1.8_release_3.10.x
Jan 12, 2021
24d85f1
Merge branch '1.8_release_3.10.x' into 1.8_release_4.0.x
Jan 12, 2021
576dfc1
Merge branch '1.8_release_4.0.x' into 1.10_release_4.0.x
Jan 12, 2021
b0fecf4
Merge branch '1.10_release_4.0.x' into temp_1.10_release_4.1.x_merge
kanata163 Jan 12, 2021
369718d
merge
kanata163 Jan 12, 2021
a2b1c3e
[fix][metadata hbase reader]修复hbase表大小不准确
Jan 12, 2021
c847b74
Merge branch 'hotfix_1.10_4.1.x_34619' into '1.10_test_4.1.x'
kanata163 Jan 12, 2021
26d0b8a
Merge branch 'hotfix_1.10_4.1.x_34618' into '1.10_test_4.1.x'
kanata163 Jan 12, 2021
1721300
Merge branch '1.10_release_4.1.x' into 1.10_test_4.1.x
kanata163 Jan 12, 2021
536bd41
Merge branch 'hotfix_1.8_3.10_34160' into 1.8_release_3.10.x
Jan 13, 2021
c730322
Merge branch '1.8_release_3.10.x' into 1.8_release_4.0.x
Jan 13, 2021
9b20f18
Merge branch '1.8_release_4.0.x' into 1.10_release_4.0.x
Jan 13, 2021
5f8d1e4
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Jan 13, 2021
30709d3
[feat-34621] [metatdata phoenix reader] phoenix支持kerberos认证
Jan 14, 2021
d42f032
[feat-34621][metdadata phoenix reader] 代码优化
Jan 14, 2021
9a57345
Merge branch 'feat_1.10_4.1.x_34621' into '1.10_test_4.1.x'
kanata163 Jan 14, 2021
6e57c58
[fix-34619][metadata hbase reader] 增加单位转换工具方法,字节输出单位为B
Jan 14, 2021
81513c8
[fix-34647][metatdata vertica reader]分区字段解析不准确
Jan 14, 2021
e043d48
[fix-34619][flinkx core] 增加方法注释
Jan 14, 2021
5ec7675
Merge branch 'hotfix_1.10_4.1.x_34619' into '1.10_test_4.1.x'
kanata163 Jan 14, 2021
1098c6f
[fix-34647][metadata vertical reader] 代码优化
Jan 14, 2021
973dc40
Merge branch 'hotfix_1.10_4.1.x_34647' into '1.10_test_4.1.x'
kanata163 Jan 14, 2021
6269af8
Merge branch '1.10_test_4.1.x' into 1.10_release_4.1.x
Jan 14, 2021
bafb072
[feat-34480][launcher]FlinkX提交和开源分支保持一致,支持-p自定义入参
kanata163 Jan 15, 2021
85bdaca
Merge branch 'feat_1.10_4.1.x_34480' into '1.10_test_4.1.x'
Jan 15, 2021
34e2316
[fix-34541][sqlserver cdc]SqlServer CDC插件采集时间字段优化,删除ingestion字段,增加ts字段
kanata163 Jan 14, 2021
e0018b7
[fix-34621][metadata phenoix reader]改为ugi doas 认证方式
Jan 15, 2021
840010b
[feat-34758][doc]FlinkX文档同步
kanata163 Jan 15, 2021
be0c246
Merge branch 'feat_1.10_4.1.x_34758' into '1.10_test_4.1.x'
Jan 15, 2021
7389a3c
[feat-34621][metadata base phoenix5 reader]将关闭连接放在close format方法中防止重复…
Jan 15, 2021
10faeaa
Update pom.xml
Jan 15, 2021
12b16f9
[feat-34621][metadata phenoix reader] code format
Jan 15, 2021
0f3e33f
Merge branch 'feat_1.10_4.1.x_34621' into '1.10_test_4.1.x'
kanata163 Jan 15, 2021
2f570f8
Merge branch '1.10_test_4.1.x' into hotfix_1.10_4.1.x_34537
kanata163 Jan 15, 2021
e188e34
Merge branch 'hotfix_1.10_4.1.x_34537' into '1.10_test_4.1.x'
Jan 15, 2021
984f533
[fix-34773][metadata phenoix reader]default库表字段丢失
Jan 15, 2021
1ca52f5
Merge branch 'hotfix_1.10_4.1.x_34773' into '1.10_test_4.1.x'
kanata163 Jan 15, 2021
d991513
Merge branch '1.8_release_3.10.x' into 1.8_release_4.0.x
Jan 15, 2021
11c03b7
Merge branch '1.8_release_4.0.x' into 1.10_release_4.0.x
Jan 15, 2021
a303094
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Jan 15, 2021
bd56c44
[fix-34829][mongdbreader]解决创建分片超时问题,collection.countDocuments()获取总数时在…
yanghuaiGit Jan 19, 2021
d8a6f36
Merge branch 'hotfix_1.8_4.0_34194' into 1.8_release_4.0.x
Jan 19, 2021
2b39763
Merge branch '1.8_release_4.0.x' into 1.10_release_4.0.x
Jan 19, 2021
d968279
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Jan 19, 2021
476e7bc
[doc][kingbasewriter] 增加writer对schema和table配置的注释
lvyanquan Jan 19, 2021
15dcb34
Merge branch 'doc_4.1.x_kingbase' into '1.10_test_4.1.x'
kanata163 Jan 19, 2021
4636245
Merge branch 'hotfix_1.8_3.10.x_34164' into 1.8_release_3.10.x
Jan 20, 2021
7e47a23
[fix-34817]修改postgresql离线采集插件给table赋值代码
Jan 20, 2021
cb04705
Merge remote-tracking branch 'origin/1.8_release_3.10.x' into hotfix_…
Jan 20, 2021
993d0fa
[fix-34164]优化Phoenix支持不配置zk端口的问题
Jan 20, 2021
8b7b08f
[fix-34164]优化Phoenix支持不配置zk端口的问题
Jan 20, 2021
e98bd51
[fix-34914][kafka]修复初始化client判断错误
kanata163 Jan 20, 2021
9d62115
Merge branch 'hotfix_1.10_4.1.x_34914' into '1.10_test_4.1.x'
Jan 20, 2021
e019c84
Merge remote-tracking branch 'origin/1.8_release_4.0.x' into hotfix_1…
Jan 20, 2021
a645ac1
[fix-34817]修改本地分支pom文件和test示例代码
Jan 20, 2021
2108b7f
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Jan 21, 2021
5c6cef6
Merge remote-tracking branch 'origin/hotfix_1.8_4.0.x_34817' into 1.1…
Jan 21, 2021
f9bc48c
[fix-34817][postgresql]将PostgresqlWriter中schema改为局部变量
Jan 25, 2021
3e779e1
Merge branch 'hotfix_1.8_4.0.x_34475' into 1.8_release_4.0.x
Jan 25, 2021
59fea87
[fix-34475][flinkx hbase reader]代码优化
Jan 25, 2021
24e0699
Merge branch 'hotfix_1.8_4.0.x_34475' into 1.8_release_4.0.x
Jan 25, 2021
cd3528f
Merge remote-tracking branch 'origin/1.8_release_4.0.x' into tmp_1.10…
Jan 25, 2021
c64d18b
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Jan 25, 2021
e89921d
Merge branch 'hotfix_1.8_3.10_34829' into 1.8_release_3.10.x
Jan 26, 2021
37b8875
Merge branch '1.8_release_3.10.x' into 1.8_release_4.0.x
Jan 26, 2021
276b174
Merge branch '1.10_release_3.10.x' into 1.10_release_4.0.x
Jan 26, 2021
3bee244
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Jan 26, 2021
77fa332
[fix-35133][binlog][sqlserverCDC][logminer] 实时插件builder里的check方法修改,将异…
yanghuaiGit Jan 27, 2021
209663a
[feat-34484][kb]内部分支新增kafka提取value写入MySQL
Jan 27, 2021
4a9b180
Merge branch 'feat_1.10_4.1.x_addKafkaToMySQL' into '1.10_test_4.1.x'
kanata163 Jan 27, 2021
5334ace
[fix-35236][logminer] oracle10设置session dateformat时 需要关闭preparement,关闭资源
yanghuaiGit Jan 29, 2021
aa4ac46
[fix-35236][logminer] oracle10设置session dateformat时 需要关闭preparement,关闭资源
yanghuaiGit Jan 29, 2021
aa87288
[fix-35236][logminer] oracle10设置session dateformat时 需要关闭preparement,关闭资源
yanghuaiGit Jan 29, 2021
56a5497
[fix-35248][logminer] java.lang.NoClassDefFoundError: org/mortbay/lo…
yanghuaiGit Jan 31, 2021
9ef863d
Merge branch '1.8_release_4.0.x' into 1.10_release_4.0.x
Feb 1, 2021
ee65eb3
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Feb 1, 2021
67c1a13
Merge branch 'hotfix_1.8_4.0.x_35236' into 1.8_release_4.0.x
Feb 1, 2021
c804b84
Merge branch '1.8_release_4.0.x' into 1.10_release_4.0.x
Feb 1, 2021
1f4de0c
Merge remote-tracking branch 'origin/1.10_release_4.0.x' into tmp_1.1…
yanghuaiGit Feb 1, 2021
e26f3b7
[feat-382][restfulapi]根据最新需求优化restful相关代码
yanghuaiGit Feb 1, 2021
670f548
[feat-382][restful]restful代码优化
yanghuaiGit Feb 3, 2021
d10a6e2
[feat-382][restful]restful代码优化,增加文档
yanghuaiGit Feb 3, 2021
b11f9b9
[feat-382][test]test模块修改
yanghuaiGit Feb 3, 2021
19e08c4
Merge remote-tracking branch 'origin/feat_1.10_release_4.0.x_restfula…
yanghuaiGit Feb 3, 2021
3fd46e8
[feat-382][restapi]metaparam修改
yanghuaiGit Feb 3, 2021
e974cfa
[feat-382][restapi]按照代码规范 修改代码
yanghuaiGit Feb 3, 2021
bdb5ada
[fix-35326][hdfs] 校验hdfs是否开启ha功能去除
yanghuaiGit Feb 4, 2021
9508670
[feat-30875][flinkx kafka writer]kafka默认按表写入不同分区
Feb 5, 2021
bdc6a72
[feat-382][restapi]部分参数key修改
yanghuaiGit Feb 5, 2021
960a90f
[fix-35437][rdb-writer] 判断是否是字符串类型 增加varchar2 nvarchar2场景
yanghuaiGit Feb 5, 2021
d23ea38
[feat-382][restapi]如果是实时任务,则任务抛出异常结束,如果是离线任务,就任务正常结束
yanghuaiGit Feb 7, 2021
5f7b7c4
[feat-382][restapi]判断变量表达式是否满足a+-b格式的判断进行修改
yanghuaiGit Feb 8, 2021
3ca8910
[feat-30875][flinkx kafka write]代码抽到公共模块
Feb 8, 2021
7d4ce8c
[feat-30875][flinkx kafka writer]代码格式规范
Feb 8, 2021
ff40ce3
[feat-30875][flinkx kafka writer]kafka按固定字段写入分区
Feb 8, 2021
09f4332
Merge branch '1.8_release_3.10.x' into 1.8_release_4.0.x
Feb 9, 2021
4789bcd
Merge branch '1.10_release_3.10.x' into 1.10_release_4.0.x
Feb 9, 2021
6dff16d
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Feb 9, 2021
3de2d74
[feat-30875][flinkx kafka writer]修复分区数为负的bug
Feb 10, 2021
082aa9a
Merge branch 'hotfix_1.8_4.0_x_35326' into 1.8_release_4.0.x
Feb 17, 2021
fab33c2
Merge branch '1.8_release_4.0.x' into 1.10_release_4.0.x
Feb 17, 2021
f11973f
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Feb 17, 2021
1edb743
Merge branch 'hotfix_1.10_4.0.x_34817' into 1.10_release_4.0.x
Feb 17, 2021
9d25d2b
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Feb 17, 2021
8305951
[feat-382][restapi]代码优化
yanghuaiGit Feb 18, 2021
a2f3ad1
[feat-30875][flinkx kafka writer]代码优化
Feb 18, 2021
7403af8
[feat-30875][flinkx pom]注释test 模块
Feb 18, 2021
e040475
[feat-30875][flinkx pom]回退pom文件
Feb 18, 2021
90eaefe
[feat-382][restapi]增加配置里的 format参数 转换为SimpleDateFormat
yanghuaiGit Feb 18, 2021
c446788
Merge branch '1.10_release_4.1.x' into 1.10_test_4.1.x
kanata163 Feb 18, 2021
e35697a
[feat-30875][flinkx kafka writer]代码优化
Feb 18, 2021
f7d902a
[feat-30875][flinkx kafka writer]代码优化
Feb 18, 2021
83badf0
Merge branch 'feat_1.10_4.1.x_restfulapireader_382' into '1.10_test_4…
kanata163 Feb 18, 2021
739be40
[feat-30875][flinkx kafka writer]文档完善,代码优化
Feb 18, 2021
ecc6b50
[fix-35376][hivewrite]hivewriter编译错误 增加对应导包
yanghuaiGit Feb 18, 2021
d91a0ae
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Feb 18, 2021
9215eb8
Merge remote-tracking branch 'origin/1.10_release_4.1.x' into 1.10_te…
yanghuaiGit Feb 18, 2021
c455ece
[feat-30875][flinkx kafka writer]代码优化
Feb 18, 2021
0b5a987
[feat-30875][flinkx kafka writer]代码优化
Feb 18, 2021
228f515
Merge branch 'feat_1.10_4.1.x_30875' into '1.10_test_4.1.x'
kanata163 Feb 18, 2021
3e7360f
[fix-486][restapi]构建请求参数优化
yanghuaiGit Feb 19, 2021
6fd748a
Merge branch 'hotfix_1.10_4_1x_restfulapireader_382_486' into '1.10_t…
kanata163 Feb 19, 2021
575daf1
[fix-35407][rdb]JDBC URL不要覆盖useCursorFetch,rewriteBatchedStatements等参数
kanata163 Feb 20, 2021
165db16
[feat-382] get请求含有特殊字符串进行encode
yanghuaiGit Feb 20, 2021
137f8d8
[feat-382][restapi] get请求含有特殊字符串进行encode
yanghuaiGit Feb 20, 2021
cfcddc1
[feat-382][restapi] get请求含有特殊字符串进行encode
yanghuaiGit Feb 20, 2021
8610354
Merge branch 'feat_1.10_4.1.x_restfulapireader_382' into '1.10_test_4…
kanata163 Feb 20, 2021
09f7bae
[feat-35377][launcher][logminer][binlog]sonar bug漏洞修复
yanghuaiGit Feb 22, 2021
8d5572b
Merge remote-tracking branch 'origin/1.10_release_4.0.x' into temp_1.…
yanghuaiGit Feb 22, 2021
77c5952
[fix-35432][core][phoenix5][logminer][cassandra] sonar漏洞解决
yanghuaiGit Feb 23, 2021
88622d2
[fix-35432][core]异常日志打印优化
yanghuaiGit Feb 23, 2021
0555830
merge hotfix_1.10_4.0_x_35432 to 1.10_release_4.0.x
Feb 23, 2021
b6dc405
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Feb 23, 2021
82bf659
Merge branch '1.10_test_4.1.x' into 1.10_release_4.1.x
Feb 25, 2021
9074e03
[fix-35459][hivewriter]hive支持接收到数据全部写入message字段
yanghuaiGit Feb 25, 2021
505b8d3
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Mar 1, 2021
4ad8063
[fix-35555][jdbcwriter]update时,column需要 加上双引号 否则在oracle的字段是小写的时候,未加引号…
yanghuaiGit Mar 16, 2021
118ca0e
Merge branch '1.10_release_4.0.x' into 1.10_release_4.1.x
Mar 16, 2021
a1f60cd
Merge remote-tracking branch 'origin/1.10_test_4.0.x' into temp_1.10_…
yanghuaiGit Mar 17, 2021
756c734
Merge branch 'temp_1.10_4.1.x_merge' into 1.10_release_4.1.x
Mar 17, 2021
65a6bd3
[fix-35871][core][test]解决launcher提交缺少-p参数报错问题
kanata163 Mar 25, 2021
4650b16
[fix-33302][core] 解决finally吞日志的问题
kanata163 Mar 26, 2021
7a35094
[fix-35555][rdb]update时,column需要 加上双引号 否则在oracle的字段是小写的时候,未加引号,默认为大写导致错误
kanata163 Mar 26, 2021
fdee7cb
[opt][test]test模块pom文件删除重复引用
kanata163 Mar 26, 2021
7e7cc14
[docs][docs]1、fix #368 钉钉群二维码失效问题;2、readme默认由英文改为中文
kanata163 Mar 26, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 47 additions & 37 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,47 +3,57 @@ FlinkX

[![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html)

English | [中文](README_CH.md)

# Communication

- We are recruiting **Big data platform development engineers**. If you want more information about the position, please add WeChat ID [**ysqwhiletrue**] or email your resume to [sishu@dtstack.com](mailto:sishu@dtstack.com).

- We use [DingTalk](https://www.dingtalk.com/) to communicate, you can search the group number [**30537511**] or scan the QR code below to join the communication group
[English](README_EN.md) | 中文

# 技术交流

- 招聘**Flink研发工程师**,如果有兴趣可以联系思枢(微信号:ysqwhiletrue)<BR>
Flink开发工程师JD要求:<BR>
1.负责袋鼠云基于Flink的衍生框架数据同步flinkx和实时计算flinkstreamsql框架的开发;<BR>
2.调研和把握当前最新大数据实时计算技术,将其中的合适技术引入到平台中,改善产品,提升竞争力;<BR>
职位要求:<BR>
1、本科及以上学历,3年及以上的Flink开发经验,精通Java,熟悉Scala、Python优先考虑;<BR>
2、熟悉Flink原理,有基于Flink做过二次源码的开发,在github上贡献者Flink源码者优先;<BR>
3、有机器学习、数据挖掘相关经验者优先;<BR>
4、对新技术有快速学习和上手能力,对代码有一定的洁癖;<BR>
加分项:<BR>
1.在GitHub或其他平台上有过开源项目<BR>
可以添加本人微信号ysqwhiletrue,注明招聘,如有意者发送简历至[sishu@dtstack.com](mailto:sishu@dtstack.com)

- 我们使用[钉钉](https://www.dingtalk.com/)沟通交流,可以搜索群号[**30537511**]或者扫描下面的二维码进入钉钉群

<div align=center>
<img src=docs/images/ding.jpg width=300 />
<img src=docs/images/ding.jpg width=300 />
</div>

# Introduction

* **FlinkX is a distributed offline and real-time data synchronization framework based on flink widely used in 袋鼠云, which realizes efficient data migration between multiple heterogeneous data sources.**
# 介绍
* **FlinkX是在是袋鼠云内部广泛使用的基于flink的分布式离线和实时的数据同步框架,实现了多种异构数据源之间高效的数据迁移。**

Different data sources are abstracted into different Reader plugins, and different data targets are abstracted into different Writer plugins. In theory, the FlinkX framework can support data synchronization of any data source type. As a set of ecosystems, every time a set of new data sources is connected, the newly added data sources can realize intercommunication with existing data sources.
不同的数据源头被抽象成不同的Reader插件,不同的数据目标被抽象成不同的Writer插件。理论上,FlinkX框架可以支持任意数据源类型的数据同步工作。作为一套生态系统,每接入一套新数据源该新加入的数据源即可实现和现有的数据源互通。

<div align=center>
<img src=docs/images/template.png width=300 />
</div>

FlinkX is a data synchronization tool based on Flink. FlinkX can collect static data, such as MySQL, HDFS, etc, as well as real-time changing data, such as MySQL binlog, Kafka, etc. FlinkX currently includes the following features:
FlinkX是一个基于Flink的批流统一的数据同步工具,既可以采集静态的数据,比如MySQL,HDFS等,也可以采集实时变化的数据,比如MySQL binlog,Kafka等。FlinkX目前包含下面这些特性:

- Most plugins support concurrent reading and writing of data, which can greatly improve the speed of reading and writing;
- 大部分插件支持并发读写数据,可以大幅度提高读写速度;

- Some plug-ins support the function of failure recovery, which can restore tasks from the failed location and save running time; [Failure Recovery](docs/restore.md)
- 部分插件支持失败恢复的功能,可以从失败的位置恢复任务,节约运行时间;[失败恢复](docs/restore.md)

- The Reader plugin for relational databases supports interval polling. It can continuously collect changing data; [Interval Polling](docs/offline/reader/mysqlreader.md)
- 关系数据库的Reader插件支持间隔轮询功能,可以持续不断的采集变化的数据;[间隔轮询](docs/offline/reader/mysqlreader.md)

- Some databases support opening Kerberos security authentication; [Kerberos](docs/kerberos.md)
- 部分数据库支持开启Kerberos安全认证;[Kerberos](docs/kerberos.md)

- Limit the reading speed of Reader plugins and reduce the impact on business databases;
- 可以限制reader的读取速度,降低对业务数据库的影响;

- Save the dirty data when writing data;
- 可以记录writer插件写数据时产生的脏数据;

- Limit the maximum number of dirty data;
- 可以限制脏数据的最大数量;

- Multiple running modes: Local,Standalone,Yarn Session,Yarn Per;
- 支持多种运行模式;

The following databases are currently supported:
FlinkX目前支持下面这些数据库:

| | Database Type | Reader | Writer |
|:----------------------:|:--------------:|:-------------------------------:|:-------------------------------:|
Expand Down Expand Up @@ -75,42 +85,42 @@ The following databases are currently supported:
| | Hive | | [doc](docs/offline/writer/hivewriter.md) |
| Stream Synchronization | Kafka | [doc](docs/realTime/reader/kafkareader.md) | [doc](docs/realTime/writer/kafkawriter.md) |
| | EMQX | [doc](docs/realTime/reader/emqxreader.md) | [doc](docs/realTime/writer/emqxwriter.md) |
| | RestApi || [doc](docs/realTime/writer/restapiwriter.md) |
| | RestApi |[doc](docs/realTime/reader/restapireader.md) | [doc](docs/realTime/writer/restapiwriter.md) |
| | MySQL Binlog | [doc](docs/realTime/reader/binlogreader.md) | |
| | MongoDB Oplog | [doc](docs/realTime/reader/mongodboplogreader.md)| |
| | PostgreSQL WAL | [doc](docs/realTime/reader/pgwalreader.md) | |
| | Oracle LogMiner| [doc](docs/realTime/reader/LogMiner.md) | |
| | Sqlserver CDC| [doc](docs/realTime/reader/sqlservercdc.md) | |
| | Oracle LogMiner | [doc](docs/realTime/reader/LogMiner.md) | |
| | Sqlserver CDC | [doc](docs/realTime/reader/sqlservercdc.md) | |

# Fundamental
In the underlying implementation, FlinkX relies on Flink, and the data synchronization task will be translated into StreamGraph and executed on Flink. The basic principle is as follows:
# 基本原理
在底层实现上,FlinkX依赖Flink,数据同步任务会被翻译成StreamGraph在Flink上执行,基本原理如下图:
<div align=center>
<img src=docs/images/diagram.png width=700 />
</div>

# Quick Start
# 快速开始

Please click [Quick Start](docs/quickstart.md)
请点击[快速开始](docs/quickstart.md)

# General Configuration
# 通用配置

Please click [General Configuration](docs/generalconfig.md)
请点击[插件通用配置](docs/generalconfig.md)

# Statistics Metric
# 统计指标

Please click [Statistics Metric](docs/statistics.md)
请点击[统计指标](docs/statistics.md)

# Kerberos

Please click [Kerberos](docs/kerberos.md)
请点击[Kerberos](docs/kerberos.md)

# Questions

Please click [Questions](docs/questions.md)
请点击[Questions](docs/questions.md)

# How to contribute FlinkX
# 如何贡献FlinkX

Please click [Contribution](docs/contribution.md)
请点击[如何贡献FlinkX](docs/contribution.md)

# License

Expand Down
84 changes: 37 additions & 47 deletions README_CH.md → README_EN.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,57 +3,47 @@ FlinkX

[![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html)

[English](README.md) | 中文

# 技术交流

- 招聘**Flink研发工程师**,如果有兴趣可以联系思枢(微信号:ysqwhiletrue)<BR>
Flink开发工程师JD要求:<BR>
1.负责袋鼠云基于Flink的衍生框架数据同步flinkx和实时计算flinkstreamsql框架的开发;<BR>
2.调研和把握当前最新大数据实时计算技术,将其中的合适技术引入到平台中,改善产品,提升竞争力;<BR>
职位要求:<BR>
1、本科及以上学历,3年及以上的Flink开发经验,精通Java,熟悉Scala、Python优先考虑;<BR>
2、熟悉Flink原理,有基于Flink做过二次源码的开发,在github上贡献者Flink源码者优先;<BR>
3、有机器学习、数据挖掘相关经验者优先;<BR>
4、对新技术有快速学习和上手能力,对代码有一定的洁癖;<BR>
加分项:<BR>
1.在GitHub或其他平台上有过开源项目<BR>
可以添加本人微信号ysqwhiletrue,注明招聘,如有意者发送简历至[sishu@dtstack.com](mailto:sishu@dtstack.com)

- 我们使用[钉钉](https://www.dingtalk.com/)沟通交流,可以搜索群号[**30537511**]或者扫描下面的二维码进入钉钉群
English | [中文](README.md)

# Communication

- We are recruiting **Big data platform development engineers**. If you want more information about the position, please add WeChat ID [**ysqwhiletrue**] or email your resume to [sishu@dtstack.com](mailto:sishu@dtstack.com).

- We use [DingTalk](https://www.dingtalk.com/) to communicate, you can search the group number [**30537511**] or scan the QR code below to join the communication group

<div align=center>
<img src=docs/images/ding.jpg width=300 />
<img src=docs/images/ding.jpg width=300 />
</div>

# 介绍
* **FlinkX是在是袋鼠云内部广泛使用的基于flink的分布式离线和实时的数据同步框架,实现了多种异构数据源之间高效的数据迁移。**
# Introduction

* **FlinkX is a distributed offline and real-time data synchronization framework based on flink widely used in 袋鼠云, which realizes efficient data migration between multiple heterogeneous data sources.**

不同的数据源头被抽象成不同的Reader插件,不同的数据目标被抽象成不同的Writer插件。理论上,FlinkX框架可以支持任意数据源类型的数据同步工作。作为一套生态系统,每接入一套新数据源该新加入的数据源即可实现和现有的数据源互通。
Different data sources are abstracted into different Reader plugins, and different data targets are abstracted into different Writer plugins. In theory, the FlinkX framework can support data synchronization of any data source type. As a set of ecosystems, every time a set of new data sources is connected, the newly added data sources can realize intercommunication with existing data sources.

<div align=center>
<img src=docs/images/template.png width=300 />
</div>

FlinkX是一个基于Flink的批流统一的数据同步工具,既可以采集静态的数据,比如MySQL,HDFS等,也可以采集实时变化的数据,比如MySQL binlog,Kafka等。FlinkX目前包含下面这些特性:
FlinkX is a data synchronization tool based on Flink. FlinkX can collect static data, such as MySQL, HDFS, etc, as well as real-time changing data, such as MySQL binlog, Kafka, etc. FlinkX currently includes the following features:

- 大部分插件支持并发读写数据,可以大幅度提高读写速度;
- Most plugins support concurrent reading and writing of data, which can greatly improve the speed of reading and writing;

- 部分插件支持失败恢复的功能,可以从失败的位置恢复任务,节约运行时间;[失败恢复](docs/restore.md)
- Some plug-ins support the function of failure recovery, which can restore tasks from the failed location and save running time; [Failure Recovery](docs/restore.md)

- 关系数据库的Reader插件支持间隔轮询功能,可以持续不断的采集变化的数据;[间隔轮询](docs/offline/reader/mysqlreader.md)
- The Reader plugin for relational databases supports interval polling. It can continuously collect changing data; [Interval Polling](docs/offline/reader/mysqlreader.md)

- 部分数据库支持开启Kerberos安全认证;[Kerberos](docs/kerberos.md)
- Some databases support opening Kerberos security authentication; [Kerberos](docs/kerberos.md)

- 可以限制reader的读取速度,降低对业务数据库的影响;
- Limit the reading speed of Reader plugins and reduce the impact on business databases;

- 可以记录writer插件写数据时产生的脏数据;
- Save the dirty data when writing data;

- 可以限制脏数据的最大数量;
- Limit the maximum number of dirty data;

- 支持多种运行模式;
- Multiple running modes: Local,Standalone,Yarn Session,Yarn Per;

FlinkX目前支持下面这些数据库:
The following databases are currently supported:

| | Database Type | Reader | Writer |
|:----------------------:|:--------------:|:-------------------------------:|:-------------------------------:|
Expand Down Expand Up @@ -85,42 +75,42 @@ FlinkX目前支持下面这些数据库:
| | Hive | | [doc](docs/offline/writer/hivewriter.md) |
| Stream Synchronization | Kafka | [doc](docs/realTime/reader/kafkareader.md) | [doc](docs/realTime/writer/kafkawriter.md) |
| | EMQX | [doc](docs/realTime/reader/emqxreader.md) | [doc](docs/realTime/writer/emqxwriter.md) |
| | RestApi |[doc](docs/realTime/reader/restapireader.md) | [doc](docs/realTime/writer/restapiwriter.md) |
| | RestApi || [doc](docs/realTime/writer/restapiwriter.md) |
| | MySQL Binlog | [doc](docs/realTime/reader/binlogreader.md) | |
| | MongoDB Oplog | [doc](docs/realTime/reader/mongodboplogreader.md)| |
| | PostgreSQL WAL | [doc](docs/realTime/reader/pgwalreader.md) | |
| | Oracle LogMiner | [doc](docs/realTime/reader/LogMiner.md) | |
| | Sqlserver CDC | [doc](docs/realTime/reader/sqlservercdc.md) | |
| | Oracle LogMiner| [doc](docs/realTime/reader/LogMiner.md) | |
| | Sqlserver CDC| [doc](docs/realTime/reader/sqlservercdc.md) | |

# 基本原理
在底层实现上,FlinkX依赖Flink,数据同步任务会被翻译成StreamGraph在Flink上执行,基本原理如下图:
# Fundamental
In the underlying implementation, FlinkX relies on Flink, and the data synchronization task will be translated into StreamGraph and executed on Flink. The basic principle is as follows:
<div align=center>
<img src=docs/images/diagram.png width=700 />
</div>

# 快速开始
# Quick Start

请点击[快速开始](docs/quickstart.md)
Please click [Quick Start](docs/quickstart.md)

# 通用配置
# General Configuration

请点击[插件通用配置](docs/generalconfig.md)
Please click [General Configuration](docs/generalconfig.md)

# 统计指标
# Statistics Metric

请点击[统计指标](docs/statistics.md)
Please click [Statistics Metric](docs/statistics.md)

# Kerberos

请点击[Kerberos](docs/kerberos.md)
Please click [Kerberos](docs/kerberos.md)

# Questions

请点击[Questions](docs/questions.md)
Please click [Questions](docs/questions.md)

# 如何贡献FlinkX
# How to contribute FlinkX

请点击[如何贡献FlinkX](docs/contribution.md)
Please click [Contribution](docs/contribution.md)

# License

Expand Down
Binary file modified docs/images/ding.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,8 @@ public void run(SourceContext<OUT> ctx) throws Exception {
isRunning = splitIterator.hasNext();
}
}
} catch (Exception exception){
tryException = exception;
} finally {
isRunning = false;
try {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ protected String getUpdateSql(List<String> column, String leftTable, String righ
String prefixRight = StringUtils.isBlank(rightTable) ? "" : quoteTable(rightTable) + ".";
List<String> list = new ArrayList<>();
for(String col : column) {
list.add(prefixLeft + col + "=" + prefixRight + col);
list.add(prefixLeft + quoteColumn(col) + "=" + prefixRight + quoteColumn(col));
}
return StringUtils.join(list, ",");
}
Expand Down
5 changes: 0 additions & 5 deletions flinkx-test/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -441,11 +441,6 @@
<artifactId>flinkx-restapi-reader</artifactId>
<version>1.6</version>
</dependency>
<dependency>
<groupId>com.dtstack.flinkx</groupId>
<artifactId>flinkx-restapi-writer</artifactId>
<version>1.6</version>
</dependency>
<dependency>
<groupId>com.dtstack.flinkx</groupId>
<artifactId>flinkx-dm-reader</artifactId>
Expand Down