[PIP-71][SQL]Pulsar SQL migrate SchemaHandle to presto decoder by hnail · Pull Request #8422 · apache/pulsar

hnail · 2020-11-02T05:49:40Z

Motivation

PIP-71: https://github.com/apache/pulsar/wiki/PIP-71:-Pulsar-SQL-migrate-SchemaHandle-to-presto-decoder

Pip-Doc : [PIP-71][SQL]Migrate SchemaHandle to Presto-decoder

In current version , pulsar-presto deserialize fields rely on SchemaHandler , but this causes the following restrictions :

Metadata: current nested field is dissociate with presto ParameterizedType , It treated nested field as a separated field , so presto compiler can’t understand the type hierarchy . nested field should be Row type in presto (e.g. Hive struct type support) . In the same way，array \ map type also shoud associate with presto ParameterizedTypes.
Decoder : SchemaHandler is hard to work with RecordCursor.getObject() to support ROW,MAP,ARRAY .etc

The motivations of this pull request :

PulsarMetadata take advantage of ParameterizedType to describe row/array/map Type instead of resolve nested columns in pulsar-presto connecter.
Customize RowDecoder | RowDecoderFactory | ColumnDecoder to work with pulsar interface, and with some our own extensions compare to presto original version , we can support more type for backward compatible (e.g.
TIMESTAMP\DATE\TIME\Real\ARRAY\MAP\ROW support).
Decouple avro or schema type with pulsar-presto main module (RecordSet,ConnectorMetadata .etc ), aim to friendly with other schema type ( ProtobufNative 、thrift etc..).

Modifications

Describe in PIP-71: Pulsar SQL migrate SchemaHandle to presto decoder

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (yes )
The public API: (no)
The schema: ( no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation

Does this pull request introduce a new feature? (yes)
[PIP][SQL]Migrate SchemaHandle to Presto-decoder

…grate_decoder

hnail · 2020-11-10T14:32:25Z

/pulsarbot run-failure-checks

codelipenghui · 2020-11-11T13:52:15Z

@gaoran10 Could you please help review this PR?

codelipenghui · 2020-11-17T03:45:07Z

move to 2.8.0 first.

jiazhai · 2020-11-22T13:31:33Z

@gaoran10 to help review this PR

sijie · 2020-11-30T03:45:41Z

@hnail Can you rebase this pull request to the latest master?

…tect

…grate_decoder

hnail · 2020-12-23T08:28:10Z

@hnail Great work. Could you rebase the branch with apache/master first?

@gaoran10 Thanks a lot for your review and help with this PR, i fixed as your suggest. can you help re-review it ?

Anonymitaet · 2020-12-29T08:19:02Z

Confirmed w/ @congbobo184, no need to update doc for this PR.

…l_migrate_decoder

hnail · 2021-01-19T06:56:06Z

/pulsarbot run-failure-checks

gaoran10 · 2021-01-24T15:10:56Z

/pulsarbot run-failure-checks

jiazhai · 2021-01-28T03:31:18Z

/pulsarbot run-failure-checks

jiazhai · 2021-01-29T01:16:32Z

/pulsarbot run-failure-checks

…grate_decoder

hnail · 2021-01-29T09:29:57Z

/pulsarbot run-failure-checks

gaoran10 · 2021-01-30T03:59:33Z

/pulsarbot run-failure-checks

codelipenghui · 2021-01-30T15:50:00Z

@hnail @gaoran10 Could you please help take a look the failed test?

Error:  Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:testCompile (default-testCompile) on project pulsar-presto-connector-original: Compilation failure
Error:  /Users/runner/work/pulsar/pulsar/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/TestPulsarRecordCursor.java:[352,40] method asyncReadEntries in class org.apache.bookkeeper.mledger.impl.ManagedCursorImpl cannot be applied to given types;
Error:    required: int,org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntriesCallback,java.lang.Object,org.apache.bookkeeper.mledger.impl.PositionImpl
Error:    found: int,java.lang.Object,java.lang.Object
Error:    reason: actual and formal argument lists differ in length

The interesting thing this why other tests are passed. Might be related to the other tests only run the install for profile core-modules ?

hnail · 2021-01-31T08:26:31Z

@hnail @gaoran10 Could you please help take a look the failed test?

Error:  Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:testCompile (default-testCompile) on project pulsar-presto-connector-original: Compilation failure
Error:  /Users/runner/work/pulsar/pulsar/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/TestPulsarRecordCursor.java:[352,40] method asyncReadEntries in class org.apache.bookkeeper.mledger.impl.ManagedCursorImpl cannot be applied to given types;
Error:    required: int,org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntriesCallback,java.lang.Object,org.apache.bookkeeper.mledger.impl.PositionImpl
Error:    found: int,java.lang.Object,java.lang.Object
Error:    reason: actual and formal argument lists differ in length

The interesting thing this why other tests are passed. Might be related to the other tests only run the install for profile core-modules ?

@codelipenghui fixed. cc @gaoran10 @jiazhai

jiazhai · 2021-02-01T14:56:11Z

@hnail Thanks a lot for this great feature.

apache#8422)" This reverts commit a1107ad.

[SQL] migrate SchemaHandle to decoder

86fb725

hnail changed the title ~~[SQL][WIP]migrate SchemaHandle to Presto-decoder~~ [SQL]migrate SchemaHandle to Presto-decoder Nov 3, 2020

wangguowei added 2 commits November 9, 2020 11:23

Merge branch 'master' of https://github.com/apache/pulsar into sql_mi…

703c15c

…grate_decoder

commit unit-tests

f94e770

hnail changed the title ~~[SQL]migrate SchemaHandle to Presto-decoder~~ [PIP][SQL]Migrate SchemaHandle to Presto-decoder Nov 9, 2020

wangguowei added 4 commits November 9, 2020 19:40

fix Pair bug

4d2755b

codestyle fix

98d5ea3

codestyle fix v2

9dd57e0

codestyle fix v3

66be5b2

wangguowei added 5 commits November 10, 2020 22:36

codestyle fix v4

e61a37d

codestyle fix v5

8f93600

codeStyle fix v6

203dc36

codeStyle fix v7

aa636b9

add License

6cfe40e

sijie requested review from codelipenghui, jerrypeng and merlimat November 11, 2020 08:54

sijie assigned hnail Nov 11, 2020

sijie added this to the 2.7.0 milestone Nov 11, 2020

sijie added the area/sql Pulsar SQL related features label Nov 11, 2020

hnail changed the title ~~[PIP][SQL]Migrate SchemaHandle to Presto-decoder~~ [PIP-71][SQL]Pulsar SQL migrate SchemaHandle to presto decoder Nov 11, 2020

wangguowei added 2 commits November 12, 2020 14:08

add snappy license

592130b

codeStyle fix

7c5f93c

codelipenghui modified the milestones: 2.7.0, 2.8.0 Nov 17, 2020

wangguowei added 2 commits December 23, 2020 15:24

add keyValue\Primitive schema test && add schema cyclic definition de…

11c0cef

…tect

Merge branch 'master' of https://github.com/apache/pulsar into sql_mi…

7f20d17

…grate_decoder

gaoran10 mentioned this pull request Jan 17, 2021

[Pulsar SQL] Some optimized points in PR 8422 #9224

Closed

sijie mentioned this pull request Jan 17, 2021

ISSUE-9224: [Pulsar SQL] Some optimized points in PR 8422 streamnative/pulsar-archived#2021

Open

hnail and others added 3 commits January 18, 2021 16:04

Merge branch 'master' into sql_migrate_decoder

6091b22

Merge branch 'sql_migrate_decoder' of github.com:hnail/pulsar into sq…

b91a0e4

…l_migrate_decoder

merge master

c54c1d0

Merge branch 'master' of https://github.com/apache/pulsar into sql_mi…

16626f2

…grate_decoder

jiazhai approved these changes Jan 29, 2021

View reviewed changes

codelipenghui approved these changes Jan 30, 2021

View reviewed changes

merge master

73dadc6

jiazhai merged commit a1107ad into apache:master Feb 1, 2021

congbobo184 mentioned this pull request Feb 2, 2021

[Pulsar sql] Support keyValue Schema add integration test. #9388

Merged

merlimat mentioned this pull request Feb 4, 2021

CI is broken on master failing on SQL integration test #9470

Closed

merlimat added a commit to merlimat/pulsar that referenced this pull request Feb 4, 2021

Revert "[PIP-71][SQL]Pulsar SQL migrate SchemaHandle to presto decoder (

f927671

apache#8422)" This reverts commit a1107ad.

merlimat mentioned this pull request Feb 4, 2021

Revert "[PIP-71][SQL]Pulsar SQL migrate SchemaHandle to presto decoder (#8422)" #9471

Closed

sijie mentioned this pull request Feb 4, 2021

ISSUE-9470: CI is broken on master failing on SQL integration test streamnative/pulsar-archived#2122

Closed

hnail deleted the sql_migrate_decoder branch February 5, 2021 10:12

zymap mentioned this pull request Mar 1, 2021

[Pulsar SQL] Fix Pulsar SQL query bytes schema data error #9631

Merged

Uh oh!

Conversation

hnail commented Nov 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

hnail commented Nov 10, 2020

Uh oh!

codelipenghui commented Nov 11, 2020

Uh oh!

codelipenghui commented Nov 17, 2020

Uh oh!

jiazhai commented Nov 22, 2020

Uh oh!

sijie commented Nov 30, 2020

Uh oh!

hnail commented Dec 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Anonymitaet commented Dec 29, 2020

Uh oh!

hnail commented Jan 19, 2021

Uh oh!

gaoran10 commented Jan 24, 2021

Uh oh!

jiazhai commented Jan 28, 2021

Uh oh!

jiazhai commented Jan 29, 2021

Uh oh!

hnail commented Jan 29, 2021

Uh oh!

gaoran10 commented Jan 30, 2021

Uh oh!

codelipenghui commented Jan 30, 2021

Uh oh!

hnail commented Jan 31, 2021

Uh oh!

jiazhai commented Feb 1, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

hnail commented Nov 2, 2020 •

edited

Loading

hnail commented Dec 23, 2020 •

edited

Loading