[improve][io] Fix Kotlin version issue for functions built in Kotlin by abhilashmandaliya · Pull Request #19924 · apache/pulsar

abhilashmandaliya · 2023-03-25T06:16:48Z

Motivation

I was trying to implement a custom Sink connector using Kotlin. My Kotlin version was 1.8.10 and was using Jackson in my code. I saw a weird reflection-related error in my code. The debugging ended up with a finding that the Pulsar broker was bringing the Kotlin library dependency which was version 1.4.32 and that was getting loaded before my version. Hence my sink was using the older version of a specific class. While looking for Kotlin support in Pulsar code and feedback from the committers in this PR, I realized that Pulsar code is not using the Kotlin library directly but it is a transitive dependency. The latest stable version of the main dependency(okhttp3) bringing Kotlin(1.6.20) as a transitive dependency doesn't use the latest version of Kotlin(1.8.10). Hence, updated okhttp3, okio, and Kotlin separately for the convenience of an existing structure.

Putting the exception that I got here. (Not sure whether that is required):

Caused by: kotlin.reflect.jvm.internal.KotlinReflectionInternalError: Cannot create type for an unsupported classifier: class kotlin.Unit (Kotlin reflection is not available) (class kotlin.jvm.internal.ClassReference)
        at kotlin.reflect.full.KClassifiers.createType(KClassifiers.kt:48) ~[kotlin-reflect-1.8.10.jar:1.8.10-release-430(1.8.10)]
        at kotlin.reflect.full.KClassifiers.createType$default(KClassifiers.kt:42) ~[kotlin-reflect-1.8.10.jar:1.8.10-release-430(1.8.10)]
        at com.fasterxml.jackson.module.kotlin.KotlinAnnotationIntrospector.<clinit>(KotlinAnnotationIntrospector.kt:218) ~[jackson-module-kotlin-2.14.2.jar:2.14.2]
        at com.fasterxml.jackson.module.kotlin.KotlinModule.setupModule(KotlinModule.kt:121) ~[jackson-module-kotlin-2.14.2.jar:2.14.2]
        at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:853) ~[com.fasterxml.jackson.core-jackson-databind-2.13.4.2.jar:2.13.4.2]

Here kotlin.Unit class is loaded from Kotlin 1.4.32 whereas Jackson was compiled with Kotlin 1.5.32.

Modifications

Upgraded transitive org.jetbrains.kotlin:kotlin-stdlib library from 1.4.32 to 1.8.10
Upgraded transitive okhttp3 version from 4.9.3 to 4.10.0
Upgraded transitive okio version from 2.8.0 to 3.0.0

Verifying this change

Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository: abhilashmandaliya#1

nodece · 2023-03-25T17:06:43Z

        <test.additional.args/>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
-        <pulsar.broker.compiler.release></pulsar.broker.compiler.release>


This should be done intentionally.

@abhilashmandaliya to fix IDE warning, you can collapse the tag instead of removing it.

Anyway, keep one PR for one purpose (upgrade kotlin lib) can help you get a smooth review experience - we don't distract attention.

Thanks, @nodece and @tisonkun. Have reverted that change.

tisonkun

I don't think it's an isolated version issue. Instead, you should take a look of how Pulsar uses okio and make an upgrade from the direct deps and then propagate to transitive deps.

tisonkun · 2023-03-26T10:24:20Z

cc @nicoloboschi since you're the last author for editing these versions in #13065.

Code review changes

abhilashmandaliya · 2023-03-27T08:06:03Z

@nodece @tisonkun @lhotari @nicoloboschi
I have made the changes as per the feedback and also updated the PR description.
GitHub UI did some weird things hence there are several reviews and remove status lines.

nicoloboschi · 2023-03-27T09:49:10Z

-    <okio.version>2.8.0</okio.version>
-    <!-- override kotlin-stdlib used by okio in order to address CVE-2020-29582 -->
-    <kotlin-stdlib.version>1.4.32</kotlin-stdlib.version>
+    <okio.version>3.0.0</okio.version>


we should move to 3.3.0, based on the changelog it's more stable and it upgrades kotlin to 1.8 as well

@nicoloboschi Latest stable version of okhttp3 4.10.0 uses 3.0.0 version of okio. Is it okay to use 3.3.0 instead?

…t version of okio brings non-vulnerable kotlin dependency

abhilashmandaliya · 2023-03-28T08:37:37Z

@nicoloboschi PTAL at the updated library version change.

nicoloboschi

LGTM

tisonkun · 2023-03-28T09:27:49Z

@abhilashmandaliya license check failed:

com.squareup.okio-okio-jvm-3.0.0.jar unaccounted for in LICENSE
org.jetbrains.kotlin-kotlin-stdlib-1.6.20.jar unaccounted for in LICENSE
org.jetbrains.kotlin-kotlin-stdlib-common-1.5.31.jar unaccounted for in LICENSE
org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.6.10.jar unaccounted for in LICENSE
org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.6.10.jar unaccounted for in LICENSE
com.squareup.okio-okio-3.3.0.jar mentioned in LICENSE, but not bundled
org.jetbrains.kotlin-kotlin-stdlib-1.8.0.jar mentioned in LICENSE, but not bundled
org.jetbrains.kotlin-kotlin-stdlib-common-1.8.0.jar mentioned in LICENSE, but not bundled
org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.8.0.jar mentioned in LICENSE, but not bundled
org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.8.0.jar mentioned in LICENSE, but not bundled

The first lines can be redundant that you should remove. The final lines about kotlin-stdlib, I'm afraid, indicate an issue that we are missing to bundle them?

abhilashmandaliya · 2023-03-28T09:43:10Z

@abhilashmandaliya license check failed:

com.squareup.okio-okio-jvm-3.0.0.jar unaccounted for in LICENSE
org.jetbrains.kotlin-kotlin-stdlib-1.6.20.jar unaccounted for in LICENSE
org.jetbrains.kotlin-kotlin-stdlib-common-1.5.31.jar unaccounted for in LICENSE
org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.6.10.jar unaccounted for in LICENSE
org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.6.10.jar unaccounted for in LICENSE
com.squareup.okio-okio-3.3.0.jar mentioned in LICENSE, but not bundled
org.jetbrains.kotlin-kotlin-stdlib-1.8.0.jar mentioned in LICENSE, but not bundled
org.jetbrains.kotlin-kotlin-stdlib-common-1.8.0.jar mentioned in LICENSE, but not bundled
org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.8.0.jar mentioned in LICENSE, but not bundled
org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.8.0.jar mentioned in LICENSE, but not bundled

The first lines can be redundant that you should remove. The final lines about kotlin-stdlib, I'm afraid, indicate an issue that we are missing to bundle them?

@tisonkun With the latest okio library, we don't need to use a different version of Kotlin hence I removed it from the pom.xml. I forgot to remove it from the LICENCE file. I have fixed that. Not sure why okio error is there.

tisonkun · 2023-03-28T09:49:50Z

@abhilashmandaliya I get the error report wrong.

com.squareup.okio-okio-jvm-3.0.0.jar unaccounted for in LICENSE
org.jetbrains.kotlin-kotlin-stdlib-1.6.20.jar unaccounted for in LICENSE
org.jetbrains.kotlin-kotlin-stdlib-common-1.5.31.jar unaccounted for in LICENSE
org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.6.10.jar unaccounted for in LICENSE
org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.6.10.jar unaccounted for in LICENSE

These lines mean that you're now (transitive) depending on these libs but they are not listed (mentioned) in the LICENSE file.

You may double check if the version is correct (kotlin 1.5 and 1.6 while we plan to use 1.8? okio-jvm 3.0.0?). The LICENSE updating job I can do you a favor. But we should confirm the final status logically first.

github-actions · 2023-05-18T01:54:28Z

The pr had no activity for 30 days, mark with Stale label.

Signed-off-by: tison <wander4096@gmail.com>

tisonkun

I push a commit to suppress:

org.jetbrains.kotlin-kotlin-stdlib-1.8.20.jar mentioned in LICENSE, but not bundled
org.jetbrains.kotlin-kotlin-stdlib-common-1.8.20.jar mentioned in LICENSE, but not bundled
org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.8.20.jar mentioned in LICENSE, but not bundled
org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.8.20.jar mentioned in LICENSE, but not bundled
org.jetbrains-annotations-13.0.jar mentioned in LICENSE, but not bundled

But then the changeset seems a bit strange. Comment inline.

Also, you can ping the reviewers when you make progress - pushing new commits doesn't send notification.

tisonkun · 2023-05-18T02:25:18Z

        <groupId>com.squareup.okhttp3</groupId>
        <artifactId>logging-interceptor</artifactId>
        <version>${okhttp3.version}</version>
+        <exclusions>


@abhilashmandaliya if we exclude kotlin dep here, is it possible we are missing bundling several jars so that the function shade jar won't work?

abhilashmandaliya · 2023-05-25T06:46:41Z

Closing the PR as Pulsar 3.0.0 doesn't bring the older Kotlin version. Thanks a lot, @tisonkun for your help.

github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Mar 25, 2023

abhilashmandaliya marked this pull request as draft March 25, 2023 06:32

abhilashmandaliya marked this pull request as ready for review March 25, 2023 10:31

nodece requested review from lhotari and tisonkun March 25, 2023 17:03

nodece assigned abhilashmandaliya Mar 25, 2023

nodece reviewed Mar 25, 2023

View reviewed changes

tisonkun requested changes Mar 26, 2023

View reviewed changes

tisonkun requested a review from nicoloboschi March 26, 2023 10:24

abhilashmandaliya force-pushed the kotlin-upgrade branch from 56625c8 to 8144bf9 Compare March 27, 2023 05:17

abhilashmandaliya added 3 commits March 27, 2023 12:58

Upgraded kotlin library to 1.8.10

1a60b01

Upgraded okhttp3 and okio dependencies

517dd51

Code review changes

Code review change

104e44e

abhilashmandaliya force-pushed the kotlin-upgrade branch from cb3031d to 104e44e Compare March 27, 2023 07:29

abhilashmandaliya requested review from nodece and tisonkun and removed request for lhotari, nicoloboschi, nodece and tisonkun March 27, 2023 08:03

abhilashmandaliya changed the title ~~[improve][misc] Updated org.jetbrains.kotlin:kotlin-stdlib version~~ [improve][io] Fix Kotlin version issue for functions built in Kotlin Mar 27, 2023

nicoloboschi requested changes Mar 27, 2023

View reviewed changes

abhilashmandaliya added 3 commits March 27, 2023 17:50

Using the latest version of okio. Removed kotlin library as the lates…

ec93b1f

…t version of okio brings non-vulnerable kotlin dependency

Corrected kotlin version in license file

ce23027

Corrected okio version in the license file

1015db1

abhilashmandaliya requested review from nicoloboschi and removed request for tisonkun March 28, 2023 08:37

nicoloboschi approved these changes Mar 28, 2023

View reviewed changes

tisonkun approved these changes Mar 28, 2023

View reviewed changes

Removed un-packaged Kotlin libraries from LICENSE.bin.txt

30cd5da

Excluded transitive dependency due to older versions

6306caa

abhilashmandaliya force-pushed the kotlin-upgrade branch from 690c8e3 to 6306caa Compare April 13, 2023 09:06

abhilashmandaliya added 5 commits April 13, 2023 15:25

Trigger Build

5920332

Changed nullability annotation

63da2b0

Trigger Build

9802061

Removed unused libraries from the licence file

c46df39

Merge branch 'master' into kotlin-upgrade

8825434

github-actions Bot added the Stale label May 18, 2023

tisonkun added 2 commits May 18, 2023 09:56

Merge branch 'master' into kotlin-upgrade

18dc44c

fix binary license check

588c0c0

Signed-off-by: tison <wander4096@gmail.com>

tisonkun requested changes May 18, 2023

View reviewed changes

abhilashmandaliya closed this May 25, 2023

Uh oh!

Conversation

abhilashmandaliya commented Mar 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Matching PR in forked repository

Uh oh!

nodece Mar 25, 2023

Choose a reason for hiding this comment

Uh oh!

tisonkun Mar 26, 2023

Choose a reason for hiding this comment

Uh oh!

abhilashmandaliya Mar 27, 2023

Choose a reason for hiding this comment

Uh oh!

tisonkun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tisonkun commented Mar 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

abhilashmandaliya commented Mar 27, 2023

Uh oh!

nicoloboschi Mar 27, 2023

Choose a reason for hiding this comment

Uh oh!

abhilashmandaliya Mar 27, 2023

Choose a reason for hiding this comment

Uh oh!

abhilashmandaliya commented Mar 28, 2023

Uh oh!

nicoloboschi left a comment

Choose a reason for hiding this comment

Uh oh!

tisonkun commented Mar 28, 2023

Uh oh!

abhilashmandaliya commented Mar 28, 2023

Uh oh!

tisonkun commented Mar 28, 2023

Uh oh!

github-actions Bot commented May 18, 2023

Uh oh!

tisonkun left a comment

Choose a reason for hiding this comment

Uh oh!

tisonkun May 18, 2023

Choose a reason for hiding this comment

Uh oh!

abhilashmandaliya commented May 25, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

abhilashmandaliya commented Mar 25, 2023 •

edited

Loading

tisonkun left a comment •

edited

Loading

tisonkun commented Mar 26, 2023 •

edited

Loading