CI: Rework publish pipeline for "openapi-generator-gradle-plugin"

[jmini]

The automated deploy of the openapi-generator-gradle-plugin is not working.

With 3.0.1 and 3.0.2 there was a TravisCI timeout in the job.
With 3.0.3 the TravisCI Job is OK.

Log: https://api.travis-ci.org/v3/job/397359374/log.txt

Maven deploy:

Source:

openapi-generator/.travis.yml

Lines 108 to 109 in 9b909df

	mvn clean deploy -DskipTests=true -B -U -P release --settings CI/settings.xml;
	echo "Finished mvn clean deploy for $TRAVIS_BRANCH";

Log Extract: end of the maven deploy step:

Finished mvn clean deploy for master

Gradle Uplaoad:

Source:

openapi-generator/.travis.yml

Lines 112 to 113 in 9b909df

	./gradlew -Psigning.keyId="$SIGNING_KEY" -Psigning.password="$SIGNING_PASSPHRASE" -Psigning.secretKeyRingFile="${TRAVIS_BUILD_DIR}/sec.gpg" -PossrhUsername="${SONATYPE_USERNAME}" -PossrhPassword="${SONATYPE_PASSWORD}" uploadArchives --no-daemon;
	echo "Finished ./gradlew uploadArchives";

Log Extract for the uploadArchives task:

~/build/OpenAPITools/openapi-generator ~/build/OpenAPITools/openapi-generator

Welcome to Gradle 4.7!

Here are the highlights of this release:
 - Incremental annotation processing
 - JDK 10 support
 - Grouped non-interactive console logs
 - Failed tests are re-run first for quicker feedback

For more details see https://docs.gradle.org/4.7/release-notes.html

> Task :compileKotlin UP-TO-DATE
> Task :compileJava NO-SOURCE
> Task :pluginDescriptors UP-TO-DATE
> Task :processResources UP-TO-DATE
> Task :classes UP-TO-DATE
> Task :jar UP-TO-DATE
> Task :javadoc NO-SOURCE
> Task :javadocJar UP-TO-DATE
> Task :sourcesJar UP-TO-DATE
> Task :signArchives
> Task :uploadArchives

BUILD SUCCESSFUL in 3m 42s
8 actionable tasks: 2 executed, 6 up-to-date
Finished ./gradlew uploadArchives

The artefacts were uploaded to Nexus. But a look in the Staging repository show multiple staging repo:

https://oss.sonatype.org/#stagingRepositories

Content of the different repos:

• openapitools-1031 contains only:
  • /org/openapitools/openapi-generator-gradle-plugin/3.0.3/openapi-generator-gradle-plugin-3.0.3.pom.asc.md5
• openapitools-1030 contains:
  • /org/openapitools/openapi-generator-gradle-plugin/3.0.3/openapi-generator-gradle-plugin-3.0.3-javadoc.jar.sha1
  • /org/openapitools/openapi-generator-gradle-plugin/3.0.3/openapi-generator-gradle-plugin-3.0.3-sources.jar
  • /org/openapitools/openapi-generator-gradle-plugin/3.0.3/openapi-generator-gradle-plugin-3.0.3.jar.asc.md5
• … and so on
• openapitools-1024

---

This is a known issue:

https://issues.sonatype.org/browse/OSSRH-37302?attachmentViewMode=list

---

I see multiple solutions:

1/ Use the nexus-staging-maven-plugin:

My understanding (see also #366) is that if we manage to install the artifacts (jars, pom, signature files) to <root repo>/target/nexus-staging/deferred/ during the maven build (as a regular step of the openapi-generator-gradle-plugin-mvn-wrapper project) then it will be uploaded in one step at the end of the build.

This is more or less what is recommended in OSSRH-37302

2/ Switch CI Server. According to @jimschubert there is no problem with GitLab-CI

3/ Switch Staging server.

• Deploy to Bintray (instead of Sonatype Nexus)
• From Bintray to JCenter and Maven Central (one click)

Bintray is free for Open Source project.

I think that a lot of projects using gradle are using Bintray. I think that AsciidoctorJ is doing it like this.

[jmini]

Some notes about publishing the gradle plugin during releases:

If the build is not failing (green build) then you might be able to just redo the publication. (This was the case for 3.0.3 and 3.1.0

Go to: https://oss.sonatype.org/#stagingRepositories
And search for the multiple orgopenapitools-0000 repositories (see screenshot in the previous message)

Collect the items:

• openapi-generator-gradle-plugin-{version}-javadoc.jar
• openapi-generator-gradle-plugin-{version}-javadoc.jar.asc
• openapi-generator-gradle-plugin-{version}-sources.jar
• openapi-generator-gradle-plugin-{version}-sources.jar.asc
• openapi-generator-gradle-plugin-{version}.jar
• openapi-generator-gradle-plugin-{version}.jar.asc
• openapi-generator-gradle-plugin-{version}.pom
• openapi-generator-gradle-plugin-{version}.pom.asc

There is more file *.md5, *.xml…

Put them in a folder /org/openapitools/openapi-generator-gradle-plugin/{version}

Edit the gradle.build file in the gradle sample folder:

    repositories {
        mavenCentral()
        maven {
            url "file://<path to folder>"
        }
    }

Then run:

gradle -PopenApiGeneratorVersion={version} openApiMeta

Example for the 3.1.0 release:

gradle -PopenApiGeneratorVersion=3.1.0 openApiMeta

In the <folder> repository run:

jar -cvf bundle.jar openapi-generator-gradle-plugin-*

In Nexus Select “Staging Upload” https://oss.sonatype.org/#staging-upload

Select:

• Upload Mode: “Artifact Bundle”
• Click on “Select Bundle to Upload...” and select the generated bundle file from the previous step
• Click on “Upload Bundle”

Wait until the upload is complete.
=> If it seems to never end, redo the upload. Nexus seems to have some problems sometimes.

When it is successful you see something like this:

Test it again locally, edit the gradle.build file in the gradle sample folder:

    repositories {
        mavenCentral()
        maven {
            url "https://oss.sonatype.org/content/repositories/orgopenapitools-1034"
        }
    }

Then run again:

gradle -PopenApiGeneratorVersion=3.1.0 openApiMeta

If the gradle build is successful you can go to
https://oss.sonatype.org/#stagingRepositories again and press “Release” on your staged repository (orgopenapitools-1034 in the previous example).

[jimschubert]

@jmini I'd brought this up on Gitter, but you may not have seen it.

The Travis timeout is because Travis defaults to 10 min of timeout. We can increase this ad hoc per command using travis_wait (see Travis CI: Common Build Problems). The decision before the last release was to see if the problem continued before trying out travis_wait, since it's one of those things we have to try at the time of publish.

The issue of splitting repositories is because Travis CI is apparently proxying requests and reporting multiple IP addresses. We can reach out to the Travis team to understand if there's a workaround.

[jimschubert]

I sent an email to Travis CI support and CC'd the OpenAPI Tools team. I'll report back here when I have a response.

[jmini]

I sent an email to Travis CI support and CC'd the OpenAPI Tools team. I'll report back here when I have a response.

We got an answer, they stick to not changing anything on there side as indicated in https://issues.sonatype.org/browse/OSSRH-37302

We can still work on one of the solution exposed in my first message here.

I also spoke with @wing328 about it. For the moment it is a low prio. Republishing the gradle plugin manually is not such a big deal.

[jmini]

1/ Use the nexus-staging-maven-plugin

PR for this approach: #902

[jimschubert]

Revisiting this as requested recently.

The issue we're seeing is caused by:

• Travis using different IP addresses for one or more subsequent requests to Sonatype (see Maven seemingly broken recently due to varying IP addresses travis-ci/travis-ci#9555 which was closed without a fix)
• Sonatype automatically creating a new staging repository for a new unique IP address, then reusing that staging repository for that IP address; I'm guessing they recycle this association after some amount of time
• Gradle's Upload task not directly supporting Nexus staging repository id configuration

What this means:

When file upload requests are passed through Travis CI's load balancer, they have a different IP address and Sonatype ends up creating a new staging repository for each new IP. So as each jar is uploaded, it may or may not end up in the same staging repository as the previously uploaded jar.

There's a feature request open on gradle (gradle/gradle#5711) to support adding staging repository id for Nexus repos (Sonatype).

Apparently, some users have workarounds (taken from the gradle issue above):

https://github.com/h2oai/sparkling-water/blob/master/gradle/publish/createStagingId.sh
https://github.com/h2oai/sparkling-water/blob/73d2e2bd11ce35c6379f6961188fb2af1a7bc04b/build.gradle#L228

I have seen other attempts at resolution with disabling sudo on Travis configuration. That doesn't seem to work.

As Jérémie suggested, we could publish to Bintray and then sync to Maven Central. Grails does this, and they also use Travis (see grails-core).

Discussion

The workarounds aren't really ideal because they end up coupling our deployments to the Sonatype API, and those types of workarounds are arguably more difficult to reason about than just publishing locally until there's a better solution.

I would think that Sonatype could easily support tracking a unique series of request by a trace id HTTP header, and that the uploader in Gradle could then support that trace header. But, I don't know if Sonatype's strategy for creating new staging repositories per IP address regardless of the user/pass/repo/artifact version defined in the publish has some other meaning that we're unaware of. That is, if I'm publishing openapi-generator-gradle-plugin-3.3.1 artifacts and my requests have authenticated successfully, I don't see reason that these multiple artifacts should be shuffled to separate staging repositories.

I'll be moving in a couple weeks, so hopefully my free time to investigate further will pick up afterward.

I would suggest publishing manually for now. We could discuss potential solutions in one of the linked issues as well.

[jimschubert]

I've found what appears to be a solid approach to publishing the Gradle artifact as part of the travis ci build.

@jmini https://github.com/osmlab/atlas/wiki/Gradle,-Travis-CI-and-Maven-Central looks like an approach that I'd be more comfortable evaluating than the link to the sparkling-water repo above. Would you have any concerns with evaluating this approach? If not, I'll work on integrating it over the next week or two.

[jmini]

If I understood the article correctly, they perform the "upload to nexus" part not with normal gradle publish task but with a custom uploadAndRelease.sh script.

Why not... but it sounds complicated to me.

---

Following the links and the issues, I have the feeling that a gradle solution exists:
https://github.com/Codearte/gradle-nexus-staging-plugin#2-why-my-release-build-on-travis-suddenly-started-to-fail-with-wrong-number-of-received-repositories

I think that the maven plugin we use in our gradle build:

openapi-generator/modules/openapi-generator-gradle-plugin/build.gradle

Line 33 in faf1f5d

apply plugin: 'maven'

Is no longer recommended with newer version of gradle. the publish from the gradle plugin maven-publish seems to be preferred.

[jimschubert]

@jmini the plugin you've linked only closes and promotes the repository. Our issue is one phase earlier, where the upload splits into multiple repositories.

The scripted solution just uses Sonatype's API to upload the artifacts. It's not too different from what the plugin is doing, except that it supports creating the repository where the plugin does not.

Three are two maven plugins, we're using the only one that works, unfortunately. The other doesn't support code signing or javadocs. I forget which one isn't supported, but it is one of the required artifacts for Sonatype's deployment.

I'll try to find the time to POC this. I think 4.0.0 isn't the right time to try out a new publish workflow.

[jmini]

I have observed the problem that multiple distant repository are created on the maven-central nexus, when I publish from my company network (no idea how the infrastructure).
So I took the time to look at these gradle plugins again.

You are right io.codearte.nexus-staging (github) is about closing a repo.

But there is an other one: de.marcphilipp.nexus-publish (github). It adds a task initializeNexusStagingRepository that opens a new staging repository with a fix ID on nexus, and then publishToNexus is using this stable repository when pushing artifacts to nexus.

Both plugins io.codearte.nexus-staging and de.marcphilipp.nexus-publish can be combined together. nexus-publish can reuse the config from nexus-staging if this plugin is present.

---

de.marcphilipp.nexus-publish work only with maven-publish (which is newer and recommended anyway).

At the beginning the signing plugin was only working with maven but now it works also with maven-publish.
I have a setup that publish to maven central with maven-publish with source-jar, javadoc-jar and signing.