From 166b0366f07af9eef9f5a4884345795ba6360025 Mon Sep 17 00:00:00 2001 From: Maximilian Michels Date: Mon, 3 Aug 2020 18:37:24 +0200 Subject: [PATCH 1/3] [BEAM-10630] Include data from load tests in the release process In the past, we have seen performance regressions in releases. We should make sure that the release guide includes checking available performance measurements. --- .../content/en/contribute/release-guide.md | 37 ++++++++++++++----- 1 file changed, 27 insertions(+), 10 deletions(-) diff --git a/website/www/site/content/en/contribute/release-guide.md b/website/www/site/content/en/contribute/release-guide.md index 87eb7327d186..a117c6492169 100644 --- a/website/www/site/content/en/contribute/release-guide.md +++ b/website/www/site/content/en/contribute/release-guide.md @@ -37,6 +37,9 @@ The release process consists of several steps: 1. Decide to release 1. Prepare for the release +1. Investigate performance regressions +1. Create a release branch +1. Verify release branch 1. Build a release candidate 1. Vote on the release candidate 1. During vote process, run validation tests @@ -45,7 +48,7 @@ The release process consists of several steps: 1. Promote the release -### Decide to release +## 1. Decide to release Deciding to release and selecting a Release Manager is the first step of the release process. This is a consensus-based decision of the entire community. @@ -60,7 +63,7 @@ In general, the community prefers to have a rotating set of 3-5 Release Managers ********** -## 1. Prepare for the release +## 2. Prepare for the release Before your first release, you should perform one-time configuration steps. This will set up your security keys for signing the release and access to various release repositories. @@ -244,7 +247,21 @@ __Attention__: Only PMC has permission to perform this. If you are not a PMC, pl ********** -## 2. Create a release branch in apache/beam repository +## 3. Investigate performance regressions + +Check the Beam load tests for possible performance regressions. Measurements are available on [metrics.beam.apache.org](http://metrics.beam.apache.org). + +All Runners which publish data should be checked for the following, in both *batch* and *streaming* mode: + +- [ParDo](http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests) and [GBK](http://metrics.beam.apache.org/d/UYZ-oJ3Zk/gbk-load-test): Runtime, latency, checkpoint duration +- [Nexmark](http://metrics.beam.apache.org/d/ahuaA_zGz/nexmark): Query runtime for all queries +- [IO](http://metrics.beam.apache.org/d/bnlHKP3Wz/java-io-it-tests-dataflow): Runtime + +If regressions are found, the release branch can still be created, but the regressions should be investigated and fixed as part of the release process. +JIRA issues should be created for each regression with the 'Fix Version' set to the to-be-released version. +Next, the mailing list should be informed to allow fixing the regressions in the course of the release. + +## 4. Create a release branch in apache/beam repository Attention: Only committer has permission to create release branch in apache/beam. @@ -367,7 +384,7 @@ There are 2 ways to trigger a nightly build, either using automation script(reco ********** -## 3. Verify release branch +## 5. Verify release branch After the release branch is cut you need to make sure it builds and has no significant issues that would block the creation of the release candidate. There are 2 ways to perform this verification, either running automation script(recommended), or running all commands manually. @@ -481,7 +498,7 @@ The verify_release_build.sh script may include failing or flaky tests. For each ********** -## 4. Triage release-blocking issues in JIRA +## 6. Triage release-blocking issues in JIRA There could be outstanding release-blocking issues, which should be triaged before proceeding to build a release candidate. We track them by assigning a specific `Fix version` field even before the issue resolved. @@ -528,7 +545,7 @@ _Tip_: Another tool in your toolbox is the known issues section of the release b ********** -## 5. Build a release candidate +## 7. Build a release candidate ### Checklist before proceeding @@ -599,7 +616,7 @@ For this step, we recommend you using automation script to create a RC, but you ********** -## 6. Prepare documents +## 8. Prepare documents ### Update and Verify Javadoc @@ -748,7 +765,7 @@ docker pull apache/beam_python3.5_sdk:2.16.0_rc1 ********** -## 7. Vote and validate release candidate +## 9. Vote and validate release candidate Once you have built and individually reviewed the release candidate, please share it for the community-wide review. Please review foundation-wide [voting guidelines](http://www.apache.org/foundation/voting.html) for more information. @@ -1074,7 +1091,7 @@ Once all issues have been resolved, you should go back and build a new release c ********** -## 8. Finalize the release +## 10. Finalize the release Once the release candidate has been reviewed and approved by the community, the release should be finalized. This involves the final deployment of the release candidate to the release repositories, merging of the website changes, etc. @@ -1163,7 +1180,7 @@ Use reporter.apache.org to seed the information about the release into future pr ********** -## 9. Promote the release +## 11. Promote the release Once the release has been finalized, the last step of the process is to promote the release within the project and beyond. From 36e06b830c4426e4b1e4d8ea705582eb3f01748e Mon Sep 17 00:00:00 2001 From: Maximilian Michels Date: Wed, 5 Aug 2020 21:52:30 +0200 Subject: [PATCH 2/3] Address PR comments --- website/www/site/content/en/contribute/release-guide.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/website/www/site/content/en/contribute/release-guide.md b/website/www/site/content/en/contribute/release-guide.md index a117c6492169..fa1e1a0b9484 100644 --- a/website/www/site/content/en/contribute/release-guide.md +++ b/website/www/site/content/en/contribute/release-guide.md @@ -258,8 +258,10 @@ All Runners which publish data should be checked for the following, in both *bat - [IO](http://metrics.beam.apache.org/d/bnlHKP3Wz/java-io-it-tests-dataflow): Runtime If regressions are found, the release branch can still be created, but the regressions should be investigated and fixed as part of the release process. -JIRA issues should be created for each regression with the 'Fix Version' set to the to-be-released version. -Next, the mailing list should be informed to allow fixing the regressions in the course of the release. +The role of the release manager is to file JIRA issues for each regression with the 'Fix Version' set to the to-be-released version. The release manager +oversees these just like any other JIRA issue marked with the 'Fix Version' of the release. + +The mailing list should be informed to allow fixing the regressions in the course of the release. ## 4. Create a release branch in apache/beam repository From d285c5855c1dd02180762c8e3848822b24eba572 Mon Sep 17 00:00:00 2001 From: Maximilian Michels Date: Wed, 5 Aug 2020 21:56:56 +0200 Subject: [PATCH 3/3] Add note about informing the mailing list after creating the release branch --- website/www/site/content/en/contribute/release-guide.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/website/www/site/content/en/contribute/release-guide.md b/website/www/site/content/en/contribute/release-guide.md index fa1e1a0b9484..8599b414c4a4 100644 --- a/website/www/site/content/en/contribute/release-guide.md +++ b/website/www/site/content/en/contribute/release-guide.md @@ -496,6 +496,11 @@ The verify_release_build.sh script may include failing or flaky tests. For each * Description: Description of failure +#### Inform the mailing list + +The dev@beam.apache.org mailing list should be informed about the release branch being cut. Alongside with this note, +a list of pending issues and to-be-trigated issues should be included. Afterwards, this list can be refined and updated +by the release manager and the Beam community. **********