Skip to content

Add step to load pods in cache before building and testing #19551

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mokagio wants to merge 1 commit into from
Closed

Add step to load pods in cache before building and testing #19551

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .buildkite/commands/load-pods.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
#!/bin/bash -eu

echo "--- :rubygems: Set up Gems"
install_gems

echo "--- :cocoapods: Set up Pods and cache them if needed"
install_cocoapods
12 changes: 12 additions & 0 deletions .buildkite/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,23 @@ common_params:
# This is the default pipeline – it will build and test the app
steps:

# Make sure there is a cached version of the pods avaialble before testing
# the app or deploying installable builds to avoid parallel steps wasting
# time building the same `Podfile.lock` from scratch.
- label: ':cocoapods: Pre-load Pods'
key: pods
command: .buildkite/commands/load-pods.sh
env: *common_env
plugins: *common_plugins
Comment on lines +16 to +23

@AliSoftware AliSoftware Nov 2, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this really work? I thought that each step in a Buildkite pipeline were independent from one another in terms of the agents they might be using, which means that this step downloading / installing pods might run on one agent, while the following steps that depends on it might run on a different one:

However you add steps to your pipeline, keep in mind that steps may run on different agents. It is good practice to install your dependencies in the same step that you run them.
— Source: https://buildkite.com/docs/pipelines/defining-steps#adding-steps

So if we want this to work as expected, we'd also have to e.g. upload a .tar of the Pods/ folder that got installed by this step in artefacts, then download and uncompress that Pods.tar back in the subsequent steps (similar to how we do it for build-for-testing uploading the Build Products here then unit-tests.sh downloading them back here)

@mokagio mokagio Nov 2, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if we want this to work as expected, we'd also have to e.g...

That's what install_cocoapods does at the bash-cache plugin level.

@mokagio mokagio Nov 2, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or, I should say "install_cocoapods should be doing". I haven't gotten to the point of testing a build that would trigger a new cache generation. But I'm pretty sure it works as install_cocoapods is tried and tested.

@AliSoftware AliSoftware Nov 2, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I didn't realize that install_cocoapods was already I closing the cache dance! Your initial approach makes more sense now 🙂

So I guess the only thing left to solve is how to tell install_cocoapods told only install them (and thus ultimately rebuild the cache) if there was a cache miss… but don't bother installing them during this check step in case of a cache hit.


#################
# Create Installable Builds for WP and JP
#################
- group: "🛠 Installable Builds"
steps:
- label: "🛠 WordPress Installable Build"
command: ".buildkite/commands/installable-build-wordpress.sh"
depends_on: "pods"
env: *common_env
plugins: *common_plugins
if: "build.pull_request.id != null || build.pull_request.draft"
Expand All @@ -29,6 +39,7 @@ steps:

- label: "🛠 Jetpack Installable Build"
command: ".buildkite/commands/installable-build-jetpack.sh"
depends_on: "pods"
env: *common_env
plugins: *common_plugins
if: "build.pull_request.id != null || build.pull_request.draft"
Expand All @@ -42,6 +53,7 @@ steps:
- label: "🛠 Build for Testing"
key: "build"
command: ".buildkite/commands/build-for-testing.sh"
depends_on: "pods"
env: *common_env
plugins: *common_plugins
notify:
Expand Down
12 changes: 12 additions & 0 deletions .buildkite/release-builds.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,22 +14,34 @@ common_params:

steps:

# Make sure there is a cached version of the pods avaialble before testing
# the app or deploying installable builds to avoid parallel steps wasting
# time building the same `Podfile.lock` from scratch.
- label: ':cocoapods: Pre-load Pods'
key: pods
command: .buildkite/commands/load-pods.sh
env: *common_env
plugins: *common_plugins

- label: ":wordpress: :testflight: WordPress Release Build (App Store Connect)"
command: ".buildkite/commands/release-build-wordpress.sh $BETA_RELEASE"
depends_on: "pods"
env: *common_env
plugins: *common_plugins
notify:
- slack: "#build-and-ship"

- label: ":wordpress: :appcenter: WordPress Release Build (App Center)"
command: ".buildkite/commands/release-build-wordpress-internal.sh"
depends_on: "pods"
env: *common_env
plugins: *common_plugins
notify:
- slack: "#build-and-ship"

- label: ":jetpack: :testflight: Jetpack Release Build (App Store Connect)"
command: ".buildkite/commands/release-build-jetpack.sh"
depends_on: "pods"
env: *common_env
plugins: *common_plugins
notify:
Expand Down