Skip to content

CI: dynamic jobs#124175

Merged
bors merged 8 commits into
rust-lang:masterfrom
Kobzol:ci-dynamic-job
Apr 24, 2024
Merged

CI: dynamic jobs#124175
bors merged 8 commits into
rust-lang:masterfrom
Kobzol:ci-dynamic-job

Conversation

@Kobzol

@Kobzol Kobzol commented Apr 19, 2024 •
edited
Loading

Copy link
Copy Markdown
Member

This PR modifies our CI workflows to be dynamic. This means that when a GitHub event is generated, we will run a Python script (calculate-job-matrix.py), which decides which CI jobs should be generated. These jobs are defined in src/ci/github-actions/jobs.yml).

This should provide a few benefits:

  • Once the migration to dynamic jobs is complete, we shouldn't need expand-yaml-anchors anymore.
  • The job table on PRs (and also the left job column on auto/try builds) should be much cleaner and contain only the jobs that are actually relevant/executed.
  • It should be much easier to support dynamic try builds, i.e. to run an arbitrary CI job on a try build.

See this Zulip discussion for more context.

r? @ghost

@rustbot rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Apr 19, 2024
@Kobzol Kobzol force-pushed the ci-dynamic-job branch 3 times, most recently from b325a04 to 0e9eed1 Compare April 19, 2024 19:56
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@Kobzol Kobzol force-pushed the ci-dynamic-job branch 3 times, most recently from 55550c7 to 476d4ae Compare April 19, 2024 20:45
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@Kobzol Kobzol force-pushed the ci-dynamic-job branch 4 times, most recently from 65141ab to 8b945c3 Compare April 19, 2024 21:19
@Kobzol

Kobzol commented Apr 19, 2024

Copy link
Copy Markdown
Member Author

@bors try

bors added a commit to rust-lang-ci/rust that referenced this pull request Apr 19, 2024
@bors
Auto merge of rust-lang#124175 - Kobzol:ci-dynamic-job, r=<try>
5b35171 
CI: dynamic jobs

Opening as a draft to experiment with [dynamic CI jobs](https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/job.20matrix.20re-ordered.20PR.20list).

r? `@ghost`
@bors

bors commented Apr 19, 2024

Copy link
Copy Markdown
Collaborator

⌛ Trying commit 8b945c3 with merge 5b35171...

@rust-log-analyzer

This comment has been minimized.

Sign in to view
@Kobzol Kobzol force-pushed the ci-dynamic-job branch 2 times, most recently from 97ca4d3 to 8f21998 Compare April 19, 2024 21:33
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@bors

bors commented Apr 20, 2024

Copy link
Copy Markdown
Collaborator

☔ The latest upstream changes (presumably #124177) made this pull request unmergeable. Please resolve the merge conflicts.

@Kobzol Kobzol force-pushed the ci-dynamic-job branch 2 times, most recently from 0efaec5 to 545c12f Compare April 20, 2024 07:18
@Kobzol Kobzol force-pushed the ci-dynamic-job branch 2 times, most recently from dc3c544 to 4d3e89c Compare April 20, 2024 07:45
emilyalbini
emilyalbini reviewed Apr 22, 2024
View reviewed changes

@emilyalbini emilyalbini left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few minor nits, but this otherwise looks great!

Comment thread src/ci/github-actions/jobs.yml Outdated
Comment thread src/ci/github-actions/calculate-job-matrix.py Outdated
Comment thread src/ci/github-actions/jobs.yml Outdated
Comment thread src/ci/github-actions/ci.yml
@Kobzol

Kobzol commented Apr 23, 2024

Copy link
Copy Markdown
Member Author

Thanks for the review, tried to fix all remarks in separate commits.

@emilyalbini

emilyalbini commented Apr 23, 2024

Copy link
Copy Markdown
Member

@bors r+

@bors

bors commented Apr 23, 2024

Copy link
Copy Markdown
Collaborator

📌 Commit 2632c61 has been approved by pietroalbini

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 23, 2024
@Kobzol

Kobzol commented Apr 23, 2024

Copy link
Copy Markdown
Member Author

@bors rollup=never

(Who knows what this might do in the auto build).

RalfJung
RalfJung reviewed Apr 23, 2024
View reviewed changes
Comment thread src/ci/github-actions/jobs.yml
<<: *job-linux-16c

# Jobs that run when you perform a try build (@bors try)
# These jobs automatically inherit envs.production, to avoid repeating

@RalfJung RalfJung Apr 23, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't they inherit envs.try? (And similar for auto below)

@Kobzol Kobzol Apr 23, 2024 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point, I forgot to update the comment. I'll add it to a follow up PR.

@bors

bors commented Apr 24, 2024

Copy link
Copy Markdown
Collaborator

⌛ Testing commit 2632c61 with merge e7da0fa...

@bors

bors commented Apr 24, 2024

Copy link
Copy Markdown
Collaborator

☀️ Test successful - checks-actions
Approved by: pietroalbini
Pushing e7da0fa to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 24, 2024
@bors bors merged commit e7da0fa into rust-lang:master Apr 24, 2024
@rustbot rustbot added this to the 1.79.0 milestone Apr 24, 2024
@bors bors mentioned this pull request Apr 24, 2024
Closed
@Kobzol Kobzol deleted the ci-dynamic-job branch April 24, 2024 09:07
@Kobzol

Kobzol commented Apr 24, 2024

Copy link
Copy Markdown
Member Author

Wow, it went through. Great.

This was referenced Apr 24, 2024
Merged
Closed
Merged
Closed
@rust-timer

rust-timer commented Apr 24, 2024

Copy link
Copy Markdown
Collaborator

Finished benchmarking commit (e7da0fa): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 673.216s -> 673.226s (0.00%)
Artifact size: 315.45 MiB -> 315.42 MiB (-0.01%)

@Zalathar Zalathar mentioned this pull request May 3, 2024
Merged
@Kobzol Kobzol mentioned this pull request May 6, 2024
Merged
@Kobzol Kobzol mentioned this pull request May 24, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@RalfJung RalfJung RalfJung left review comments

@emilyalbini emilyalbini emilyalbini left review comments

Assignees

@emilyalbini emilyalbini

Labels

A-testsuite Area: The testsuite used to check the correctness of rustc merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.

Projects

None yet

Milestone

1.79.0

Development

Successfully merging this pull request may close these issues.

7 participants

@Kobzol @rust-log-analyzer @bors @emilyalbini @rust-timer @RalfJung @rustbot