Add jit config

[gabriel-samfira]

This PR changes the way GARM registers runners. With this change, GARM will now attempt to use Just-In-Time configuration instead of passing a runner registration token to the instance to run against the ./config.sh script.

This adds a number of advantages:

• The runner registration token is valid for one hour, and can theoretically be used to register any number of runners. If compromised, an attacker could inject their own runners inside the entity that generated the token. GARM attempts to mitigate this by not sending the token via user-data, and will only allow the retrieval of the token once. If only saved in memory, that should mitigate the issue to some extent.
• The JIT configuration can only be used by one runner at any given point in time. Once we spin up the runner and it connects to github, it should not matter if an attacker manages to read the config (they can do that anyway once a job runs), because:
  • A runner is already running
  • The runner is ephemeral. Killing it while the job is running will invalidate the credentials, as JIT runners are ephemeral by nature.
• With JIT runners, we finally have the ability to skip adding the default labels to runners (eg: self-hosted, x64, linux)

This change is currently blocked until the following PRs merge and a release is cut:

• Add enterprise runner group operations google/go-github#2891
• Add GenerateEnterpriseJITConfig google/go-github#2890

[gabriel-samfira]

CC @maigl This change will work with GHES 3.10 and above. Before this is ready to merge, I will ensure that it can fall back to the old way of registering a runner. It should already be able to do that, but some extra checks would not hurt.

[maigl]

CC @maigl This change will work with GHES 3.10 and above. Before this is ready to merge, I will ensure that it can fall back to the old way of registering a runner. It should already be able to do that, but some extra checks would not hurt.

Okay thx, we're not at 3.10 yet, so the fallback is important to us.

[gabriel-samfira]

@maigl This should fall back to the old way of registering a runner. Any error in creating a JIT config should make it fall back to the old registration method.

[gabriel-samfira]

@maigl with the latest commit, fallback should work.

The logs will show something like this:

Aug 24 13:37:36 garm garm[3161868]: 2023/08/24 13:37:36 [Pool mgr ghadmin/test-repo] failed to get JIT config, falling back to registration token: failed to get JIT config: POST https://ghe.example.com/api/v3/repos/ghadmin/test-repo/actions/runners/generate-jitconfig: 404 Not Found []

If you have some cycles, it would be great if you could test it against your env. Note, you will also need to update the openstack external provider to latest main and optionally add:

[default]
webhook_url = "https://garm.example.com/webhooks"
enable_webhook_management = true

in your garm config.

Once your instance updates to 3.10 or newer, it should automatically switch to JIT.

This PR will most likely linger in draft until there is an upstream go-github release that includes the PRs in the description.

[gabriel-samfira]

CC @mkulke

I know you were looking into adding JIT at some point.

[mkulke]

CC @mkulke

I know you were looking into adding JIT at some point.

Awesome, thanks. I'll take a look