Add Token Based Support for BuildStream to REAPI servers

[cphang99]

Currently REAPI clients such as Bazel support token based authenication to REAPI server implementations by passing this into authorisation headers that the server implementation then parses.

Adding this support to BuildStream would be very useful, which to the best of my knowledge only supports TLS certificate authentication currently.

A suggested sketch of things to add as part of this:

• BuildStream backend implementation for passing authentication tokens to REAPI servers
• Extending buildstream configuration for users to be able to pass authentication tokens.
• Documentation of the above.

[juergbi]

buildbox-casd already supports bearer authentication for remote servers, reading the access token from a file. However, it currently supports this only for remote servers configured on the command line, not with the dynamic configuration used by BuildStream.

Tentative list of tasks to address this issue:

• buildbox-casd
  • Support dynamic remotes also for execution servers (requires small extension to the LocalCAS protocol)
  • Support access tokens (+ optional reload interval) for dynamic remotes (requires small extension to the LocalCAS protocol)
  • If netrc support is desired, also support access tokens sent as strings over the local protocol
• BuildStream
  • Support configuring access tokens as paths (+ reload interval) in the auth dict of remotes (as alternative to client-key / client-cert) and pass that to buildbox-casd if specified
  • If netrc support is desired, add a configuration option to enable that, which then sends the password string from netrc to buildbox-casd instead of a path
  • Use buildbox-casd as proxy for all remote connections: At least remote execution currently doesn't use buildbox-casd as proxy

[juergbi]

Support dynamic remotes also for execution servers (requires small extension to the LocalCAS protocol)

MR for this: https://gitlab.com/BuildGrid/buildbox/buildbox/-/merge_requests/605.

The BuildStream change to use buildbox-casd as proxy for remote asset connections: #1922

Support for proxying remote execution is implemented in #1926

[juergbi]

• Support access tokens (+ optional reload interval) for dynamic remotes (requires small extension to the LocalCAS protocol)

MR for this: https://gitlab.com/BuildGrid/buildbox/buildbox/-/merge_requests/616

[juergbi]

Initial access token support has been merged in both buildbox-casd and BuildStream. The pending PR #1926 extends access token support also to remote execution.

This does not include netrc support but it's not clear to me whether we actually want that (as netrc is not actually designed for access tokens), so I think it's better to raise a follow-up issue if netrc support is needed.

[juergbi]

Closing this as per above comment as #1926 has been merged.