Support for `AWS_WEB_IDENTITY_TOKEN_FILE` / IAM Roles for Service Accounts (K8s)

[dancmeyers]

This is similar to #286, but is not a duplicate. I'd also like to see support for IAM Roles for Service Accounts (IRSA): https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html, which is another means to get IAM roles into containers. It was created before EKS Pod Identity (the feature referenced in #286), so there's probably a lot of people still using it (including us). Also, unlike EKS Pod Identity, it works across non-AWS-EKS clusters such as EKS Anywhere (EKS-managed clusters on your own infra), self-managed K8s clusters in AWS, etc so it still has valid use-cases that aren't covered by EKS Pod Identity.

For full support, the API would need to:

• Pick up the AWS_WEB_IDENTITY_TOKEN_FILE env var and load the token from the listed file.
  • N.B. this token value is also periodically refreshed by the system, so must be fetched from disk on every credential refresh call.
• Pick up the AWS_ROLE_ARN env var.
• Use the above two to make a request to the relevant STS endpoint (related: STS service endpoint file doesn't return regional variants #297) to fetch short lived credentials using the web identity token as the auth method.
• Parse the expected credential, TTL etc keys out of the response and auto refresh it via normal methods.

I actually have a version of this working already on our systems via a custom credentials provider, but it's not using the Cognitect http client (because other libs can take a map of kv pairs for query params, and I didn't want to have to build a query string with proper escaping manually ;)), and assumes things like AWS_REGION rather than passing through the region provider. It also builds the endpoint hostname manually due to a seeming bug with the Cognitect STS endpoint service doc (#297). It's thus not really ready for inclusion into the lib. I started work to embed it, but got a bit stuck ending up reimplementing a load of private functions from cognitect.aws.signers and the like to encode query params, and wasn't sure where the best place to put those functions was.

[marcobiscaro2112]

About the STS regional endpoint, does IRSA require issuing requests to the regional endpoint, or will it work with the global one? I'm trying to understand if #297 is a hard dependency for this or not.

[dancmeyers]

Not a hard dependency, the code I've got at the moment falls back to using the global endpoint if I create a deployment and forget to add the AWS_REGION env var, and that works just fine for those containers that are actually allowed to make connections out of our VPC (See later).

It just puts an unnecessary dependency on the us-east-1 region (I assume, I'm not aware of AWS doing anything like returning different DNS addresses based on geographic location, or using anycast DNS to route you to the closest host for the same IP).

The other thing it would do for us is break some of our firewalling. We have AWS interface VPC endpoints defined for the AWS APIs we interact with from our containers, so a DNS request for sts.eu-west-2.amazonaws.com from within our VPC resolves to an IP that exists within the VPC, and our traffic never goes outside of the VPC when it's just interacting with AWS. But the global sts.amazonaws.com endpoint still resolves to the external IP inside the VPC, so would be blocked by our current firewall policies for a lot of containers.

[dancmeyers]

Ahha, I am mistaken. AWS do route STS requests to the global endpoint to your region when running in a default-enabled region, but still recommend that you should use the regional endpoints explicitly, and will always use us-east-1 for the global endpoint if you're outside of AWS or in a non-default-enabled region.

Still doesn't fix our firewalling issue though 🤷‍♂️.