Helm Chart optionally install Gateway API's CRDs

[tisonkun]

I'm installing the AWS LBC by:

module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.23"

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  enable_external_secrets             = true
  enable_metrics_server               = true
  enable_aws_load_balancer_controller = true
  aws_load_balancer_controller = {
    chart_version = var.lbc_chart_version
    set = [
      {
        name  = "vpcId"
        value = module.vpc.vpc_id
      },
      {
        name  = "installCRDs"
        value = "true"
      },
      {
        name  = "controllerConfig.featureGates.ALBGatewayAPI"
        value = "true"
      }
    ]
  }

  depends_on = [module.eks, kubernetes_manifest.gateway_api_crds]
}

The "installCRDs" installs only CRDs dedicated to the AWS LBC, not the Gateway API's.

Then the initialized controller would restart forever because the Gateway API CRDs are not available.

Not sure if AWS EKS should provide a way to initiate out-of-the-box Gateway API CRDs, or this Helm Chart should provide a way to install the Gateway API CRDs.

I'm trying to install the CRDs separately by:

data "http" "gateway_api_crds" {
  url = "https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.0/standard-install.yaml"
}

locals {
  gateway_api_crds = [
    for manifest in provider::kubernetes::manifest_decode_multi(data.http.gateway_api_crds.response_body) :
    { for key, value in manifest : key => value if key != "status" }
  ]
}

resource "kubernetes_manifest" "gateway_api_crds" {
  count      = length(local.gateway_api_crds)
  manifest   = local.gateway_api_crds[count.index]
  depends_on = [module.eks]
}

But it fails to plan because the EKS cluster is not initiated in the very first:

│ Error: Failed to construct REST client
│ 
│   with module.base.kubernetes_manifest.gateway_api_crds[8],
│   on main.tf line 115, in resource "kubernetes_manifest" "gateway_api_crds":
│  115: resource "kubernetes_manifest" "gateway_api_crds" {
│ 
│ cannot create REST client: no client config

[tisonkun]

For example, Envoy's Gateway Helm Chart bundles the official Gateway API CRDs:

• https://github.com/envoyproxy/gateway/blob/main/charts/gateway-helm/charts/crds/crds/gatewayapi-crds.yaml

[msporleder-work]

This would be helpful so the correct CRD versions are paired with the supported LBC versions

[zac-nixon]

We didn't not include the Gateway CRDs intentionally, we didn't want to accidentally downgrade the CRD versions on the installation of the controller. We were concerned about environments that were already using the Gateway API in a different controller. If we supplied the 'suggested' CRDs in our release artifacts, would that help?

[tisonkun]

@zac-nixon Envoy provides a solution to opt-in CRDs:

https://github.com/envoyproxy/gateway/blob/dd79a83008d8172fa1065ee3cec0ea6b20261d54/charts/gateway-crds-helm/values.yaml#L1-L6

crds:
  gatewayAPI:
    enabled: false
    channel: experimental   # Available options: standard, experimental
  envoyGateway:
    enabled: false

[tisonkun]

I can try to provide a PR if this is something you guys agree to try.

[zac-nixon]

Sure, if this is an opt-in thing just like Envoy Gateway provides we would accept that as well. Thank you for checking.