Using Custom AMIs doesn't seem to work according to the docs

[jspc]

Description

Following both the main module docs, and the docs at compute_resources.md, I get the following errors in terraform:

Call to function "coalesce" failed: no non-null, non-empty-string arguments.

Error: Error in function call
on .terraform/modules/eks/modules/_user_data/main.tf line 46, in locals:

  user_data_path = coalesce(var.user_data_template_path, local.ami_type_to_user_data_path[var.ami_type])

while calling coalesce(vals...)
local.ami_type_to_user_data_path is object with 19 attributes
var.ami_type is "CUSTOM"
var.user_data_template_path is ""
Call to function "coalesce" failed: no non-null, non-empty-string arguments.

Error: reading SSM Parameter (NONE): couldn't find resource

Error: reading SSM Parameter (NONE): couldn't find resource
with module.eks.module.eks_managed_node_group["applications"].data.aws_ssm_parameter.ami[0]
on .terraform/modules/eks/modules/eks-managed-node-group/main.tf line 429, in data "aws_ssm_parameter" "ami":

data "aws_ssm_parameter" "ami" {

The pertinent bits of my config is:

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "21.15.1"

  name               = local.name
  kubernetes_version = "1.35"

  addons = {
    coredns = {}
    eks-pod-identity-agent = {
      before_compute = true
    }
    kube-proxy = {}
    vpc-cni = {
      before_compute = true
    }
  }

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets
  iam_role_additional_policies = {
    ssm : "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
  }

  endpoint_public_access  = false
  endpoint_private_access = true

  enable_irsa                              = true
  enable_cluster_creator_admin_permissions = true

  eks_managed_node_groups = {
    applications = {
      name = "applications"

      ami_type       = "CUSTOM"
      //ami_type = "AL2023_x86_64_STANDARD"

      ami_id         = data.aws_ami.ours.id
      instance_types = ["c5.large"]
      desired_size   = 2
      min_size       = 2
      max_size       = 10

      enable_bootstrap_user_data = true

      cloudinit_pre_nodeadm = [{
        content      = <<-EOT
          ---
          apiVersion: node.eks.aws/v1alpha1
          kind: NodeConfig
          spec:
            kubelet:
              config:
                shutdownGracePeriod: 30s
        EOT
        content_type = "application/node.eks.aws"
      }]

      taints = {
        applications = {
          key    = "pool"
          value  = "applications"
          effect = "NO_SCHEDULE"
        }
      }

      tags = {
        "k8s.io/cluster-autoscaler/enabled"       = "true"
        "k8s.io/cluster-autoscaler/${local.name}" = "true"
      }
    }
}

Some of this is config I've blindly copied from the doc above.

The AMI is built from the official EKS AMI, too, so I wasn't sure whether ami_type = "CUSTOM" or ami_type = "AL2023_x86_64_STANDARD" but neither helped. The details above are from ami_type = "CUSTOM".

I've searched within github to see if any open source modules do the same as I'm trying, but I've found nothing.

Because of this, I suspect either there's a regression in the module that means an SSM parameter (I assume to find the latest AMI) is being read when it shouldn't, or the docs are wrong.

I also suspect the docs are wrong about user data for custom AMIs, but I don't know what this should be.

• ✋ I have searched the open/closed issues and my issue is not listed.

Versions

• Module version [Required]: 21.15.1

• Terraform version:
  Terraform v1.14.4
  on linux_amd64

• Provider version(s):

• provider registry.terraform.io/hashicorp/aws v6.28.0
• provider registry.terraform.io/hashicorp/cloudinit v2.3.7
• provider registry.terraform.io/hashicorp/null v3.2.4
• provider registry.terraform.io/hashicorp/time v0.13.1
• provider registry.terraform.io/hashicorp/tls v4.1.0

Reproduction Code [Required]

As above:

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "21.15.1"

  name               = local.name
  kubernetes_version = "1.35"

  addons = {
    coredns = {}
    eks-pod-identity-agent = {
      before_compute = true
    }
    kube-proxy = {}
    vpc-cni = {
      before_compute = true
    }
  }

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets
  iam_role_additional_policies = {
    ssm : "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
  }

  endpoint_public_access  = false
  endpoint_private_access = true

  enable_irsa                              = true
  enable_cluster_creator_admin_permissions = true

  eks_managed_node_groups = {
    applications = {
      name = "applications"

      ami_type       = "CUSTOM"
      //ami_type = "AL2023_x86_64_STANDARD"

      ami_id         = data.aws_ami.ours.id
      instance_types = ["c5.large"]
      desired_size   = 2
      min_size       = 2
      max_size       = 10

      enable_bootstrap_user_data = true

      cloudinit_pre_nodeadm = [{
        content      = <<-EOT
          ---
          apiVersion: node.eks.aws/v1alpha1
          kind: NodeConfig
          spec:
            kubelet:
              config:
                shutdownGracePeriod: 30s
        EOT
        content_type = "application/node.eks.aws"
      }]

      taints = {
        applications = {
          key    = "pool"
          value  = "applications"
          effect = "NO_SCHEDULE"
        }
      }

      tags = {
        "k8s.io/cluster-autoscaler/enabled"       = "true"
        "k8s.io/cluster-autoscaler/${local.name}" = "true"
      }
    }
}

Steps to reproduce the behavior:

I am running terraform apply via terraform cloud

Expected behavior

I expect a cluster to come up with the AMIs I specify

Actual behavior

As above, I get the errors:

Call to function "coalesce" failed: no non-null, non-empty-string arguments.

``` Error: Error in function call on .terraform/modules/eks/modules/_user_data/main.tf line 46, in locals:

user_data_path = coalesce(var.user_data_template_path, local.ami_type_to_user_data_path[var.ami_type])

while calling coalesce(vals...)
local.ami_type_to_user_data_path is object with 19 attributes
var.ami_type is "CUSTOM"
var.user_data_template_path is ""
Call to function "coalesce" failed: no non-null, non-empty-string arguments.

</details>

<details>
<summary>Error: reading SSM Parameter (NONE): couldn't find resource</summary>

Error: reading SSM Parameter (NONE): couldn't find resource
with module.eks.module.eks_managed_node_group["applications"].data.aws_ssm_parameter.ami[0]
on .terraform/modules/eks/modules/eks-managed-node-group/main.tf line 429, in data "aws_ssm_parameter" "ami":

data "aws_ssm_parameter" "ami" {

</details>

[jspc]

I don't suppose anybody has had a chance to look at this and/or have had any thoughts?

It's not the end of the world if I can't get this to work, because I can accomplish the same thing (a private kubernetes cluster using our internal AMIs) without this module. But I'd really love to use the module, since it's far more mature and has far more eyes.

I just can't figure out how to get it to do what I want, because the docs are either wrong or are incomplete.

[bryantbiggs]

if you are starting with an EKS provided AMI as your base AMI, specify its type not CUSTOM. CUSTOM means just that - its entirely custom and the format, bootstrap, init process, etc are all unknown and you would need to provide the full necessary user data for that type of AMI. but if you are just taking an EKS AMI and hardening it or baking in agents, just use that AMI type like ami_type = "AL2023_x86_64_STANDARD" if thats the base AMI type you used