Question: GLIBC_2.27 failure (base linux image)

[Mogost]

I am not sure does it right place or not.
I am having issues with many simple actions now

2023-09-22T11:01:44.5122142Z ##[group]Run actions/checkout@3df4ab11eba7bda6032a0b82a6bb43b11571feac
2023-09-22T11:01:44.5122557Z with:
2023-09-22T11:01:44.5122788Z   fetch-depth: 10
2023-09-22T11:01:44.5123067Z   repository: X
2023-09-22T11:01:44.5123750Z   token: ***
2023-09-22T11:01:44.5123988Z   ssh-strict: true
2023-09-22T11:01:44.5124242Z   persist-credentials: true
2023-09-22T11:01:44.5124504Z   clean: true
2023-09-22T11:01:44.5124762Z   sparse-checkout-cone-mode: true
2023-09-22T11:01:44.5125047Z   fetch-tags: false
2023-09-22T11:01:44.5125283Z   show-progress: true
2023-09-22T11:01:44.5125518Z   lfs: false
2023-09-22T11:01:44.5125740Z   submodules: false
2023-09-22T11:01:44.5125987Z   set-safe-directory: true
2023-09-22T11:01:44.5126218Z ##[endgroup]
2023-09-22T11:01:44.5476016Z /opt/actions-runner/externals/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /opt/actions-runner/externals/node20/bin/node)
2023-09-22T11:01:44.5476995Z /opt/actions-runner/externals/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /opt/actions-runner/externals/node20/bin/node)
2023-09-22T11:01:44.5871283Z Post job cleanup.
2023-09-22T11:01:44.5891705Z /opt/actions-runner/externals/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /opt/actions-runner/externals/node20/bin/node)
2023-09-22T11:01:44.5892795Z /opt/actions-runner/externals/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /opt/actions-runner/externals/node20/bin/node)
2023-09-22T11:01:44.6143782Z Cleaning up orphan processes

I assume that this is due to the base image on which my actions are run.
I do not use custom AMI, therefore I think this kind of failure may affect many people. Can you please tell me which way to look to fix this error?

[enchorb]

Same issue here using anything above node 16 on the base Amazon Linux image.

I tried to add the below to use the Ubuntu one but then for some reason while the EC2 instance spins up it fails to run the job and GitHub Actions hangs.

Any guidance on getting node 18+ working for anyone?

ami_filter = {
    name  = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"],
    state = ["available"]
  }

[ay0o]

you need to use Amazon Linux 2023.

They are working on it here https://github.com/philips-labs/terraform-aws-github-runner/pull/3437.

If you want to workaround this while they fix it, modify the user-data to install docker from yum instead of using amazon-linux-extra.

[amacleay-cohere]

We just started seeing all new runner instances start failing with this issue. Curious to know why that would have happened if this underlying problem has been present for a couple weeks.

[tristanhill-iris]

This also just started failing for me, appears to be tripped by an update to the actions runner -- actions/runner#2844

The latest release https://github.com/actions/runner/releases/tag/v2.310.0 includes this change to update the default version to node20.

I've attempted adding export ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION=node16 to the userdata, and the suggestion by @ay0o to install docker using yum, but so far nothing has worked -- it still wants to use node20, unless I start the service myself after manually setting ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION=node16.

Edit: Just figured out a workaround, but it's only for ephemeral runners. Will need adjustment if you're not using ec2-user.

1. use enable_ephemeral_runners = true
2. in the userdata_pre_install file (add one if needed), add echo "export GITHUB_ACTIONS_RUNNER_FORCED_NODE_VERSION=node16" >> /home/ec2-user/.bash_profile

[noamkush]

For anyone coming here, here's a fairly simple workaround that worked for me - integrating the AL2023 PR. This works for x64 amazon linux (the defaults).
In terraform set these:

ami_filter = { name = ["al2023-ami-2023.*-kernel-6.*-x86_64"], state = ["available"] }
userdata_template = "${path.module}/user-data.sh"

Where user-data.sh is

#!/bin/bash -e

exec > >(tee /var/log/user-data.log | logger -t user-data -s 2>/dev/console) 2>&1

# AWS suggest to create a log for debug purpose based on https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-log-user-data/
# As side effect all command, set +x disable debugging explicitly.
#
# An alternative for masking tokens could be: exec > >(sed 's/--token\ [^ ]* /--token\ *** /g' > /var/log/user-data.log) 2>&1

set +x

%{ if enable_debug_logging }
set -x
%{ endif }

${pre_install}

dnf upgrade-minimal -y

# Install docker
dnf install -y docker
service docker start
usermod -a -G docker ec2-user

dnf install -y amazon-cloudwatch-agent jq git
dnf install -y --allowerasing curl

user_name=ec2-user

# Install libicu on non-ubuntu
if [[ ! "$os_id" =~ ^ubuntu.* ]]; then
  dnf install -y libicu
fi

${install_runner}

${post_install}

${start_runner}

And apply

[morganseznec]

You could also add a line:

sed -i '/\[Service\]/a Environment="GITHUB_ACTIONS_RUNNER_FORCED_NODE_VERSION=node16"' /opt/actions-runner/bin/actions.runner.service.template

or sed -i 's/node20/node16/' /opt/actions-runner/runsvc.sh

in start-runner.sh just before ## Start the runner comment.

then run a terraform apply.

Make sure to terminate previous EC2 instances.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed if no further activity occurs. Thank you for your contributions.