diff --git a/.github/gh-actions-self-hosted-runners/README.md b/.github/gh-actions-self-hosted-runners/README.md new file mode 100644 index 000000000000..7790197ee69a --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/README.md @@ -0,0 +1,121 @@ + +# GitHub Actions - Self-hosted Runners +The current GitHub Actions workflows are being tested on multiple operating systems, such as Ubuntu, Windows and MacOS. The way to migrate these runners from GitHub to GCP is by implementing self-hosted runners, so we implemented them in both Ubuntu and Windows environments, going with Google Kubernetes Engine and Google Cloud Compute VMs instances respectively. + +On the other hand, we will rely on GitHub-hosted runners for MacOS builds until a straightforward implementation approach comes out. + +## Ubuntu +Ubuntu Self-hosted runners are stored in Artifact Registry and implemented using Google Kubernetes Engine with the following specifications: + +#### Cluster +* Cluster: [gh-actions-linux-runners](https://console.cloud.google.com/kubernetes/clusters/details/us-central1-a/gh-actions-linux-runners/details?project=apache-beam-testing) +* Image: [linux-github-actions-runner](https://console.cloud.google.com/artifacts/docker/apache-beam-testing/us-central1/beam-github-actions/linux-github-actions-runner?project=apache-beam-testing) + +#### Pool +* Number of nodes: 5 +* Cluster Autoscaler: ON + * Minimum number of nodes: 5 + * Maximum number of nodes: 10 + +#### Node +* Machine Type: e2-custom-6-18432 +* Disk Size: 100 GB +* CPU: 6 vCPUs +* Memory : 18 GB + +#### Pod +##### Container 1: gh-actions-runner +* Image: $LOCAL_IMAGE_NAME LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:latest +* CPU: 2 +* Memory: 1028 Mi +* Volumes + * gcloud-key + * docker-certs-client +* Environment variables + * Container variables + * GOOGLE_APPLICATION_CREDENTIALS + * DOCKER_HOST + * DOCKER_TLS_VERIFY + * DOCKER_CERT_PATH + * Kubernetes secret env variables + * github-actions-secrets + * gcloud-key + +##### Container 2: dind +* Image: `docker:20.10.17-dind` +* Volumes + * dind-storage + * docker-certs-client + +###### Pod Diagram +![PodDiagram](diagrams/gh-actions-k8s-runners-pod.png) + +#### AutoScaling +* Horizontal Pod Autoscaling + * 5-10 nodes (From Pool Cluster Autoscaler) + * HorizontalPodAutoscaler + * Min replicas: 10 + * Max replicas: 20 + * CPU utilization: 70% +* Vertical Pod Autoscaling + * updateMode: "Auto" + + +## Windows +Windows Virtual machines have the following specifications + +#### VM specifications +* Instance Template: windows_github_actions_runner +* Machine Type: n2-standard-2 +* Disk Size: 70 GB +* Disk Image: [disk-image-windows-runner](https://console.cloud.google.com/compute/imagesDetail/projects/apache-beam-testing/global/images/disk-image-windows-runner?project=apache-beam-testing) +* CPU: 2 vCPUs +* Memory : 8 GB + +#### Instance group settings +* Region: us-west1 (multizone) +* Scale-out metric: 70% of CPU Usage. +* Cooldown period: 300s + +#### Notes: +At first glance we considered implementing Windows runners using K8s, however this was not optimal because of the following reasons: + +* VS Build tools are required for certain workflows, unfortunately official images that support this dependency are huge in size, reaching 20GB easily which is not an ideal case for k8S management. +* Windows Subsystem For Linux(WSL) is a feature that allows to execute bash scripts inside Windows which removes tech debt by avoiding writing steps in powershell, but this feature is disabled with payload removed in Windows containers. + + +## Self-Hosted Runners Architecture +![Diagram](diagrams/self-hosted-runners-architecture.png) + + +## GitHub Actions Workflow - Monitor Self-hosted Runners + +In order to monitor the Self-hosted Runners status, we have implemented a separate GitHub Actions workflow using GitHub-hosted runners, this workflow periodically calls a Cloud Function that serves data regarding the number of `active` and `offline` runners. In case of failure this workflow will send an email +alert to the dev distribution email `dev@beam.apache.org`. + +The Cloud Function uses the endpoints provided by the installed GitHub App to retrieve information about the runners. + +## Cronjob - Delete Unused Self-hosted Runners + +Depending on the termination event, sometimes the removal script for offline runners is not triggered correctly from inside the VMs or K8s pod, because of that an additional pipeline was created in order to clean up the list of GitHub runners in the group. + +This was implemented using a [GCP Cloud Function](https://console.cloud.google.com/functions/details/us-central1/remove-self-hosted-runners-group?env=gen1&project=apache-beam-testing&tab=source) [[code]](./helper-functions/cloud-functions/removeOfflineRunners) subscribed to a [Pub/Sub](https://console.cloud.google.com/cloudpubsub/topic/detail/remove-runners?referrer=search&project=apache-beam-testing) topic, the topic is triggered through a [Cloud Scheduler](https://console.cloud.google.com/cloudscheduler/jobs/edit/us-central1/runners-clean-up-schedule?project=apache-beam-testing) that is executed once per day, the function consumes a [GitHub API](https://docs.github.com/en/rest/reference/actions#delete-a-self-hosted-runner-from-an-organization) to delete offline self-hosted runners from the organization retrieving the token with its service account to secrets manager. + + +![Delete Offline Self-hosted Runners](diagrams/self-hosted-runners-delete-function.png) \ No newline at end of file diff --git a/.github/gh-actions-self-hosted-runners/diagrams/gh-actions-k8s-runners-pod.png b/.github/gh-actions-self-hosted-runners/diagrams/gh-actions-k8s-runners-pod.png new file mode 100644 index 000000000000..882aafcb246e Binary files /dev/null and b/.github/gh-actions-self-hosted-runners/diagrams/gh-actions-k8s-runners-pod.png differ diff --git a/.github/gh-actions-self-hosted-runners/diagrams/self-hosted-runners-architecture.png b/.github/gh-actions-self-hosted-runners/diagrams/self-hosted-runners-architecture.png new file mode 100644 index 000000000000..dcc337640d31 Binary files /dev/null and b/.github/gh-actions-self-hosted-runners/diagrams/self-hosted-runners-architecture.png differ diff --git a/.github/gh-actions-self-hosted-runners/diagrams/self-hosted-runners-delete-function.png b/.github/gh-actions-self-hosted-runners/diagrams/self-hosted-runners-delete-function.png new file mode 100644 index 000000000000..5b6858b26b60 Binary files /dev/null and b/.github/gh-actions-self-hosted-runners/diagrams/self-hosted-runners-delete-function.png differ diff --git a/.github/gh-actions-self-hosted-runners/helper-functions/README.md b/.github/gh-actions-self-hosted-runners/helper-functions/README.md new file mode 100644 index 000000000000..3f462f2a709e --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/helper-functions/README.md @@ -0,0 +1,39 @@ + + +## Helper Functions + +### GCP Cloud Functions +For the local implementation of the following three functions is required to create an `.env` file. If you'd like to reproduce this code in a GCP Cloud Functions, you will need to replace the variables in the `Variables` section either as `Runtime environment variables` or `Secrets`. + +``` +APP_INSTALLATION_ID= +APP_ID= +CLIENT_ID= +PEM_KEY= +CLIENT_NAME= +ORG= +PROJECT_ID= +``` + +#### generateToken +Function implemented to retrieve the GitHub App token for register a self-hosted runner. It's used in the [entrypoint](../self-hosted-linux/docker/entrypoint.sh) of the Dockerfile. +##### monitorRunnersStatus +Function implemented to get the status of the self-hosted runners. It's used in the [monitor_self_hosted_runners](../../workflows/monitor_self_hosted_runners.yml) workflow. +#### removeOfflineRunners +Function implemented to delete the unused self-hosted runners. Please refer to this [README](../self-hosted-linux/README.md) for more details. \ No newline at end of file diff --git a/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/example.env b/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/example.env new file mode 100644 index 000000000000..162c4dbaecd9 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/example.env @@ -0,0 +1,8 @@ +# Please create a .env file with the corresponding values of the variables +APP_INSTALLATION_ID= +APP_ID= +CLIENT_ID= +PEM_KEY= +CLIENT_NAME= +ORG= +PROJECT_ID= \ No newline at end of file diff --git a/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/generateToken/index.js b/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/generateToken/index.js new file mode 100644 index 000000000000..eabf1a5b3f29 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/generateToken/index.js @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// This function provides registration tokens for windows and linux self-hosted +// runners, a service account with the appropriated permissions should be used to +// invoke the Cloud Function. + +import functions from '@google-cloud/functions-framework'; +import { Octokit } from "octokit"; +import { createAppAuth } from "@octokit/auth-app"; +import { REQUIRED_ENV_VARS } from "../shared/constants" ; + +function validateEnvSet(envVars) { + envVars.forEach(envVar => { + if (!process.env[envVar]) { + throw new Error(`${envVar} environment variable not set.`) + } + }); +} +async function getRunnerToken() { + try { + //Set your GH App values as environment variables + let authOptions = { + appId: process.env.APP_ID, + privateKey: process.env.PEM_KEY, + clientId: process.env.CLIENT_ID, + clientSecret: process.env.CLIENT_SECRET, + installationId: process.env.APP_INSTALLATION_ID + } + const octokit = new Octokit({ + authStrategy: createAppAuth, + auth: authOptions + }); + let access = await octokit.request(`POST /app/installations/${process.env.APP_INSTALLATION_ID}}/access_tokens`, { + repositories: [ + 'beam' + ], + permissions: { + organization_self_hosted_runners: "write", + } + }); + //In order to access the registration token endpoint, an additional Auth token must be used + let authToken = access.data.token; + let auth = ` token ${authToken}`; + let registrationToken = await octokit.request(`POST https://api.github.com/orgs/${process.env.ORG}/actions/runners/registration-token`, { + headers: { + authorization: auth, + }, + }); + return registrationToken.data; + } catch (error) { + console.error(error); + } +} + +functions.http('generateToken', (req, res) => { + validateEnvSet(REQUIRED_ENV_VARS) + getRunnerToken().then((registrationToken) => { + res.status(200).send(registrationToken); + }); +}); diff --git a/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/generateToken/package.json b/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/generateToken/package.json new file mode 100644 index 000000000000..4ad59848970c --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/generateToken/package.json @@ -0,0 +1,14 @@ +{ + "name": "generate-token", + "version": "1.0.0", + "description": "Generates registration tokens for self-hosted runners", + "main": "", + "type": "module", + "dependencies": { + "@google-cloud/debug-agent": "^5.0.0", + "@google-cloud/functions-framework": "^2.0.0", + "@google-cloud/secret-manager": "^3.11.0", + "@octokit/auth-app": "^3.6.1", + "octokit": "^1.7.1" + } +} diff --git a/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/index.js b/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/index.js new file mode 100644 index 000000000000..0ac14a1c5daf --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/index.js @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// This function will return the number of online and offline runners for +// each OS (Windows, linux), an additional Github actions workflow will run +// to request this Cloud Function and send alerts based on the status. + +import functions from '@google-cloud/functions-framework'; +import { Octokit } from "octokit"; +import { createAppAuth } from "@octokit/auth-app"; +import { REQUIRED_ENV_VARS } from "../shared/constants" ; + +function validateEnvSet(envVars) { + envVars.forEach(envVar => { + if (!process.env[envVar]) { + throw new Error(`${envVar} environment variable not set.`) + } + }); +} + +async function monitorRunnerStatus() { + try { + //Set your GH App values as environment variables + let authOptions = { + appId: process.env.APP_ID, + privateKey: process.env.PEM_KEY, + clientId: process.env.CLIENT_ID, + clientSecret: process.env.CLIENT_SECRET, + installationId: process.env.APP_INSTALLATION_ID + } + const octokit = new Octokit({ + authStrategy: createAppAuth, + auth: authOptions + }); + + const runners = await octokit.paginate("GET /orgs/${process.env.ORG}/actions/runners", { + org: process.env.ORG + }, + ) + + //Filtering BEAM runners + let beamRunners = runners.filter(runner => { + return runner.labels.find(label => label.name == "beam") + }); + + //Dividing status for each runner OS + let osList = ["Linux", "Windows"]; + let status = {} + for (let os of osList) { + let osRunners = beamRunners.filter(runner => { + return runner.labels.find(label => label.name == os) + }); + let onlineRunners = osRunners.filter(runner => { + return runner.status == "online"; + }); + status[os] = { + "totalRunners": osRunners.length, + "onlineRunners": onlineRunners.length, + "offlineRunners": osRunners.length - onlineRunners.length + } + } + return status; + } catch (error) { + console.error(error); + } +} + +functions.http('monitorRunnerStatus', (req, res) => { + validateEnvSet(REQUIRED_ENV_VARS) + monitorRunnerStatus().then((status) => { + res.status(200).send(status); + }); +}); diff --git a/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/package.json b/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/package.json new file mode 100644 index 000000000000..a521dd9a8628 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/package.json @@ -0,0 +1,14 @@ +{ + "name": "monitor-runner-status", + "version": "1.0.0", + "description": "Monitor the status of the runners and return filtered insights", + "main": "", + "type": "module", + "dependencies": { + "@google-cloud/debug-agent": "^5.0.0", + "@google-cloud/functions-framework": "^2.0.0", + "@google-cloud/secret-manager": "^3.11.0", + "@octokit/auth-app": "^3.6.1", + "octokit": "^1.7.1" + } +} \ No newline at end of file diff --git a/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/removeOfflineRunners/index.js b/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/removeOfflineRunners/index.js new file mode 100644 index 000000000000..0271329d8464 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/removeOfflineRunners/index.js @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Unused offline self-hosted runners remains in the runners list +// unless it is explicitly removed, this function will periodically +// clean the list to only have active runners in the repo. + +import functions from '@google-cloud/functions-framework'; +import { Octokit } from "octokit"; +import { createAppAuth } from "@octokit/auth-app"; +import { REQUIRED_ENV_VARS } from "../shared/constants" ; + +function validateEnvSet(envVars) { + envVars.forEach(envVar => { + if (!process.env[envVar]) { + throw new Error(`${envVar} environment variable not set.`) + } + }); +} + +async function removeOfflineRunners() { + try { + //Set your GH App values as environment variables + let authOptions = { + appId: process.env.APP_ID, + privateKey: process.env.PEM_KEY, + clientId: process.env.CLIENT_ID, + clientSecret: process.env.CLIENT_SECRET, + installationId: process.env.APP_INSTALLATION_ID + } + const octokit = new Octokit({ + authStrategy: createAppAuth, + auth: authOptions + }); + + const runners = await octokit.paginate("GET /orgs/${process.env.ORG}/actions/runners", { + org: process.env.ORG + }, + ) + + //Filtering BEAM runners + let beamRunners = runners.filter(runner => { + return runner.labels.find(label => label.name == "beam") + }); + + //Getting offline runners only + let offlineRunners = beamRunners.filter(runner => { + return runner.status == "offline"; + }) + + //Deleting each offline runner in the list + for (let runner of offlineRunners) { + await octokit.request(`DELETE /orgs/${process.env.ORG}/actions/runners/${runner.id}`, {}); + } + return offlineRunners + } catch (error) { + console.error(error); + } +} + +functions.http('removeOfflineRunners', (req, res) => { + validateEnvSet(REQUIRED_ENV_VARS) + removeOfflineRunners().then((status) => { + res.status(200).send(status); + }); +}); diff --git a/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/removeOfflineRunners/package.json b/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/removeOfflineRunners/package.json new file mode 100644 index 000000000000..d4d00cb3da48 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/removeOfflineRunners/package.json @@ -0,0 +1,14 @@ +{ + "name": "remove-offline-runners", + "version": "1.0.0", + "description": "Removes offline runners from list", + "main": "", + "type": "module", + "dependencies": { + "@google-cloud/debug-agent": "^5.0.0", + "@google-cloud/functions-framework": "^2.0.0", + "@google-cloud/secret-manager": "^3.11.0", + "@octokit/auth-app": "^3.6.1", + "octokit": "^1.7.1" + } +} \ No newline at end of file diff --git a/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/shared/constants.js b/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/shared/constants.js new file mode 100644 index 000000000000..c582b7931b08 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/shared/constants.js @@ -0,0 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +export const REQUIRED_ENV_VARS= ["APP_ID","PEM_KEY","CLIENT_ID","CLIENT_SECRET","APP_INSTALLATION_ID","ORG"]; diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/README.md b/.github/gh-actions-self-hosted-runners/self-hosted-linux/README.md new file mode 100644 index 000000000000..f4b689a3ab2d --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/README.md @@ -0,0 +1,100 @@ + + +# GitHub Actions - Self-hosted Linux Runners +These folders contain the required resources to deploy the GitHub Actions self-hosted runners for the workflows running in Ubuntu OS. +* /docker + * Dockerfile and entrypoint: Resources needed to create a new self-hosted-runner image. + * docker-compose.yml: In case you would like to test and run the self-hosted runner locally. + +* /kubernetes + * Kubernetes files to create the resources needed to deploy the self-hosted runners. + +## Docker + +#### How to build a new image and push it to the Artifact Registry? +* Create the image locally + +`docker build -t $LOCAL_IMAGE_NAME:TAG .` + +* Tag the local image with the GCP repository name + +`docker tag $LOCAL_IMAGE_NAME LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:latest` + +* Make sure you are authenticated to the gcloud repository + +`gcloud auth configure-docker us-central1-docker.pkg.dev` + +* Push the tagged image to the repository + +`docker push LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:latest` + +* _**Important: Please add the commit hash as a tag when a new image is uploaded.**_ + +#### How to run a self-hosted runner locally? + +* Create a `.var.env` file from the `example.var.env` file and replace the corresponding values: + * GOOGLE_APPLICATION_CREDENTIALS: Path where the json file will be stored (Same path as the one given in the docker-compose.yml file) + * CLOUD_FUNCTION_NAME: Function where the self-hosted runner will get the RUNNER_TOKEN + * ORG_RUNNER_GROUP: Name of the GitHub Actions Runner Group. You can find it here: https://github.com/organizations/ORG_NAME/settings/actions/runner-groups + * ORG_NAME: Name of the Organization + * GCP_REGION: GCP Region where your Cloud Function is deployed + * GCP_PROJECT_ID: GCP Project ID where your Cloud Function is deployed + + +* Run the container + +`docker compose up` + +* You will be able to see the self-hosted runner in the Settings of your GitHub repository + +## Kubernetes + +For the current implementation we are using Google Kubernetes Engine (GKE) as a container orchestration platform. + +* Configure your Kubernetes local context + +`gcloud container clusters get-credentials $GCP_CLUSTER_NAME --zone $GCP_REGION --project $GCP_PROJECT-ID` + +* Create the desired k8s namespace + +`kubectl create namespace $NAMESPACE` + +* Create the GKE secret from a json file + +`kubectl create secret generic $k8s_SECRET_NAME --from-file=key.json=$LOCAL_PATH --namespace $NAMESPACE` + +* Update the `github-actions-secrets.yml` file with its corresponding values encrypted in base64 + +`echo -n "$VARIABLE_VALUE" | base64` + +* Replace in `github-actions-deployment.yml` file the `$IMAGE_URL` variable with the corresponding image URL: `GCP_LOCATION-docker.pkg.dev/GCP_PROJECT_ID/REPOSITORY_NAME/IMAGE_NAME` + + +* In case you would like to create the deployment from scratch, run the `run-k8s-deployment.sh` script to execute the Kubernetes deployment in the GKE cluster. + * **Important: Make sure you have the GKE context selected in your local machine:** `kubectl config current-context` + +`./run-k8s-deployment.sh $NAMESPACE` + +* Otherwise, apply only the changes. **Important: Make sure you have the GKE context selected in your local machine:** `kubectl config current-context` + +`kubectl apply -f github-actions-$FILE_NAME.yml --namespace $NAMESPACE` + +* In case you would like to delete all the Kubernetes resources, run the `delete-k8s-deployment.sh` script with its corresponding namespace value. + +`./delete-k8s-deployment.sh $NAMESPACE` \ No newline at end of file diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/Dockerfile b/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/Dockerfile new file mode 100644 index 000000000000..dd9580882178 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/Dockerfile @@ -0,0 +1,59 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +FROM ubuntu:20.04 + +ARG RUNNER_VERSION=2.296.2 + +RUN apt-get -yqq update +# Install APT packages +RUN DEBIAN_FRONTEND=noninteractive apt-get install -yqq acl aria2 apt-transport-https binutils bison brotli build-essential bzip2 ca-certificates coreutils curl dbus dnsutils dpkg fakeroot file flex fonts-noto-color-emoji ftp gnupg2 haveged imagemagick iproute2 iputils-ping jq lib32z1 libc++-dev libc++abi-dev libcurl4 libgbm-dev libgconf-2-4 libgsl-dev libgtk-3-0 libicu66 libkrb5-3 liblttng-ust0 libmagic-dev libmagickcore-dev libmagickwand-dev libsecret-1-dev libssl1.1 libsqlite3-dev libxss1 locales m4 mediainfo mercurial net-tools netcat openssh-client p7zip-full p7zip-rar parallel pass patchelf pkg-config pollinate python-is-python3 rpm rsync shellcheck sqlite3 ssh sshpass subversion sudo swig telnet texinfo time tk tzdata unzip upx wget xorriso xvfb xz-utils zip zlib1g zsync +RUN apt-get clean +RUN rm -rf /var/lib/apt/lists/* +RUN useradd -m actions && usermod -aG sudo actions && echo "%sudo ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers +USER actions +WORKDIR /home/actions + +# Get GitHub Actions Self-hosted Runners +RUN curl -O -L https://github.com/actions/runner/releases/download/v$RUNNER_VERSION/actions-runner-linux-x64-$RUNNER_VERSION.tar.gz +RUN tar xzf ./actions-runner-linux-x64-$RUNNER_VERSION.tar.gz + +# Get Docker +RUN curl -fsSL https://get.docker.com -o get-docker.sh +RUN sudo sh get-docker.sh + +# Install docker-compose +RUN sudo apt-get update && sudo apt-get install docker-compose-plugin + +# Install gcloud +# Make sure that your operating system meets the requirements +RUN sudo apt-get install apt-transport-https ca-certificates gnupg +# Add the gcloud CLI distribution URI as a package source +RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list +# Import the Google Cloud public key +RUN curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add - +# Update and install the gcloud CLI +RUN sudo apt-get update && sudo apt-get install google-cloud-cli + + +# Copy and chmod the entrypoint file +COPY --chown=actions:actions entrypoint.sh ./entrypoint.sh +RUN sudo chmod u+x ./entrypoint.sh + +CMD ["/home/actions/entrypoint.sh"] +ENTRYPOINT ["/bin/bash"] diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/docker-compose.yml b/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/docker-compose.yml new file mode 100644 index 000000000000..9f97c7cee875 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/docker-compose.yml @@ -0,0 +1,30 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +version: "3.0" +services: + runner: + hostname: local-runner-test + build: . + entrypoint: ["/home/actions/entrypoint.sh"] + image: $IMAGE_URL:${TAG:-latest} # Replace variable manually + volumes: + - "/var/run/docker.sock:/var/run/docker.sock" + - "$USER_LOCAL_PATH:/var/secrets/google/key.json" # Replace variable manually + env_file: + - ./var.env diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/entrypoint.sh b/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/entrypoint.sh new file mode 100644 index 000000000000..ab28021fe4f4 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/entrypoint.sh @@ -0,0 +1,58 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Verify docker and docker compose installation +docker -v && docker compose version + +# Verify gcloud installation +gcloud version + +# Activate the mounted Service Account. +gcloud auth activate-service-account --key-file="${GOOGLE_APPLICATION_CREDENTIALS}" --no-user-output-enabled + +# Create gcloud docker config +gcloud auth configure-docker -q --no-user-output-enabled >& /dev/null + +# Register the Runner Token in the given Organization. +CLOUD_FUNCTION_URL="https://$GCP_REGION-$GCP_PROJECT_ID.cloudfunctions.net/$CLOUD_FUNCTION_NAME" + +# Get the Runner token from the specified Google Cloud Function +response=$(curl -sX POST -H "Authorization:bearer $(gcloud auth print-identity-token)" ${CLOUD_FUNCTION_URL}) +RUNNER_TOKEN=$(echo "$response" | jq '.token' --raw-output) + +./config.sh \ + --name $(hostname) \ + --token ${RUNNER_TOKEN} \ + --url https://github.com/$ORG_NAME \ + --work _work \ + --unattended \ + --replace \ + --labels ubuntu-20.04,beam \ + --runnergroup ${ORG_RUNNER_GROUP} \ + --ephemeral + +remove() { + ./config.sh remove --token "${RUNNER_TOKEN}" +} +trap 'remove; exit 130' INT +trap 'remove; exit 143' TERM +trap 'remove; exit 143' QUIT +trap 'remove; exit 143' ABRT +trap 'remove; exit 0' EXIT + +./run.sh "$*" & +wait $! diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/example.var.env b/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/example.var.env new file mode 100644 index 000000000000..e46a186ec239 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/example.var.env @@ -0,0 +1,7 @@ +# var.env example, please create a var.env file with the corresponding values of the variables +GOOGLE_APPLICATION_CREDENTIALS= +CLOUD_FUNCTION_NAME= +ORG_RUNNER_GROUP= +ORG_NAME= +GCP_REGION= +GCP_PROJECT_ID= \ No newline at end of file diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/delete-k8s-deployment.sh b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/delete-k8s-deployment.sh new file mode 100644 index 000000000000..41f24fa84f1b --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/delete-k8s-deployment.sh @@ -0,0 +1,20 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +kubectl delete -f github-actions-hpa.yml --namespace "$1" +kubectl delete -f github-actions-vpa.yml --namespace "$1" +kubectl delete -f github-actions-secrets.yml --namespace "$1" +kubectl delete -f github-actions-deployment.yml --namespace "$1" diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-deployment.yml b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-deployment.yml new file mode 100644 index 000000000000..88acc542bc2b --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-deployment.yml @@ -0,0 +1,76 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ubuntu-runners + labels: + name: github-actions +spec: + selector: + matchLabels: + name: github-actions + replicas: 10 + template: + metadata: + labels: + name: github-actions + spec: + containers: + - name: github-actions-ubuntu-runner + image: $IMAGE_URL:latest # Replace variable manually + securityContext: + privileged: true + envFrom: + - secretRef: + name: github-actions-secrets + env: + - name: GOOGLE_APPLICATION_CREDENTIALS + value: /var/secrets/google/key.json + - name: DOCKER_HOST + value: tcp://localhost:2376 + - name: DOCKER_TLS_VERIFY + value: '1' + - name: DOCKER_CERT_PATH + value: /certs/client + volumeMounts: + - name: gcloud-key + mountPath: /var/secrets/google + - name: docker-certs-client + mountPath: /certs/client + resources: + requests: + memory: "1028Mi" + cpu: "2" + - name: dind + image: docker:20.10.17-dind + securityContext: + privileged: true + volumeMounts: + - name: dind-storage + mountPath: /var/lib/docker + - name: docker-certs-client + mountPath: /certs/client + volumes: + - name: gcloud-key + secret: + secretName: $k8s_SECRET_NAME # Replace variable manually + - name: dind-storage + emptyDir: {} + - name: docker-certs-client + emptyDir: {} diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-hpa.yml b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-hpa.yml new file mode 100644 index 000000000000..d7d2a25199fc --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-hpa.yml @@ -0,0 +1,29 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ +apiVersion: autoscaling/v1 +kind: HorizontalPodAutoscaler +metadata: + name: hpa-github-actions-linux +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: ubuntu-runners + minReplicas: 10 + maxReplicas: 20 + targetCPUUtilizationPercentage: 70 diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-secrets.yml b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-secrets.yml new file mode 100644 index 000000000000..ef6773bd14b5 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-secrets.yml @@ -0,0 +1,30 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +apiVersion: v1 +kind: Secret +metadata: + name: github-actions-secrets +type: Opaque +data: + CLOUD_FUNCTION_NAME: + GCP_PROJECT_ID: + GCP_REGION: + ORG_NAME: + ORG_RUNNER_GROUP: +immutable: true diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-vpa.yml b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-vpa.yml new file mode 100644 index 000000000000..5cb0559c23d7 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-vpa.yml @@ -0,0 +1,28 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ +apiVersion: autoscaling.k8s.io/v1 +kind: VerticalPodAutoscaler +metadata: + name: vpa-github-actions +spec: + targetRef: + apiVersion: "apps/v1" + kind: Deployment + name: ubuntu-runners + updatePolicy: + updateMode: "Auto" diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/run-k8s-deployment.sh b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/run-k8s-deployment.sh new file mode 100644 index 000000000000..f3af464b1822 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/run-k8s-deployment.sh @@ -0,0 +1,22 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +kubectl create namespace "$1" +kubectl apply -f github-actions-secrets.yml --namespace "$1" +kubectl apply -f github-actions-deployment.yml --namespace "$1" +kubectl apply -f github-actions-hpa.yml --namespace "$1" +gcloud container clusters update gh-actions-linux-runners --enable-vertical-pod-autoscaling --zone us-central1-a +kubectl apply -f github-actions-vpa.yml --namespace "$1" diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-windows/README.md b/.github/gh-actions-self-hosted-runners/self-hosted-windows/README.md new file mode 100644 index 000000000000..548d74b7626c --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-windows/README.md @@ -0,0 +1,43 @@ + + +# GitHub Actions - Self-hosted Windows Runners + +This folder contains the required resources to deploy the GitHub Actions self-hosted runners for the workflows running in Windows OS. + +#### How to build a new instance template for the instance group? + +* Create a new VM Instance using Windows 2019 Datacenter OS with at least 70GB of disk, 2vCPUs and 8 GB of RAM. + +* Install the following software: + * VS Build Tools 2019 + * MS Build Tools + * Desktop Development with C++ + * Testing tool Core Features + * Windows 10 SDK + * Git 2.34.1.windows.1 + * Git LFS + * Git Bash + +* Create an image disk from the VM instance. + +* Now that you have the image disk you can create an instance template by using it and adding the startup and shutdown scripts through the respective metadata fields. + +**Be sure that you are using a service account that has permissions to invoke cloud functions.** + +Now that the instance template is ready you can use it to either run it independently for an individual runner or create an instance group for a set of runners. \ No newline at end of file diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-windows/shutdownScript.ps1 b/.github/gh-actions-self-hosted-runners/self-hosted-windows/shutdownScript.ps1 new file mode 100644 index 000000000000..f4bb20de89d5 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-windows/shutdownScript.ps1 @@ -0,0 +1,23 @@ +# + +Write-Output "removingRunner" +Set-Location C:/actionsDir + +$token=[System.Environment]::GetEnvironmentVariable('GITHUB_TOKEN','machine') +./config.cmd remove --token $token \ No newline at end of file diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-windows/startupScript.ps1 b/.github/gh-actions-self-hosted-runners/self-hosted-windows/startupScript.ps1 new file mode 100644 index 000000000000..1d235e55b00a --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-windows/startupScript.ps1 @@ -0,0 +1,49 @@ +# + +Start-Sleep -Seconds 30 +$env:Path += ';C:\Program Files\git\bin' +$response= Invoke-RestMethod https://api.github.com/repos/actions/runner/tags +$version= $response[0].name.substring(1,$response[0].name.Length-1) + +$ORG_NAME="apache" +$ORG_RUNNER_GROUP="Beam" +$GCP_TOKEN=gcloud auth print-identity-token +$TOKEN_PROVIDER="https://$GCP_REGION-$GCP_PROJECT_ID.cloudfunctions.net/$CLOUD_FUNCTION_NAME" #Replace variables manually + +Set-Location C:/ + +Write-Output "Starting registration process" + +mkdir "actionsDir" + +Set-Location "actionsDir" + +Invoke-WebRequest -Uri https://github.com/actions/runner/releases/download/v$version/actions-runner-win-x64-$version.zip -OutFile actions-runner-win-x64-$version.zip + +Expand-Archive -LiteralPath $PWD\actions-runner-win-x64-$version.zip -DestinationPath $PWD -Force + +$RUNNER_TOKEN=(Invoke-WebRequest -Uri $TOKEN_PROVIDER -Method POST -Headers @{'Accept' = 'application/json'; 'Authorization' = "bearer $GCP_TOKEN"} -UseBasicParsing | ConvertFrom-Json).token + +[System.Environment]::SetEnvironmentVariable('GITHUB_TOKEN', $RUNNER_TOKEN,[System.EnvironmentVariableTarget]::Machine) + +$hostname= "windows-runner-"+[guid]::NewGuid() + +./config.cmd --name $hostname --token $RUNNER_TOKEN --url https://github.com/$ORG_NAME --work _work --unattended --replace --labels windows,beam,windows-server-2019 --runnergroup $ORG_RUNNER_GROUP + +./run.cmd \ No newline at end of file