New API rate limits hit by GitHub Actions node IP with 'no_auth: true'

[trinitronx]

An API limit is hit far too easily when using digitalocean/action-doctl with the no_auth: 'true' option to download a newer version of doctl than is available on the runner node.

The no_auth: 'true' option is important for security reasons when running DigitalOcean app schema validation for untrusted Pull Requests from forked repos, or from automations like @dependabot.

This option was added for use with offline app schema validation using doctl app spec validate .do/app.yaml --schema-only. A pre-commit hook exists here LyraPhase/pre-commit-digitalocean - doctl-app-spec-validate-offline which uses this functionality to run schema validation in an "offline" mode without requiring to expose a DIGITALOCEAN_ACCESS_TOKEN secret to an untrusted Pull Request or fork.

However, it now appears that there is a new API limit for downloading a version of doctl that supports the --schema-only option (doctl >= v1.101.0 digitalocean/doctl#1450). GitHub Actions node seems to only have an old fallback version by default, version 1.98.1, which does not support the --schema-only offline validation flag.

The API limit appears extremely easy to hit, perhaps due to the nature of GitHub Actions runners being shared infrastructure, and the API limit being tied to IP address. In fact, I just ran into this by merging a @dependabot PR on a project using digitalocean/action-doctl which had not had any changes or GitHub actions runs for over 5 months. On the first run of digitalocean/action-doctl after merge, it failed with the following error:

Warning: API rate limit exceeded for 52.238.24.37. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)

Failed to retrieve latest version; falling back to: 1.98.1

Expand for example pre-commit.yaml GitHub Actions workflow using no_auth: 'true'

name: pre-commit

on:
  pull_request:
    branches:
      - master
      - main
      - develop
  push:
    branches:
      - master
      - main
      - develop

jobs:
  pre-commit:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v5
    - uses: actions/setup-python@v5
      with:
        python-version: '3.11'
    - name: Install doctl
      uses: digitalocean/action-doctl@v2.5.1
      with:
        no_auth: 'true'
    - uses: pre-commit/action@v3.0.1

Expand for .pre-commit-config.yaml using LyraPhase/pre-commit-digitalocean doctl-app-spec-validate-offline

# .pre-commit-config.yaml
---
exclude: |
    (?x)^(
        .*\.svg|
        \.vscode/.*|
        assets/.*|
        target/.*|
        go.mod|
        go.sum|
        vendor/.*|
        tmp/.*
    )$
repos:
- repo: https://github.com/LyraPhase/pre-commit-digitalocean.git
  rev: v0.1.1
  hooks:
  - id: doctl-app-spec-validate-offline
    args: ['--verbose']

Logs for the failed GitHub Actions workflow can be viewed here.

Expand for relevant log sections from the failed GitHub Actions workflow

Run digitalocean/action-doctl@v2.5.1
  with:
    no_auth: true
    version: latest
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.11.13/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.11.13/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.13/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.13/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.13/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.13/x64/lib
Warning: API rate limit exceeded for 52.238.24.37. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)

Failed to retrieve latest version; falling back to: 1.98.1
/usr/bin/tar xz --warning=no-unknown-keyword -C /home/runner/work/_temp/164d3c69-a8c9-404e-8397-915de9b01fe8 -f /home/runner/work/_temp/de6a5c5e-4ddf-4bb5-88c1-25eda27a8815
>>> doctl version v1.98.1 installed to /opt/hostedtoolcache/doctl/1.98.1/x64
>>> Skipping doctl auth

Run pre-commit/action@v3.0.1
Run python -m pip install pre-commit
Collecting pre-commit
  Downloading pre_commit-4.3.0-py2.py3-none-any.whl.metadata (1.2 kB)
Collecting cfgv>=2.0.0 (from pre-commit)
  Downloading cfgv-3.4.0-py2.py3-none-any.whl.metadata (8.5 kB)
Collecting identify>=1.0.0 (from pre-commit)
  Downloading identify-2.6.13-py2.py3-none-any.whl.metadata (4.4 kB)
Collecting nodeenv>=0.11.1 (from pre-commit)
  Downloading nodeenv-1.9.1-py2.py3-none-any.whl.metadata (21 kB)
Collecting pyyaml>=5.1 (from pre-commit)
  Downloading PyYAML-6.0.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
Collecting virtualenv>=20.10.0 (from pre-commit)
  Downloading virtualenv-20.34.0-py3-none-any.whl.metadata (4.6 kB)
Collecting distlib<1,>=0.3.7 (from virtualenv>=20.10.0->pre-commit)
  Downloading distlib-0.4.0-py2.py3-none-any.whl.metadata (5.2 kB)
Collecting filelock<4,>=3.12.2 (from virtualenv>=20.10.0->pre-commit)
  Downloading filelock-3.19.1-py3-none-any.whl.metadata (2.1 kB)
Collecting platformdirs<5,>=3.9.1 (from virtualenv>=20.10.0->pre-commit)
  Downloading platformdirs-4.3.8-py3-none-any.whl.metadata (12 kB)
Downloading pre_commit-4.3.0-py2.py3-none-any.whl (220 kB)
Downloading cfgv-3.4.0-py2.py3-none-any.whl (7.2 kB)
Downloading identify-2.6.13-py2.py3-none-any.whl (99 kB)
Downloading nodeenv-1.9.1-py2.py3-none-any.whl (22 kB)
Downloading PyYAML-6.0.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (762 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 763.0/763.0 kB 93.1 MB/s  0:00:00
Downloading virtualenv-20.34.0-py3-none-any.whl (6.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/6.0 MB 156.5 MB/s  0:00:00
Downloading distlib-0.4.0-py2.py3-none-any.whl (469 kB)
Downloading filelock-3.19.1-py3-none-any.whl (15 kB)
Downloading platformdirs-4.3.8-py3-none-any.whl (18 kB)
Installing collected packages: distlib, pyyaml, platformdirs, nodeenv, identify, filelock, cfgv, virtualenv, pre-commit

Successfully installed cfgv-3.4.0 distlib-0.4.0 filelock-3.19.1 identify-2.6.13 nodeenv-1.9.1 platformdirs-4.3.8 pre-commit-4.3.0 pyyaml-6.0.2 virtualenv-20.34.0
Run python -m pip freeze --local
cfgv==3.4.0
distlib==0.4.0
filelock==3.19.1
identify==2.6.13
nodeenv==1.9.1
platformdirs==4.3.8
pre_commit==4.3.0
PyYAML==6.0.2
virtualenv==20.34.0
Run actions/cache@v4
Cache not found for input keys: pre-commit-3|/opt/hostedtoolcache/Python/3.11.13/x64|8edc02a8a73aedc615930d023e83036e854d0e6ab9c7f7c32a4e2647097b9c2b
Run pre-commit run --show-diff-on-failure --color=always --all-files
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.git.
[INFO] Initializing environment for https://github.com/igorshubovych/markdownlint-cli.
[INFO] Initializing environment for https://github.com/detailyang/pre-commit-shell.git.
[INFO] Initializing environment for https://github.com/LyraPhase/pre-commit-digitalocean.git.
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.git.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/igorshubovych/markdownlint-cli.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Check for added large files................................................Passed
Check for merge conflicts..................................................Passed
Check for broken symlinks..............................(no files to check)Skipped
Check Yaml.................................................................Passed
Fix End of Files...........................................................Passed
Trim Trailing Whitespace...................................................Passed
markdownlint...............................................................Passed
Shell Syntax Check.....................................(no files to check)Skipped
DigitalOcean App Spec Validate Offline (schema-only).......................Failed
- hook id: doctl-app-spec-validate-offline
- exit code: 1

Error: Unable to initialize DigitalOcean API client: access token is required. (hint: run 'doctl auth init')

Error: Process completed with exit code 1.

An old version of doctl was used as fallback.

Thus, the doctl app spec validate [...] --schema-only flag did not exist, resulting in job failure.

[Shakeskeyboarde]

I don't think the no_auth option has anything to do with the rate limit. The rate limit is a github rate limit when trying to download the release binary from https://github.com/digitalocean/doctl/releases/download/.... The no_auth option only affects logging into DOCTL.

The problem is that no credentials (auth parameter) are being passed to the downloadTool (@actions/tool-cache) method. If there were a way to pass in a github token for the download to use (secrets.GITHUB_TOKEN), then I think the rate limit would be increased and also based on the credentials rather than the IP.

[trinitronx]

The problem is that no credentials (auth parameter) are being passed to the downloadTool (@actions/tool-cache) method. If there were a way to pass in a github token for the download to use (secrets.GITHUB_TOKEN), then I think the rate limit would be increased and also based on the credentials rather than the IP.

Ah, yes this seems to be the issue. The auth parameter for the downloadTool function would need to be passed by action-doctl in downloadDoctl here.

It looks like the default secrets.GITHUB_TOKEN could be passed as a standard HTTP auth header like: Authorization: Bearer ${GITHUB_TOKEN}

That would require a small change to the action-doctl to pass that interpolated string as the auth parameter. Then, a new parameter could be added to the action to allow passing an arbitrary GitHub token rather than the default secrets.GITHUB_TOKEN. (Or just make it an optional parameter, and always require the user to declaratively pass it if they want one used at all.)