Unsupervised/unsupervised-scheduler
2
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases 19 Wiki Activity

CI: Setup PHP is slow and flaky for PHP 8.3 only, on arm64 runners #178

New Issue
Closed
opened 2026-08-20 17:23:44 +00:00 by thatguygriff · 5 comments
thatguygriff commented 2026-08-20 17:23:44 +00:00
Owner
Copy Link
Copy Source

The Setup PHP step fails intermittently across CI and release workflows. When it fails it sits silent for ~12 minutes and then reports only Could not setup PHP 8.3, which reads as a hang. #175 guessed a GitHub API rate limit and added a 1Password-backed token; that was the wrong diagnosis and #177 reverts it.

Evidence

Every Setup PHP step across runs 454–523:

PHP attempts range failures
8.1 6 26–40s 0
8.2 6 29–41s 0
8.3 26 143–1273s 2

Median 8.3 setup by cluster: eris 382s, kallone 452s, nemesis 151s. 8.1/8.2 are ~35s on all three. So it is not cluster-specific — every cluster is slow on 8.3, and no cluster is slow on 8.1/8.2.

The 8.3 penalty is not new: it was ~145s back on 30 July (run 454). What has grown is the tail — 879s and 1273s on 11 August (run 486), and the two hard failures on 20 August (runs 517 and 522).

Reading of it

There is a ~145s floor for 8.3 against a ~35s floor for 8.1/8.2, which points at a genuinely different install path rather than ambient network noise. The leading hypothesis is that 8.3 is the native PHP of the runner image's Ubuntu release, so setup-php installs it from apt/the ondrej PPA while 8.1 and 8.2 come down as prebuilt tarballs. On arm64 that apt path is slow and high-variance, and its tail sometimes crosses the step timeout. Unconfirmed — it needs checking from inside the runner image.

Worth trying

  • Confirm the path difference: run setup-php with SETUP_PHP_DEBUG / verbose output on an 8.3 job and see whether it is doing apt work.
  • If it is apt, an apt caching proxy or a local mirror reachable from the clusters would cut both the floor and the tail.
  • Independently: six jobs each install PHP 8.3 (lint, phpstan, tests 8.3, no-debug, build, release). Each is a separate draw on a flaky multi-minute operation. Consolidating lint and phpstan into one job would cut the exposure.
  • A retry around Setup PHP would paper over it if the above are not practical.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uw545F1vveNJKjzLxdi2ks

The `Setup PHP` step fails intermittently across CI and release workflows. When it fails it sits silent for ~12 minutes and then reports only `Could not setup PHP 8.3`, which reads as a hang. #175 guessed a GitHub API rate limit and added a 1Password-backed token; that was the wrong diagnosis and #177 reverts it. ## Evidence Every `Setup PHP` step across runs 454–523: | PHP | attempts | range | failures | |-----|----------|-------|----------| | 8.1 | 6 | 26–40s | 0 | | 8.2 | 6 | 29–41s | 0 | | 8.3 | 26 | 143–1273s | 2 | Median 8.3 setup by cluster: eris 382s, kallone 452s, nemesis 151s. 8.1/8.2 are ~35s on all three. So it is not cluster-specific — every cluster is slow on 8.3, and no cluster is slow on 8.1/8.2. The 8.3 penalty is not new: it was ~145s back on 30 July (run 454). What has grown is the tail — 879s and 1273s on 11 August (run 486), and the two hard failures on 20 August (runs 517 and 522). ## Reading of it There is a ~145s floor for 8.3 against a ~35s floor for 8.1/8.2, which points at a genuinely different install path rather than ambient network noise. The leading hypothesis is that 8.3 is the native PHP of the runner image's Ubuntu release, so setup-php installs it from apt/the ondrej PPA while 8.1 and 8.2 come down as prebuilt tarballs. On arm64 that apt path is slow and high-variance, and its tail sometimes crosses the step timeout. **Unconfirmed** — it needs checking from inside the runner image. ## Worth trying - Confirm the path difference: run setup-php with `SETUP_PHP_DEBUG` / verbose output on an 8.3 job and see whether it is doing apt work. - If it is apt, an apt caching proxy or a local mirror reachable from the clusters would cut both the floor and the tail. - Independently: six jobs each install PHP 8.3 (lint, phpstan, tests 8.3, no-debug, build, release). Each is a separate draw on a flaky multi-minute operation. Consolidating lint and phpstan into one job would cut the exposure. - A retry around `Setup PHP` would paper over it if the above are not practical. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Uw545F1vveNJKjzLxdi2ks
thatguygriff commented 2026-08-20 17:38:10 +00:00
Author
Owner
Copy Link
Copy Source

Root cause confirmed — and my hypothesis in the description was wrong

I guessed 8.3 was the native Ubuntu version and so went down an apt path while 8.1/8.2 came as tarballs. It is the other way round, and the real mechanism is in setup-php's source.

src/scripts/unix.sh:

export php_builder_versions="8.[3-9]"
...
[[ ( -z "$ImageOS" && -z "$ImageVersion" ) ||
   ( -n "$RUNNER_ENVIRONMENT" && "$RUNNER_ENVIRONMENT" = "self-hosted" ) ||
   -n "$ACT" || -n "$CONTAINER" ]] && _runner=self-hosted || _runner=github

ImageOS/ImageVersion are GitHub-hosted-image variables and are unset on our act_runner, so runner=self-hosted. Then in src/scripts/linux.sh:

add_php() {
  if [ "$runner" = "self-hosted" ] || [ "$use_package_cache" = "false" ]; then
    if [[ "$version" =~ ${php_builder_versions} || "$ts" = "zts" ]]; then
        setup_php_builder     # 8.3 and up
    else
        add_packaged_php      # 8.1, 8.2 — ondrej PPA, ~15 runtime packages

So the split is exactly 8.[3-9], which matches the timing data perfectly.

Why the builder path is minutes, not seconds

php-builder's install.sh does two things in parallel and waits on both:

if [ "$1" != "github" ]; then
  local_deps &          # <-- our case
else
  github_deps &         # 19 packages
fi
tar_file="php_$version$PHP_PKG_SUFFIX+$ID$VERSION_ID$ARCH_SUFFIX.tar.zst"
get -q "/tmp/$tar_file" "https://github.com/shivammathur/php-builder/releases/download/$version/$tar_file"

local_deps apt-installs ~70 -dev packages — firebird-dev, freetds-dev, libmagickwand-dev, libsmbclient-dev, libicu-dev, libgearman-dev, snmp, systemd, and so on — plus their transitive dependencies. The github_deps list used on GitHub-hosted runners is 19 packages. We take the 70-package branch, on arm64.

That is the ~145s floor, the long tail, and the occasional hard timeout. Those packages are build-time deps for compiling extensions from source; we compile nothing, we only need mbstring and intl, both already inside the tarball.

Run 526 shows it live: 8.1 in 57s and 8.2 in 1m, while 8.3 and 8.5 were still installing minutes later.

Note on #179

Adding PHP 8.5 to the matrix puts a second job on the builder path, since 8.5 also matches 8.[3-9]. Per run we now draw on it four times: tests 8.3, tests 8.5, the quality job, and build. Worth weighing against the coverage.

Can it be cached?

The tarball is the cheap half; the ~70 apt packages are the expensive half, and they land in root-owned system paths spread across /usr, /etc and /var/lib/dpkg — a bad fit for actions/cache. Ranked options:

  1. Prebake into the runner image (best). Once /usr/bin/php8.3 and /usr/bin/php-config8.3 exist, setup_php() skips add_php entirely and just runs switch_version. Baking in either the PHP versions themselves or merely local_deps' package list makes the apt half a no-op and leaves only the tarball fetch.
  2. Run the matrix in prebuilt PHP containers via container:, bypassing setup-php altogether. Docker layers cache locally on the runner after first pull.
  3. Cache /var/cache/apt/archives — the cheapest thing to try from this repo. Saves the package downloads but not dpkg unpack/configure. Worth measuring first, since it only helps if we are download-bound rather than CPU-bound.

Options 1 and 2 are infra-repo work. Option 3 could go in this repo as an experiment.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uw545F1vveNJKjzLxdi2ks

## Root cause confirmed — and my hypothesis in the description was wrong I guessed 8.3 was the *native* Ubuntu version and so went down an apt path while 8.1/8.2 came as tarballs. It is the other way round, and the real mechanism is in setup-php's source. **`src/scripts/unix.sh`:** ```bash export php_builder_versions="8.[3-9]" ... [[ ( -z "$ImageOS" && -z "$ImageVersion" ) || ( -n "$RUNNER_ENVIRONMENT" && "$RUNNER_ENVIRONMENT" = "self-hosted" ) || -n "$ACT" || -n "$CONTAINER" ]] && _runner=self-hosted || _runner=github ``` `ImageOS`/`ImageVersion` are GitHub-hosted-image variables and are unset on our act_runner, so **`runner=self-hosted`**. Then in `src/scripts/linux.sh`: ```bash add_php() { if [ "$runner" = "self-hosted" ] || [ "$use_package_cache" = "false" ]; then if [[ "$version" =~ ${php_builder_versions} || "$ts" = "zts" ]]; then setup_php_builder # 8.3 and up else add_packaged_php # 8.1, 8.2 — ondrej PPA, ~15 runtime packages ``` So the split is exactly `8.[3-9]`, which matches the timing data perfectly. ### Why the builder path is minutes, not seconds `php-builder`'s `install.sh` does two things in parallel and waits on both: ```bash if [ "$1" != "github" ]; then local_deps & # <-- our case else github_deps & # 19 packages fi tar_file="php_$version$PHP_PKG_SUFFIX+$ID$VERSION_ID$ARCH_SUFFIX.tar.zst" get -q "/tmp/$tar_file" "https://github.com/shivammathur/php-builder/releases/download/$version/$tar_file" ``` `local_deps` apt-installs **~70 `-dev` packages** — `firebird-dev`, `freetds-dev`, `libmagickwand-dev`, `libsmbclient-dev`, `libicu-dev`, `libgearman-dev`, `snmp`, `systemd`, and so on — plus their transitive dependencies. The `github_deps` list used on GitHub-hosted runners is 19 packages. We take the 70-package branch, on arm64. That is the ~145s floor, the long tail, and the occasional hard timeout. Those packages are build-time deps for compiling extensions from source; we compile nothing, we only need mbstring and intl, both already inside the tarball. Run 526 shows it live: 8.1 in 57s and 8.2 in 1m, while 8.3 and 8.5 were still installing minutes later. ### Note on #179 Adding PHP 8.5 to the matrix puts a **second** job on the builder path, since `8.5` also matches `8.[3-9]`. Per run we now draw on it four times: tests 8.3, tests 8.5, the quality job, and build. Worth weighing against the coverage. ### Can it be cached? The tarball is the cheap half; the ~70 apt packages are the expensive half, and they land in root-owned system paths spread across `/usr`, `/etc` and `/var/lib/dpkg` — a bad fit for `actions/cache`. Ranked options: 1. **Prebake into the runner image** (best). Once `/usr/bin/php8.3` and `/usr/bin/php-config8.3` exist, `setup_php()` skips `add_php` entirely and just runs `switch_version`. Baking in either the PHP versions themselves or merely `local_deps`' package list makes the apt half a no-op and leaves only the tarball fetch. 2. **Run the matrix in prebuilt PHP containers** via `container:`, bypassing setup-php altogether. Docker layers cache locally on the runner after first pull. 3. **Cache `/var/cache/apt/archives`** — the cheapest thing to try from this repo. Saves the package *downloads* but not dpkg unpack/configure. Worth measuring first, since it only helps if we are download-bound rather than CPU-bound. Options 1 and 2 are infra-repo work. Option 3 could go in this repo as an experiment. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Uw545F1vveNJKjzLxdi2ks
thatguygriff commented 2026-08-20 18:00:27 +00:00
Author
Owner
Copy Link
Copy Source

Option 3 measured: it cannot work, because the cache service is switched off

Tried it on #179 (run 527) and reverted it. Every actions/cache step on these runners prints:

[warning]Cache action is only supported on GHES version >= 3.5. If you are on version >= 3.5
Please check with GHES admin if Actions cache service is enabled or not.

and then no-ops. Two independent confirmations:

  • The save step completes in 0.19–0.31s across all five jobs. A few hundred megabytes of .debs is not going anywhere in 0.2s.
  • setup-php timings were unchanged against the run 526 baseline, within normal variance — 8.3 went 454s → 148s and 8.5 733s → 446s, which is noise, not signal.

The Composer cache has never worked either

The same warning appears on the Cache Composer packages step this workflow has carried all along — it is there in run 523 and earlier. So that step has been decorative for its entire life. Nothing is broken by it, but nobody is getting the speedup it implies.

Fix is in act_runner's config.yaml, not this repo:

cache:
  enabled: true
  dir: ""
  host: ""   # must be an address reachable from *inside* the job container,
  port: 0    # not 127.0.0.1 — this is the usual reason it stays broken

Once that is on, the apt-cache experiment is worth re-applying — it is in the history of #179 as commit 85c7a01. It should be a large win, since the build tarball is only 19MB and the ~70 apt packages are essentially the entire cost.

Containers: what would actually work

Checked what we really need. composer check-platform-reqs across the whole dependency tree wants only:

ctype, curl, dom, filter, json, libxml, mbstring, phar, simplexml, tokenizer, xml, xmlreader, xmlwriter

Every one is built into the official php:X-cli images, so no docker-php-ext-install is needed at all. Also worth noting: the workflow asks setup-php for intl, and nothing in the tree requires it — composer why ext-intl finds no dependent. All of php:{8.1,8.2,8.3,8.4,8.5}-cli publish arm64.

The catch is that a plain php:X-cli will not work directly. act invokes a bare node inside the job container:

docker exec cmd=[node /var/run/act/actions/.../dist/index.js]

so actions/checkout and actions/cache need Node present in the image, and the official PHP images do not ship it. The practical form is a small custom image per version, pushed to Gitea's own container registry so pulls stay in-cluster:

ARG PHP_VERSION=8.3
FROM php:${PHP_VERSION}-cli

RUN apt-get update \
 && apt-get install -y --no-install-recommends git unzip ca-certificates curl \
 && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
 && apt-get install -y --no-install-recommends nodejs \
 && rm -rf /var/lib/apt/lists/*

COPY --from=composer:2 /usr/bin/composer /usr/bin/composer

and then the whole Setup PHP step disappears:

  test:
    runs-on: ubuntu-latest
    container: git.unsupervised.ca/unsupervised/ci-php:${{ matrix.php }}
    steps:
      - uses: actions/checkout@v4
      - run: composer install --prefer-dist --no-progress --no-interaction
      - run: composer test

That removes setup-php, php-builder, and the ~70 apt packages in one go, and turns install time into a layer pull that is cached on the runner host after the first use. It is the most complete fix of the three, at the cost of maintaining four small images.

Recommendation

Enabling the runner cache service is the smallest change with the widest benefit — it fixes the Composer cache immediately and unlocks the apt cache. The container images are the durable fix if we want Setup PHP gone entirely.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uw545F1vveNJKjzLxdi2ks

## Option 3 measured: it cannot work, because the cache service is switched off Tried it on #179 (run 527) and reverted it. Every `actions/cache` step on these runners prints: ``` [warning]Cache action is only supported on GHES version >= 3.5. If you are on version >= 3.5 Please check with GHES admin if Actions cache service is enabled or not. ``` and then no-ops. Two independent confirmations: - The save step completes in **0.19–0.31s** across all five jobs. A few hundred megabytes of `.debs` is not going anywhere in 0.2s. - setup-php timings were unchanged against the run 526 baseline, within normal variance — 8.3 went 454s → 148s and 8.5 733s → 446s, which is noise, not signal. ### The Composer cache has never worked either The same warning appears on the `Cache Composer packages` step this workflow has carried all along — it is there in run 523 and earlier. So that step has been decorative for its entire life. Nothing is broken by it, but nobody is getting the speedup it implies. **Fix is in act_runner's `config.yaml`, not this repo:** ```yaml cache: enabled: true dir: "" host: "" # must be an address reachable from *inside* the job container, port: 0 # not 127.0.0.1 — this is the usual reason it stays broken ``` Once that is on, the apt-cache experiment is worth re-applying — it is in the history of #179 as commit `85c7a01`. It should be a large win, since the build tarball is only 19MB and the ~70 apt packages are essentially the entire cost. ## Containers: what would actually work Checked what we really need. `composer check-platform-reqs` across the whole dependency tree wants only: > ctype, curl, dom, filter, json, libxml, mbstring, phar, simplexml, tokenizer, xml, xmlreader, xmlwriter Every one is built into the official `php:X-cli` images, so **no `docker-php-ext-install` is needed at all**. Also worth noting: the workflow asks setup-php for `intl`, and nothing in the tree requires it — `composer why ext-intl` finds no dependent. All of `php:{8.1,8.2,8.3,8.4,8.5}-cli` publish arm64. The catch is that a plain `php:X-cli` will **not** work directly. act invokes a bare `node` inside the job container: ``` docker exec cmd=[node /var/run/act/actions/.../dist/index.js] ``` so `actions/checkout` and `actions/cache` need Node present in the image, and the official PHP images do not ship it. The practical form is a small custom image per version, pushed to Gitea's own container registry so pulls stay in-cluster: ```dockerfile ARG PHP_VERSION=8.3 FROM php:${PHP_VERSION}-cli RUN apt-get update \ && apt-get install -y --no-install-recommends git unzip ca-certificates curl \ && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \ && apt-get install -y --no-install-recommends nodejs \ && rm -rf /var/lib/apt/lists/* COPY --from=composer:2 /usr/bin/composer /usr/bin/composer ``` and then the whole `Setup PHP` step disappears: ```yaml test: runs-on: ubuntu-latest container: git.unsupervised.ca/unsupervised/ci-php:${{ matrix.php }} steps: - uses: actions/checkout@v4 - run: composer install --prefer-dist --no-progress --no-interaction - run: composer test ``` That removes setup-php, php-builder, and the ~70 apt packages in one go, and turns install time into a layer pull that is cached on the runner host after the first use. It is the most complete fix of the three, at the cost of maintaining four small images. ## Recommendation Enabling the runner cache service is the smallest change with the widest benefit — it fixes the Composer cache immediately and unlocks the apt cache. The container images are the durable fix if we want `Setup PHP` gone entirely. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Uw545F1vveNJKjzLxdi2ks
thatguygriff commented 2026-08-20 18:08:46 +00:00
Author
Owner
Copy Link
Copy Source

Correction, and a test that narrows it down

Good challenge on the warning — it is actions/cache's own GHES gate, not proof the runner lacks a cache server. Those are different things and I conflated them. Researched and tested; here is where it lands.

What the action does

Both v3 and v4 gate on isGhes(), which reads GITHUB_SERVER_URL and treats anything that is not github.com as GitHub Enterprise (v4 also permits *.ghe.com and *.localhost). Our server is git.unsupervised.ca, so the check trips and the action returns before touching any cache. That alone explains the 0.2s save, with no runner setting involved.

What Gitea says should happen

Per the Runner 3.0.0 release notes, every runner starts its own cache server, and the runner "patches action bundles at load time to open the GHES gate and read the cache endpoint from ACTIONS_CACHE_URL", with actions/cache supported unforked. Our runners report v3.0.0, so this should already be happening.

The test

Bumped every actions/cache to v4 and re-ran (run 529). It resolved correctly — Download action repository 'actions/cache@v4' (SHA:0057852bfaa89a56745cba8c7296529d2fc39830) — and printed the same GHES warning. So the action version is not what is closing the gate, and my "just upgrade the action" theory is dead too.

Where that leaves it

The caching docs note the runner only rewrites bundles it recognises, and "a bundle it does not recognise is left alone". Neither v3 nor v4 is being patched here. The explanation that fits everything is that cache is off in the runner config: with cache.enabled false there is no ACTIONS_CACHE_URL to point a patched bundle at, so no patching happens and the action falls back to its own gate.

I cannot read the runner config from here, so that last step is inference from observed behaviour rather than something I have confirmed. The alternative — that caching is on but the patcher does not match these particular bundles — would be a runner bug worth reporting upstream. Either way the next move is on the runner host, not in this repository.

cache:
  enabled: true
  dir: ""
  host: ""   # address reachable from inside the job container; not 0.0.0.0,
  port: 0    # not 127.0.0.1. Empty auto-detects.

Quickest way to tell the two apart: run a job with a step that echoes $ACTIONS_CACHE_URL. Empty means cache is off in config; populated means the patcher is the problem.

Once a cache is actually served, both halves should land together — the apt cache and the move to v4 — since v4.2+ is what speaks the cache service v2 API. Both are in #179's history as 85c7a01 and d5eb276.

Sources: Runner 3.0.0 notes · Gitea caching docs · Enable Gitea Actions Cache tutorial

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uw545F1vveNJKjzLxdi2ks

## Correction, and a test that narrows it down Good challenge on the warning — it is `actions/cache`'s own GHES gate, not proof the runner lacks a cache server. Those are different things and I conflated them. Researched and tested; here is where it lands. ### What the action does Both v3 and v4 gate on `isGhes()`, which reads `GITHUB_SERVER_URL` and treats anything that is not github.com as GitHub Enterprise (v4 also permits `*.ghe.com` and `*.localhost`). Our server is `git.unsupervised.ca`, so the check trips and the action returns before touching any cache. That alone explains the 0.2s save, with no runner setting involved. ### What Gitea says should happen Per the [Runner 3.0.0 release notes](https://blog.gitea.com/release-of-runner-3.0.0/), every runner starts its own cache server, and the runner *"patches action bundles at load time to open the GHES gate and read the cache endpoint from `ACTIONS_CACHE_URL`"*, with `actions/cache` supported unforked. Our runners report **v3.0.0**, so this should already be happening. ### The test Bumped every `actions/cache` to v4 and re-ran (run 529). It resolved correctly — `Download action repository 'actions/cache@v4' (SHA:0057852bfaa89a56745cba8c7296529d2fc39830)` — and printed **the same GHES warning**. So the action version is not what is closing the gate, and my "just upgrade the action" theory is dead too. ### Where that leaves it The [caching docs](https://docs.gitea.cn/en-us/runner/develop/cache/) note the runner only rewrites bundles it recognises, and *"a bundle it does not recognise is left alone"*. Neither v3 nor v4 is being patched here. The explanation that fits everything is that **cache is off in the runner config**: with `cache.enabled` false there is no `ACTIONS_CACHE_URL` to point a patched bundle at, so no patching happens and the action falls back to its own gate. I cannot read the runner config from here, so that last step is inference from observed behaviour rather than something I have confirmed. The alternative — that caching is on but the patcher does not match these particular bundles — would be a runner bug worth reporting upstream. Either way the next move is on the runner host, not in this repository. ```yaml cache: enabled: true dir: "" host: "" # address reachable from inside the job container; not 0.0.0.0, port: 0 # not 127.0.0.1. Empty auto-detects. ``` Quickest way to tell the two apart: run a job with a step that echoes `$ACTIONS_CACHE_URL`. Empty means cache is off in config; populated means the patcher is the problem. Once a cache is actually served, both halves should land together — the apt cache and the move to v4 — since v4.2+ is what speaks the cache service v2 API. Both are in #179's history as `85c7a01` and `d5eb276`. Sources: [Runner 3.0.0 notes](https://blog.gitea.com/release-of-runner-3.0.0/) · [Gitea caching docs](https://docs.gitea.cn/en-us/runner/develop/cache/) · [Enable Gitea Actions Cache tutorial](https://about.gitea.com/resources/tutorials/enable-gitea-actions-cache-to-accelerate-cicd/) 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Uw545F1vveNJKjzLxdi2ks
thatguygriff commented 2026-08-20 18:19:33 +00:00
Author
Owner
Copy Link
Copy Source

Probe results — settled

Ran the probe (#180, run 533). It answered the question and killed a tempting wrong turn on the way.

What the runner exports

GITHUB_SERVER_URL        = http://gitea-unsupervised-git-http.gitea.svc.cluster.local:3000
ACTIONS_CACHE_URL        = <unset>
ACTIONS_RESULTS_URL      = http://gitea-unsupervised-git-http.gitea.svc.cluster.local:3000
ACTIONS_CACHE_SERVICE_V2 = <unset>
ACTIONS_RUNTIME_TOKEN: set

ACTIONS_CACHE_URL is unset, so act_runner's built-in cache server is not running. That is hypothesis 1, confirmed. ACTIONS_RESULTS_URL is set, but read on — it is not what it looks like.

The tempting wrong turn

ACTIONS_RESULTS_URL is the v2 endpoint, and actions/cache v4.2+ can speak v2 — gated behind ACTIONS_CACHE_SERVICE_V2, which the runner does not set. So I forced it. Three variants:

variant GHES gate result
cache@v4 + ACTIONS_CACHE_SERVICE_V2=true open Cache not found, then Failed to save: Unable to reserve cache with key ...
cache@v6 (v6.1.0) closed GHES warning, no-op
cache@v6 + ACTIONS_CACHE_SERVICE_V2=true open same reserve failure

So the flag genuinely opens the gate and the action starts talking to the endpoint. It just does not work, and go-gitea/gitea#33393 says why: the v2 cache service is not implemented on the Gitea side. ACTIONS_RESULTS_URL points at the artifacts service. Reads answer "not found" because nothing is there, and reserves fail because there is no cache service behind it.

Do not set ACTIONS_CACHE_SERVICE_V2. It converts a loud no-op into a silent one: the warning disappears, every cache step fails to save, and it looks like caching works. Strictly worse than today.

Also worth recording: actions/cache@v6 alone does not help — the gate is version-independent. Only the flag moves it, and the flag leads somewhere broken.

The actual fix

Enable act_runner's own cache server on each of eris, kallone and nemesis. That is the supported path per the same upstream issue, and it sets ACTIONS_CACHE_URL, which is what the runner's bundle patcher needs in order to open the GHES gate for real.

cache:
  enabled: true
  dir: ""
  host: ""   # address reachable from inside the job container.
  port: 0    # Not 0.0.0.0, not 127.0.0.1. Empty auto-detects.

The host line is the one that usually gets this wrong — the runner must advertise an address the job container can dial, not one that only works on the runner host.

Once ACTIONS_CACHE_URL shows up in a probe, re-apply both halves together: the apt cache (85c7a01) and the move off actions/cache@v3 (d5eb276). Given the build tarball is only 19MB and the ~70 apt packages are the entire cost, that should take the 8.3/8.5 floor from ~145s to near nothing.

Status

  • #179 stays clean: consolidation + 8.5 only.
  • #180 was diagnostic; closing it and deleting the branch. The probe workflow is in its history if it is ever needed again.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uw545F1vveNJKjzLxdi2ks

## Probe results — settled Ran the probe (#180, run 533). It answered the question and killed a tempting wrong turn on the way. ### What the runner exports ``` GITHUB_SERVER_URL = http://gitea-unsupervised-git-http.gitea.svc.cluster.local:3000 ACTIONS_CACHE_URL = <unset> ACTIONS_RESULTS_URL = http://gitea-unsupervised-git-http.gitea.svc.cluster.local:3000 ACTIONS_CACHE_SERVICE_V2 = <unset> ACTIONS_RUNTIME_TOKEN: set ``` **`ACTIONS_CACHE_URL` is unset**, so act_runner's built-in cache server is not running. That is hypothesis 1, confirmed. `ACTIONS_RESULTS_URL` is set, but read on — it is not what it looks like. ### The tempting wrong turn `ACTIONS_RESULTS_URL` is the v2 endpoint, and actions/cache v4.2+ can speak v2 — gated behind `ACTIONS_CACHE_SERVICE_V2`, which the runner does not set. So I forced it. Three variants: | variant | GHES gate | result | |---|---|---| | `cache@v4` + `ACTIONS_CACHE_SERVICE_V2=true` | **open** | `Cache not found`, then `Failed to save: Unable to reserve cache with key ...` | | `cache@v6` (v6.1.0) | closed | GHES warning, no-op | | `cache@v6` + `ACTIONS_CACHE_SERVICE_V2=true` | **open** | same reserve failure | So the flag genuinely opens the gate and the action starts talking to the endpoint. It just does not work, and [go-gitea/gitea#33393](https://github.com/go-gitea/gitea/issues/33393) says why: **the v2 cache service is not implemented on the Gitea side.** `ACTIONS_RESULTS_URL` points at the *artifacts* service. Reads answer "not found" because nothing is there, and reserves fail because there is no cache service behind it. **Do not set `ACTIONS_CACHE_SERVICE_V2`.** It converts a loud no-op into a silent one: the warning disappears, every cache step fails to save, and it looks like caching works. Strictly worse than today. Also worth recording: `actions/cache@v6` alone does *not* help — the gate is version-independent. Only the flag moves it, and the flag leads somewhere broken. ### The actual fix Enable act_runner's own cache server on each of eris, kallone and nemesis. That is the supported path per the same upstream issue, and it sets `ACTIONS_CACHE_URL`, which is what the runner's bundle patcher needs in order to open the GHES gate for real. ```yaml cache: enabled: true dir: "" host: "" # address reachable from inside the job container. port: 0 # Not 0.0.0.0, not 127.0.0.1. Empty auto-detects. ``` The `host` line is the one that usually gets this wrong — the runner must advertise an address the *job container* can dial, not one that only works on the runner host. Once `ACTIONS_CACHE_URL` shows up in a probe, re-apply both halves together: the apt cache (`85c7a01`) and the move off `actions/cache@v3` (`d5eb276`). Given the build tarball is only 19MB and the ~70 apt packages are the entire cost, that should take the 8.3/8.5 floor from ~145s to near nothing. ### Status - #179 stays clean: consolidation + 8.5 only. - #180 was diagnostic; closing it and deleting the branch. The probe workflow is in its history if it is ever needed again. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Uw545F1vveNJKjzLxdi2ks
thatguygriff commented 2026-08-20 19:20:54 +00:00
Author
Owner
Copy Link
Copy Source

Resolved in #181

Resolved in #181
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something isn't working
 enhancement
Extends an existing feature
 feature
New user-facing capability
 payments
Billing, Stripe, receipts, reporting
 security
Security vulnerability or hardening
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
Kydoimos (Kydoimos) thatguygriff (James Griffin-Allwood)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Unsupervised/unsupervised-scheduler#178