Prevent S6 startup races: services wait for their config oneshots and depend on base

[LorenzoRogai]

Note

Note from @jaydrogers: I took over this PR to finish it and rewrote this description to match our PR format. The scope grew beyond the original patch (see my comment below for why). @LorenzoRogai's original description is preserved at the bottom of this comment.

Why we created this PR

When a container built from our S6 images runs as root (for example, following the advice in #425), php-fpm sometimes fails on its first start:

ALERT: [pool www] user has not been defined
ERROR: failed to post process the configuration
ERROR: FPM initialization failed

S6 restarts it and the container recovers, so it looks like noise. It's a race. PHP-FPM refuses to start as root without a user in the pool config. Our 5-fpm-pool-user oneshot writes that line, but php-fpm never declared that it depends on the oneshot, so S6 started both at once. On the same boot, nginx and apache2 could also start before 10-init-webserver-config rendered their config.

Rootless containers never hit it because PHP-FPM ignores the user directive when it isn't root. On main, the error reproduces on every root-mode boot.

While fixing the dependency, we found two more things that S6 Overlay documents and we weren't doing:

1. We used the deprecated flat dependencies file. The s6-overlay README only documents dependencies.d/ directories, and s6-rc marks the flat file deprecated. With the directory format, a dependency is one empty file, so the whole fix is a touch.
2. None of our services depended on base. The README is direct about it: services that don't depend on base "might have been started earlier, which may cause race conditions - so it's recommended to always make them depend on base."

How to test

serversideup/php-dev:689-*

View the testing images →

To see the race yourself, build this against serversideup/php:8.4-fpm-nginx-alpine (fails) and against a 689- image (passes):

FROM serversideup/php:8.4-fpm-nginx-alpine
USER root
RUN docker-php-serversideup-s6-init

cid=$(docker run -d --rm my-test-image); sleep 6
docker logs $cid 2>&1 | grep -c 'user has not been defined'   # 1 before, 0 after
docker stop $cid

What this PR does

Startup order

• php-fpm now waits for 5-fpm-pool-user; nginx and apache2 wait for 10-init-webserver-config and php-fpm
• Every service and every generated oneshot now depends on base, as S6 Overlay recommends

Before (main): php-fpm, nginx, and apache2 start the moment S6 is up, in parallel with the oneshots that write their config.

flowchart LR
    subgraph oneshots [Entrypoint oneshots]
        a[0-container-info] --> b[1-log-output-level]
        b --> c[5-fpm-pool-user]
        b --> d[10-init-webserver-config]
        d --> e[50-laravel-automations]
    end
    subgraph services [Long-running services]
        fpm[php-fpm] --> web[nginx / apache2]
    end
    c -. race .- fpm
    d -. race .- web

Loading

After: every service and oneshot depends on base, and each service waits for the oneshot that configures it.

flowchart LR
    base --> a[0-container-info] --> b[1-log-output-level]
    b --> c[5-fpm-pool-user] --> fpm[php-fpm]
    b --> d[10-init-webserver-config] --> e[50-laravel-automations]
    d --> web[nginx / apache2]
    fpm --> web

Loading

Every node above also lists base directly in its dependencies.d/; those edges are omitted to keep the diagram readable.

S6 Overlay alignment

• Replaced the flat dependencies files for php-fpm, nginx, and apache2 with dependencies.d/ directories
• docker-php-serversideup-s6-init now has one add_dependency helper (a mkdir -p and a touch) used for oneshot chaining and for the service dependencies above

Docs

• Added dependencies.d and base guidance to Adding your own start-up scripts

Compatibility

Stock images and rootless containers behave the same. If you appended lines to our old flat dependencies files in your own Dockerfile, S6 now ignores those files once dependencies.d/ exists. Move each line to an empty file in dependencies.d/ instead.

---

Original description by @LorenzoRogai

Problem

When a container built on the s6 images runs as root, php-fpm and the web server (nginx/apache2) are brought up in parallel with the entrypoint oneshots that configure them — the long-running services declare no dependency on those oneshots.

As root this is a race:

• php-fpm reads its pool config before 5-fpm-pool-user appends user/group, failing with:

  ALERT: [pool www] user has not been defined
  ERROR: failed to post process the configuration
  ERROR: FPM initialization failed
  

• the web server starts before 10-init-webserver-config renders its config, e.g. nginx: [emerg] open() "/etc/nginx/nginx.conf" failed (2: No such file or directory).

s6 restarts the crashed longruns, so the container eventually recovers — which is exactly why this is intermittent and hard to reproduce (see #425). But it emits alarming errors on every boot that loses the race, slows startup, and leaves a brief window with no service.

Fix

docker-php-serversideup-s6-init now adds a dependency from each web-facing longrun to the entrypoint oneshot that configures it, after the oneshots are created:

add_startup_dependency php-fpm 5-fpm-pool-user
add_startup_dependency nginx 10-init-webserver-config
add_startup_dependency apache2 10-init-webserver-config

The entrypoint oneshots are chained in alphabetical order, so depending on one transitively waits for all earlier ones (php-fpm → 5-fpm-pool-user; the web servers → 10-init-webserver-config, which already sits after 5-*). Each dependency is added only when both the service and the oneshot exist, so cli/fpm/frankenphp images — and images where a user removes a script — are unaffected. It does not make services wait on later app oneshots such as 50-laravel-automations, so a failing app hook won't block the web server from starting.

Testing

• Reproduced as root on 8.4-fpm-nginx-alpine: first boot shows the fpm + nginx errors, then self-heals.
• With the fix: 3/3 clean boots — no errors, php-fpm/nginx reach "ready to handle connections" on the first attempt.
• Rootless (default) behavior is unchanged (5-fpm-pool-user is a no-op when not root; the dependency just orders startup).

Refs #425

[jaydrogers]

Thanks @LorenzoRogai and @mbrodala! We definitely had a problem here and I want to thank you both for helping me identify and resolve this.

I noticed a few other things so I made a bigger refactor (289b4b0) of this process and gave you both as credit. Here's why:

Why the format changed

While verifying the fix I checked the s6-overlay README and the s6-rc docs. The flat dependencies file we've been using is deprecated. S6 only documents dependencies.d/ directories, where each dependency is an empty file. Once I switched to that, the grep, the newline handling, and the dedupe check all disappeared, because touch is already idempotent. That's the same simplification Matthias was pushing toward, taken one step further. The touch suggestion is in; the printf and long-flag grep suggestions became moot because those lines no longer exist.

Why base was added.

The README is explicit that services which don't depend on base "might have been started earlier, which may cause race conditions." Since this PR is about a startup race, it felt wrong to fix one ordering gap and leave the documented one open.

Verification.

On main, the FPM error reproduced on 3 of 3 root-mode boots. With this branch it's 0 of 3 as root and 0 of 3 rootless, s6-rc-compile accepts the tree, and shellcheck is clean.

One small correction to the original write-up: 10-init-webserver-config does not run after 5-fpm-pool-user. They run in parallel because the init script processes 10-* before 5-* in glob order. The fix still holds because nginx and apache2 depend on php-fpm, which depends on 5-fpm-pool-user.

Thanks again for working together on this!

I made a bigger refactor and explained why here #689 (comment)

[LorenzoRogai]

@jaydrogers thanks for the update! Will this require any changes on our side if we’re using custom s6 scripts?

[jaydrogers]

Yes, I ahve that added here: #645

Let me know if you have any other questions. Thanks again!