feat: declared background workers + frankenphp_get_worker_handle()

[nicolas-grekas]

Background workers run a script in a loop outside the HTTP request cycle, sharing the PHP runtime with the request threads. This is the smallest useful slice of #2398, rebuilt on the Server of #2499: the parallel Scope machinery is gone, a background worker attaches to a php_server through WithWorkerServerScope() like any other worker.

Declared with background in a worker block (php_server or global) or WithWorkerBackground() in Go. name is required, it is the script's identity; match is rejected; num >= 1, no lazy start here. $_SERVER['FRANKENPHP_WORKER'] now carries the name for every worker, HTTP ones included (the documented contract is to test its presence, not its value, as suggested on #2393), and $_SERVER['FRANKENPHP_WORKER_BACKGROUND'] is set in background workers so a script serving both roles can tell them apart with isset(). The lifecycle mirrors HTTP workers: re-run on a cooperative exit, restart with a capped quadratic backoff on a crash, max_consecutive_failures fails Init() during startup only. A crash past the ready point is paced by that backoff but never counted toward the cap, which is about a script that never boots: unlike an HTTP worker, paced by the traffic it needs before it can crash, a background worker reaches its ready point on its own and would otherwise spin. drain() now runs on shutdown, reboot and handler transitions, so a parked script wakes up instead of waiting out the force-kill grace period.

The script gets one handle, frankenphp_get_worker_handle(): resource, a stream that reaches EOF when the worker is drained. It is meant to carry control messages later, hence one handle rather than one per purpose. It is backed by a socket pair, not a pipe: on Windows PHP's php_select() only waits properly on sockets before 8.5, and the socket path is version-independent. Streams don't own the socket (php_sockop_close() would shutdown() it on Windows), so a run gets one stream, and closing it then fetching again yields a fresh one over the same socket without losing the drain signal; the read timeout is infinite so a blocking read parks as well as stream_select() does.

A worker counts as ready on its first wait on the handle (select cast or read), the background analog of frankenphp_handle_request(): Init() waits for it, ready_workers counts from it, and an exit before it is a boot failure. Fetching the handle is not the ready point, nothing forces a script to fetch it after bootstrapping.

Worker names are now scoped like paths: unique within a php_server or among global workers, so two blocks may each declare queue. The script sees the declared name; metrics and logs report a scoped worker as <server name>:<name>, with a numeric suffix on server names when two blocks resolve to the same one. The collision-driven renaming in the Caddy module is gone.

Deferred: lazy start (frankenphp_ensure_background_worker()), catch-all workers, shared-state APIs, and the orchestrator-style runtime API discussed in #2398.

Supersedes #2543 and #2398.

[henderkes]

Please rewrite the PR description to not be LLM slop reasoning with itself about what it did and why. I've tried reading this three times and I just can't.

[nicolas-grekas]

Sure, I'll let you know when I'm done, for now I just let it do the rebase 😅

[henderkes]

What happens here when a global background worker and a php_server scoped background worker share the same name and are both eligible for the same source file?

[henderkes]

found another one, anyway, have you tested this on windows?

[Copilot]

Pull request overview

Adds declared background PHP workers with graceful stop-stream handling and Caddy configuration support.

Changes:

• Adds background-worker lifecycle, validation, and thread allocation.
• Exposes frankenphp_get_worker_handle().
• Adds Caddy integration, documentation, fixtures, and tests.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
worker.go	Registers and validates background workers.
threadbackgroundworker.go	Implements background-worker lifecycle.
requestoptions.go	Rejects background workers for HTTP requests.
phpthread.go	Drains handlers during shutdown and transitions.
phpmainthread.go	Drains handlers during reboot.
options.go	Adds WithWorkerBackground().
frankenphp.go	Reserves background-worker threads.
frankenphp.c	Implements stop pipes and PHP API.
frankenphp.h	Declares C primitives.
frankenphp.stub.php	Declares the PHP function.
frankenphp_arginfo.h	Registers generated arginfo.
docs/config.md	Documents background configuration.
caddy/workerconfig.go	Parses background worker blocks.
caddy/config_test.go	Tests Caddy parsing and validation.
bgworker_test.go	Tests lifecycle, restart, scope, and validation.
testdata/bgworker/basic.php	Provides lifecycle fixture.
testdata/bgworker/crash.php	Provides restart fixture.
testdata/bgworker/early-return.php	Provides startup-failure fixture.
testdata/bgworker/named.php	Provides named-worker fixture.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[AlliBalliBaba]

Hmm I don't remember the specific reason why background workers are treated separately here. Wouldn't it make sense to just do this and count it into the general pool, like other workers:

if w.num <= 0 {
    if w.isBackgroundWorker {
		opt.workers[i].num = 1
    } else {
		opt.workers[i].num = maxProcs
    }
}

Worker thread count is already added on top of the general thread count. It would only overflow in case someone sets a general cap on global threads, in which case it should probably still honor that cap.

[nicolas-grekas]

The num_threads / max_threads budget exists for autoscaling HTTP workers, and background workers take no part in that: they don't scale, don't queue requests and never compete for a free thread. Counting them into the pool would change what the budget means depending on how many background workers a config declares: declare five and you silently get five fewer HTTP threads, so people would have to bump the budget just to keep the capacity they had, and the setting stops describing HTTP capacity. That's why they're reserved on top: the HTTP admission math is untouched and the totals are bumped afterwards (reservedThreads). Requiring an explicit num keeps that reservation visible in the config rather than defaulted.

[alexandre-daubois]

The reservation is right for num_threads, but max_threads auto is memory-derived and then floored back up to numThreads, so background workers silently push past that ceiling. Failing Init() on the total, instead of bumping max_threads, would keep it.

[nicolas-grekas]

auto is a memory heuristic that num_threads already overrides through the same floor; background threads are fixed threads and get the same treatment. Failing Init() would let a heuristic reject an explicit config, so they stay on top.

[AlliBalliBaba]

IMO it's a bit unfortunate that the ceiling exists in the first place, it would be better to have something like num_regular_threads and max_regular_threads, so people don't have to do maths with worker thread count.

But with the current logic all workers count toward that ceiling (they take away threads from the regular threads), so I think it makes more sense to be consistent when it comes to background workers, or we'll just make it even more confusing.

[nicolas-grekas]

Agreed on num_regular_threads / max_regular_threads being the shape that avoids the maths; that would be a separate change to the existing settings.

On consistency: HTTP workers count against the ceiling because they draw from the same pool, autoscale into it and compete with regular threads for it. Background threads never enter that pool: fixed count, no scaling, no requests. Counting them would make num_threads mean a different HTTP capacity depending on how many background workers a config declares.

What the review did change is the plumbing. calculateMaxThreads() used to bump the totals and subtract them back later; it now resolves num_threads / max_threads against the HTTP workers alone and returns the background threads separately, for Init() to add where a real total is needed. No addition-then-subtraction anywhere.

[AlliBalliBaba]

It looks like currently the handle is only used for shutdown. IIRC in the future you'd also want to use the handle to send messages or even requests.

Would it maybe be cleaner to have a separate handle for each? Makes the api look more like we're selecting over different channels, in other words:

frankenphp_get_shutdown_handle(); # instead of frankenphp_get_worker_handle
frankenphp_get_message_handle(); # future scope: can return a dedicated message
frankenphp_get_request_handle(); # future scope: can return a dedicated request object

[nicolas-grekas]

I'd rather keep one handle. In the prototype built on this primitive, shutdown, messages and requests all arrive on the same stream as typed messages, and the worker loop is a single stream_select() plus a dispatch on what was read; that worked well in practice. One handle per kind means selecting over N streams, N functions to document and keep in sync, and ordering questions between them (a message landing after shutdown was signalled on another stream). Fewer functions is also less API to get wrong. This PR only uses the EOF-on-drain part, but the handle is meant to carry the rest.

[AlliBalliBaba]

Hmm I think you're right since streams can only send strings.

What do you think about something like this? Abstracting things a bit allows us to do more in the future without BC breaks.

$worker = new \FrankenPHP\Worker(
    onMessage: fn(\FrankenPHP\Message $message) => ...,
    onRequest: fn(\FrankenPHP\Request $request) => ...,
    onShutdown: fn() => ...
);

$handle = $worker->getHandle();

while ($message = fgets($handle)) { # or the equivalent with stream_select
  $worker->handle($message);
}

The message can literally be "1", "2", "3", it will be handled internally and forwarded to onMessage() or onShutdown()

[nicolas-grekas]

That's pure PHP over the primitive, so it can ship as a package or a docs example and evolve without a FrankenPHP release; in the engine it freezes the callback signatures and the Message / Request shapes before anything uses them. A class can be added later, not removed.

BC-wise the primitive is the smaller surface: "lines, then EOF on drain", where a new kind of message is a new prefix. Three callback signatures and two classes are more to keep stable, not less.

Callbacks also take ownership of the loop, so anything a handler waits on has to be routed back through the dispatcher. The stream composes with Revolt, amphp, ReactPHP or a plain blocking read, none of which have to know about each other.

The tasks of #2636 are the first real message type here and they wanted functions: the loop drains the queue on each wake-up, since a line is a wake-up and not a count, and inside a task the worker does a stream_select() on the task's own stream to notice the sender giving up. A dispatcher handing out one message at a time makes both awkward. frankenphp_handle_request() is callback-shaped because a request has a beginning and an end; a background worker's loop owns the process lifetime.

Hand-rolled dispatch being easy to get wrong is fair, so I'd answer it with a documented loop, and a package if it earns its place.

[AlliBalliBaba]

The tasks of #2636 are the first real message type here and they wanted functions: the loop drains the queue on each wake-up, since a line is a wake-up and not a count, and inside a task the worker does a stream_select() on the task's own stream to notice the sender giving up. A dispatcher handing out one message at a time makes both awkward. frankenphp_handle_request() is callback-shaped because a request has a beginning and an end; a background worker's loop owns the process lifetime.

That's kind of the point, we're locking ourselves out of any changes/extensions to the api by requiring very specific steps to be followed (receiving literal "task" -> checking frankenphp_receive_task() -> receiving a stream -> passing the stream to frankenphp_update_task -> fclose)

$handle = frankenphp_get_worker_handle();
while ($message =  fgets($handle)) {
    if ($message === "task") {
        while ($task = frankenphp_receive_task()) {
            [$stream, $payload] = $task;
            frankenphp_update_task($stream, ['progress' => 50]);
            frankenphp_update_task($stream, ['result' => process($payload)]);
            fclose($stream);
        }
    }
}

It's not just about making the API less awkward, it's also about keeping control over how we handle what is sent in the streams. Doing it somehow like this still allows integrating the handle into amphp/react/etc with minimal surface.

$worker = new \FrankenPHP\Worker(onMessage: function(\FrankenPHP\Message $message){
   $message->respond(['result' => process($message->payload)]);
});
$handle = $worker->getHandle();
while ($message =  fgets($handle)) {
   $worker->handle($message);
}

Also allows us to just call exit() directly on shutdown if we want to.

[nicolas-grekas]

You're right that "task\n" is a bad idea, my mistake, and the pool case rules it out: a line is a wake-up, not a description. A plain "\n" would be enough, content unspecified.

That leaves the kind to the receive side, which may be where an object fits: frankenphp_receive([Task::class]) returning a Task. The list is the opt-in, so a script only ever sees kinds it knows, the runtime keeps the wire format and can answer for scripts that don't ask, and a new kind is a new class instead of a new function. Does that give you the room you're after? About callback-based approaches, they're doomed to fail: a handler that runs to completion holds one task at a time, so anything that keeps several open and selects across them has to escape it, which is the case multiplexing needs.

Mostly a #2636 discussion, but it belongs here too, since it confirms the shape of this PR: one handle, an opaque wake-up, EOF on drain, nothing parsed.

On exit(): EOF already ends the loop, so a script can return or exit on shutdown with no API for it.

[AlliBalliBaba]

Yes on one hand a Task class would be much cleaner, on the other hand, forwarding the message on the handler back to us leaves us in control of what to do.

For example when we get back "1", we poll for a message and call the onMessage handler with the message as argument. When we receive "2", we call the onShutdown handler and exit out of the script.

In the future we might want to do something like send "3" when there is a "tick" or "request" event. Gives us more control, eg. we can add a custom \FrankenPHP\Tick and throw an Exception if the handler is not implemented. The user does not need to worry about what "1", "2", "3" means.

Doing this via callback doesn't stop us from handling multiple tasks at a time, $worker->handle($message) just calls another function immediately. The event loop can call $worker->handle($message) before the callback is finished again no problem (unless I'm missing something)

[AlliBalliBaba]

Looking at how react/revolt incorporate handles, maybe we could even do something like this:

$worker = new \FrankenPHP\Worker(onTask: function(\FrankenPHP\Message $message){
   $message->respond(process($message->payload));
});
$handle = $worker->getHandle();

// --- ReactPHP ---
Loop::addReadStream($handle, function () use ($worker) {
    $worker->tick(); // handle just is a wakeup mechanism, we can decide what to do here non-blocking
});

[nicolas-grekas]

Two review-level items.

Name collision between a global and a php_server background worker: names are now scoped like paths, unique within a php_server or among global workers, so both start. Each script sees jobs in FRANKENPHP_WORKER, WithWorkerName() resolves within the request's server first, and metrics and logs report the scoped one as <server name>:jobs (server names get a numeric suffix when two blocks resolve to the same one). Two workers with the same name inside one php_server fail Init(). TestBackgroundWorkerOnServer and same_worker_name_in_two_servers cover both directions.

Windows: the Windows workflow runs the full suite on PRs and it passes here on 8.5.10, background worker tests included. It also surfaced that stream_select() on a pipe would have spun on 8.4 and older, since php_select() only waits properly on sockets before the GH-16889 fix, which is 8.5 only. The handle is therefore backed by a socket pair on every platform, which takes the version-independent Winsock path.

The branch is squashed to 3e93a24; sha references in earlier replies predate the squash.

[nicolas-grekas]

Since the replies above, a self-review pass amended into the single commit (2f9c5b6):

• a worker scoped to a Server never passed to WithServer() panicked on a nil map; Init() now rejects it
• failureCount reset at the ready point, like HTTP workers: a crash after it restarts right away, only boot failures count toward max_consecutive_failures
• the drain race check in setupScript also covers TransitionRequested
• Windows: PHP's socketpair() emulation listens on INADDR_ANY and accepts the first peer, so the pair is verified with getpeername/getsockname; SOCK_CLOEXEC where available
• a warning after 10s when a script has not waited on its handle, since Init() and Shutdown() wait for that point; docs say to block on the handle, feof() polling is not a wait
• max_threads is rejected on background workers instead of ignored, and the Caddyfile requires num
• untyped resource return in the stub, dead zend_unset_timeout() between requests removed, background threads no longer reported busy by the threads endpoint
• tests: parking on a blocking read, RestartWorkers() draining a parked script, the handle throwing outside a background worker, the new validations

CI: all test jobs pass. The Windows job's caddy-suite timeout (POSTed configuration isn't active, then an admin GET hangs) is the same flake main's nightly runs hit (33416988237, 33302318173) and passes on rerun. The Docker matrix fails on the Go 1.27 runtime/cgo link error that #2634 and main's nightly show too.

[Copilot]

🟡 Changes recommended

Handle aliasing, unbounded crash loops, ambiguous metric identities, and extension-worker regressions remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 45/45 changed files
• Comments generated: 5
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Scoped extension dispatch drops request options, and background-worker configuration has inconsistent global and environment behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 47/47 changed files
• Comments generated: 3
• Review effort level: Balanced

[henderkes]

I'm largely happy with the API decisions.

Mostly architectural pointers, I'm a bit concerned with the amount of code duplication and extra code debt.

Also raising this from above here again: #2617 (comment)

[henderkes]

I'd be okay with this, if we weren't immediately introducing another implicit contract by setting FRANKENPHP_WORKER_BACKGROUND to literal 1.
Don't have a solution, but I suspect @AlliBalliBaba may have a smart idea

[nicolas-grekas]

Thanks, all items should be addressed now!

[henderkes]

Thanks, all items should be addressed now!

I'm not that fast in reviewing 😆. I hope to look again tomorrow.

blub

[henderkes]

I tickled the slop machine and it spew the following out. Skip reading if you'd like to preserve your sanity.

[henderkes]

LLM: [P1] Validate newly rejected duplicate names before stopping the active runtime. This check runs inside Init(), but caddy/app.go:145–149 calls Shutdown() before Init(). Reloading with two distinct existing scripts sharing a name therefore stops the working runtime and only then rejects the configuration.

Reproduced with an actual Caddy server: PHP returned 200 before reload; POST /load returned 400 for the collision; the stored configuration remained the original one, but subsequent PHP requests returned 500 (server is not registered), while static requests still returned 200. Caddy rolls back configuration state, not the stopped PHP runtime.

The destructive ordering predates this PR, but duplicate names are a new trigger: Caddy previously renamed these collisions. A shared, side-effect-free declaration preflight before Shutdown() would preserve the active site and consolidate the rules currently split between Caddyfile parsing, thread calculation, newWorker(), and addWorker(). Include effective names/scopes and background constraints so JSON and Go declarations follow the same rules. Please add a rejected-reload regression that verifies the old PHP site still serves afterward.

[nicolas-grekas]

The destructive ordering is real but it is not specific to duplicate names: Start() calls Init() after the old app stopped, so any declaration error, a missing file, a bad num, an invalid ini, already leaves the runtime down while Caddy rolls the configuration back. Duplicate names being an error rather than an auto-rename adds one more trigger, it does not create the failure mode.

The fix belongs in its own change: a Validate() on the app, which Caddy runs before stopping the previous one, over a side-effect-free preflight shared with Init(). That is worth doing, and worth doing for every declaration rule at once rather than for this one.

[henderkes]

LLM: Readiness-contract clarification: PHP_STREAM_AS_FD_FOR_SELECT is also used by introspection, not only stream_select(). Reproduced that stream_isatty($handle) returns false but still releases Init() before any read or select; a fetch-only control at the same bootstrap gate kept Init blocked.

This is consistent with the commit's literal 'select cast' mechanism, so I would treat it as a design/documentation clarification rather than an unequivocal correctness regression. However, user-facing 'first wait' wording suggests stronger semantics than this hook can guarantee, and an introspection call before later bootstrap work changes failure classification. Please settle and document the supported readiness boundary before adding more per-function detection. Checking ret != NULL alone would not exclude isatty, because it also performs a real cast.

[nicolas-grekas]

Accepted as the boundary: readiness is the first operation that waits or prepares to wait on the stream, and a cast for select is that. stream_isatty() also casts, so it marks the worker ready early; harmless, since the only effect is that startup stops waiting for a script that already has its handle.

[AlliBalliBaba]

Posting it here again for visiblity, since the end goal is to be able to integrate into a PHP event loop I'd much prefer an API that looks somewhat like this since it gives us more control to optimize and extend WDYT @henderkes @alexandre-daubois @dunglas @nicolas-grekas

$worker = new \FrankenPHP\Worker(
    onTask: function(\FrankenPHP\Task $task){
       $task->respond(process($task->payload));
    }.
    onShutdown: function(){
        // exit afterwards
    }
);
$handle = $worker->getHandle();

// event loop example (this is how Revolt adds streams)
Loop::addReadStream($handle, function () use ($worker) {
    $worker->tick(); // calls onTask if there is a task, calls onShutdown if shutting down, nonblocking
});

[henderkes]

Isn't this one PR early to focus on this? The only contract here is a stream select + EOF to close, which wouldn't prevent a userland wrapper around it.

I think I'd rather avoid an OOP API that suggests users can programmatically control the server like in Swoole, until that's actually true for more than just stream creation.

[AlliBalliBaba]

We kind of need to look ahead, frankenphp_get_worker_handle() does not make sense without selecting over multiple streams or an event loop. Could also do it like this with global functions only:

frankenphp_worker_on_task(function($task){
  $task->respond(process($task->payload));
});

frankenphp_worker_on_shutdown(function($task){
  //...
});

$handle = frankenphp_worker_get_handle();

Loop::addReadStream($handle, function () {
    frankenphp_worker_tick();
});

Allows us to keep in control over what happens on wakeup. Also allows us to cleanup after the callback (not possible in the current api). Also allows us to send an initial readiness check as requirement to reach frankenphp_worker_tick(), which makes much more sense than only requiring reaching frankenphp_worker_get_handle.

[henderkes]

The current contract requires both get_worker_handle and a stream read/select, but you're right that an explicit readiness is better.

Why do you want to turn worker_on_task into a callback that's invoked by worker_tick? Or what's the point of the latter? I think for now I'd favour the global function variant with a potential oop wrapper left to userland.

[AlliBalliBaba]

2 main reasons looking at the example here.

If the response stream of the task is not closed, it will lead to endless hanging/stream leak. Having a callback allows us to close the stream automatically, making those bugs not possible to begin with.

Extending the api is more awkward since you leave it up to the user to type check whatever comes back from frankenphp_receive_task(). With callbacks you can just throw a proper error if the handler is not implemented. (eg if we someday want to add something like a frankenphp_on_async_request() or frankenphp_on_tick())

[henderkes]

Oh... the new proposal went a lot further than the original draft PR.

I don't like that at all. For a dispatch-and-receive Task worker implementation we need better than manually selecting and writing over streams in userland. Cleanup concern aside, it's too much of an implicit contract that will be gotten wrong.

You're right that, at the very least, we need a Task abstraction. Operations like progress(), complete(), fail(), and cancel() on it, await() and abandon() on the future a task sender would get. The part I still wouldn't want is the new Worker(), because for now it'd be misleading with bg workers actually configured via Caddyfile.

But I'm not even sure I want all that in FrankenPHP right now. If we, as in @php/frankenphp-collaborators do, then we actually need to think over changing the surface this PR introduces too. Perhaps good to talk this over in person at API Con... are you coming @AlliBalliBaba ?

[nicolas-grekas]

The task PR is really to be considered & discussed in depth after the first two.

This very one on its own is great already as it allows eg running a messenger consumer in the same process.

The next one is the whiteboard idea and is really this first innovative step. Achieving that alone would be a big win for the ecosystem already.

Then the task API can be shipped on it's own. Happy to discuss it later.