Rolling deploy wedges when stopped replicas count toward Compose scale target

[vishr]

Summary

A rolling deploy can halt with scale up produced no new container when the existing replicas are stopped/exited. Onebox appears to calculate the next scale target from live/eligible replicas, while Docker Compose counts stopped project containers toward --scale.

Environment

• Onebox: v2026.9.1 (46793e2ae3537aea86708f9e9910bb3b65456586)
• Workload strategy: rolling
• Desired replicas: 3
• Existing replicas: 3, all crash-looping and unhealthy

Reproduction

1. Have a rolling workload with 3 desired replicas.
2. Put all 3 existing replicas into a crash loop.
3. Stop the replicas and disable their restart policies so host health is stable enough to pass the plan/apply drift fence.
4. Run ob deploy for a release with a healthy replacement image.

The generated plan correctly shows a rolling replacement and a per-replica Compose scale-up.

Actual behavior

The server release phase halts immediately:

server rolling ×3
release server: role server: scale up produced no new container
(deploy halted — `ob resume` after fixing, or `ob abort`)

At this point Compose had counted the stopped containers toward the scale target instead of creating a replacement. In the observed incident, two of the three stopped replicas were removed during the failed attempt and one remained.

After manually removing the remaining stopped replica, ob resume succeeded: it created three new replicas, waited for each to become healthy, and completed the rolling phase.

Confirmed mechanism

Onebox side:

• cur is running-only — roll.go:85 calls containerIDs, which runs
  docker ps -q with no -a (preflight.go:130-132). The scale target is
  len(cur)+1 (roll.go:114).
• scale up produced no new container is emitted at roll.go:131-132 when no
  ID returned after the up --scale falls outside known — and newcomerIDs
  is also docker ps -q (roll.go:52-55).

Compose side (docker/compose v2.39.1):

• up observes containers with all=true, stopped included
  (pkg/compose/create.go:103 → containers.go:43-48).
• ensureService sets actual := len(containers) and creates expected-actual
  (convergence.go:120-121, :203-204) — zero or negative in this repro.
• Surplus is not merely ignored, it is destroyed:
  if i >= expected { stopAndRemoveContainer } (convergence.go:155-163),
  oldest-created first.
• Under --no-recreate an Exited container is left in place
  (convergence.go:187-195) and then restarted by up's start phase with its
  old config (up.go:41-46, start.go:44).

So with 3 desired, 3 stopped and --scale 1: Compose removed two stopped
replicas Onebox never chose to remove, restarted the third on the crash-looping
image, and created nothing. That restarted leftover is exactly the one replica
the reporter had to delete by hand before ob resume worked.

Expected behavior

Onebox should replace stopped/exited replicas without requiring manual Docker
cleanup. Of the three approaches originally listed, remove stopped managed
replicas before scaling is the one that holds up:

• Count all project containers Compose counts — rejected. Compose then creates
  one newcomer, but its start phase restarts every stopped old container on the
  known-bad image. Those flap in and out of docker ps, so retireContainer's
  docker exec … touch on olds[0] (roll.go:186) can hit a container that
  just exited, halting the roll.
• Create replacements independently of Compose scale accounting — rejected.
  It means reimplementing the Compose service model (networks, volumes,
  env_file, healthcheck, labels), and Compose's observed state includes even
  one-off containers (create.go:103), so the accounting never actually becomes
  independent.
• Remove stopped managed replicas first — recommended. The
  operator-stopped-a-container concern does not apply here: RollRole runs only
  for a workload the plan is replacing (retained workloads never reach it,
  deploy.go:332-350), its contract is "N replicas of the new release", and a
  running old would be retired anyway (roll.go:148-151). A stopped one carries
  no traffic to protect. A stopped container of the target release on
  ob resume is a failed newcomer the roll itself would have rm -f'd
  (roll.go:142). Scope the sweep to
  com.docker.compose.project=<app> + service=<workload> + non-running
  states, log each removal, and state it in the plan output.

Recreate mode is unaffected: --force-recreate --scale N replaces or scales
down every observed container, stopped ones included.

Two related defects in the same loop

Worth fixing together, since both are the same running-only assumption:

• Newcomer detection is running-only (roll.go:120-133). A newcomer that
  crashes before the post-up docker ps reads as scale up produced no new container instead of reaching the health diagnosis — and leaves behind a
  stopped container that triggers this very bug on ob resume. Using
  docker ps -aq for that step lets the existing waitHealth → rm -f path
  (roll.go:140-143) report the real cause and clean up after itself.
• reslot builds taken from running containers only
  (roll.go:289-310). Docker names are unique across all states, so a stopped
  container holding app-web-2 makes the docker rename at roll.go:325 fail.

On the drift fence

The original report said the host-drift fence behaved correctly and is not the
issue. On review that is worth revisiting, because the fence is what pushed the
operator into the failing state.

WorkloadHealth is derived from docker ps .Status text
(preflight.go:271-289), so a crash-looping container flaps between starting
and down. VerifyBinding then DeepEquals the whole vector
(plan.go:106-108) and refuses nondeterministically. That is precisely why the
replicas were stopped in the first place — and stopping them is what triggered
the scale wedge.

Only retained workloads need health in the binding, and they are re-validated
at apply anyway (workload_plan.go:32-71). Either drop health from the binding
for workloads the plan is replacing, or collapse all non-ready states to a
single token. Without that, the deadlock in "Operational impact" below is
reachable again by a different route.

Operational impact

This creates a recovery deadlock during an outage: crash-looping replicas continually trip the host-drift fence, but stopping them to stabilize health can make rolling scale-up produce no replacement. Recovery then requires out-of-band Docker mutation before ob resume can proceed.

Ordering

Land this before #165. That issue makes ob secrets push a new caller of
RollRole; if it lands first, the new rolling secrets path is born with this
wedge — a stopped replica mid-rotation gives scale up produced no new container, and recovery rolls into the same wedge. Both change RollRole's
loop and signature, so they should be sequential rather than parallel.