fix: gRPC fork safety for Gunicorn prefork (grpcio >= 1.83), sw_grpc aio compatibility, websockets >= 13 support

[wu-sheng]

Fixes apache/skywalking#13958

Root cause

sw-python run -p gunicorn started a full agent in the Gunicorn master, creating a gRPC channel before fork(). gRPC Python has been migrating to the EventEngine poller since grpcio 1.80.0, and a channel that lives across fork() is no longer safe: the at-fork handlers self-skip while EventEngine threads are inside gRPC (fork_posix.cc:71), leaving stale poll-engine eventfds that produce the reported continuous Kick Failure (eventfd_write: Bad file descriptor) spam — and, worse, a racy silent deadlock of forked workers inside gRPC's own at-fork handlers (worker never boots, gunicorn never notices). grpcio 1.83.0 removed the last legacy-poller opt-out (grpc/grpc#42828); upstream reports grpc/grpc#43055 and grpc/grpc#43062 are open and unfixed. The error is client-side only — unrelated to the OAP version, reproduced against a mock collector with no OAP.

Changes

Gunicorn prefork fix

• Master = instrumentation only (SkyWalkingAgent.start_prefork_master()): config finalize + plugin/log patching + at-fork hooks; queues, reporter threads and the gRPC channel are created only in forked workers — the same model the uWSGI path has always used. The master no longer registers as a service instance.
• agent.started() guard: instrumented code becomes a no-op (NoopSpan, dropped logs/meters) in a process whose reporters are not active — protects gunicorn --preload app imports in the master (covered by the test, which preloads and performs an instrumented call at module import).
• GRPC_POLL_STRATEGY is no longer set; grpcio floor raised to >= 1.83 (wheels' generated stubs already hard-require it at import via GRPC_GENERATED_VERSION), and codegen grpcio-tools is pinned in lockstep in the Makefile so future wheels cannot silently raise the runtime floor.
• Gunicorn prefork + SW_AGENT_ASYNCIO_ENHANCEMENT is rejected with a clear error instead of starting an unsafe pre-fork agent (the async agent has no fork support); the incompatibility is documented in the Gunicorn FAQ, AsyncEnhancement.md and the config reference.
• Explicit os.fork() with the gRPC reporter is documented (and warned at runtime) as unreliable on grpcio >= 1.80 — background reporters enter gRPC independent of application requests, and upstream races remain open; forking applications are directed to SW_AGENT_PROTOCOL=http/kafka. Gunicorn via -p is the supported channel-after-fork model.
• Fixed duplicate at-fork/atexit registrations in forked workers (both now register once per process lineage); hardened stop()/__fini; the worker rebuilds the profiling executor instead of inheriting a fork-dead one.
  CI repairs (pre-existing master failures, no support-scope changes)
• sw_grpc / grpcio 1.83.0: grpcio 1.83.0 validates isinstance(x, Channel) against the grpc.aio._channel module global, which the plugin had rebound to a factory function → TypeError on every aio stub creation. Replaced with a real Channel subclass. Verified on grpcio 1.83.0; also exercised against 1.82.1 in an isolated harness purely to confirm the subclass changes no pre-1.83 behavior (the agent package itself now requires >= 1.83, and the plugin note documents that effective floor alongside the 1.* instrumentation range).
• sw_websockets harness: uvicorn 0.50.0 removed its legacy websockets fallback (websockets.server.ServerProtocol is websockets >= 11 only), crashing the test provider with websockets 10.x. Pinned uvicorn < 0.50 in the test compose only; websockets 10.3/10.4 remain tested.
• Mock-collector first-insert race discovered: SegmentItems.addSegmentItem is check-then-act on the first segment for a service name — two processes posting their first segments concurrently (the fork test's parent+child share one service name) silently lose one while both get HTTP 200 (15/200 reproduction with barrier-synchronized POSTs; 0/200 once the key exists). The fork test seeds the service name with a confirmed warm-up segment before the concurrent pair; the real computeIfAbsent fix belongs in skywalking-agent-test-tool.
  websockets >= 13 support (new)
• The plugin now instruments both client implementations: legacy websockets.legacy and the new websockets.asyncio ClientConnection.handshake (websockets >= 13, the default since 14) — previously apps on the modern API silently got no spans. Handles the sans-io wsuri->uri attribute rename across 13.x-17.x. Support matrix: 10.3, 10.4, 13.1 on all Pythons plus 17.0.1 on >= 3.11 (upstream requires it), so CI exercises the legacy path on 10.x and the new path on 13.1/17.0.1 across the version range.

Tests

• tests/plugin/web/sw_gunicorn: full cross-process trace — consumer (entry + requests exit) -> provider under gunicorn --preload -w 2 prefork (entry with CrossProcess ref) — validated against the mock collector, plus log assertions: exact worker-boot counts (catches the silent deadlock), agent spawned only in workers, master instrumented-only, async-rejection case, zero Kick Failure.
• tests/plugin/web/sw_fork_support: explicit os.fork() with SW_AGENT_EXPERIMENTAL_FORK_SUPPORT over the HTTP reporter (deterministic — no gRPC at-fork hazard), validating the complete cross-fork trace: parent entry/exit -> child entry with CrossProcess ref.
• Both auto-collect into the existing CI matrix (no CI.yaml changes).

Validation

Docker matrix on the patched agent, grpcio 1.80.0 / 1.82.1 / 1.83.0 x 8-10 boots each: zero error spam, zero worker deadlocks (the old agent deadlocked a worker in ~1/2 to ~1/24 forks), all segments delivered. fd-level inspection: the gunicorn master holds no collector connection; each worker holds exactly one. websockets validated end-to-end on 10.4 (legacy path), 13.1 and 17.0.1 (new asyncio path) with full span validation.

[Copilot]

Pull request overview

This PR improves the SkyWalking Python agent’s compatibility with modern gRPC (grpcio >= 1.83) in pre-fork server environments (Gunicorn), fixes sw_grpc asyncio-channel instrumentation for grpcio 1.83, and extends the sw_websockets plugin to support websockets >= 13’s new asyncio client API.

Changes:

• Reworks Gunicorn prefork startup to avoid creating gRPC channels before fork() (master becomes “instrumentation-only”, workers run full agent) and adds fork-safety regression tests.
• Fixes sw_grpc aio channel patching to remain compatible with grpcio 1.83’s Channel type checks.
• Extends websockets client instrumentation to cover both legacy and new asyncio implementations; pins uvicorn in the websockets test harness for stability.

Reviewed changes

Copilot reviewed 28 out of 29 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/plugin/web/sw_gunicorn/test_gunicorn.py	Adds regression test assertions around worker boot, agent start behavior, and absence of grpc fork error signatures.
tests/plugin/web/sw_gunicorn/services/provider.py	Adds minimal Flask provider endpoint for Gunicorn prefork testing.
tests/plugin/web/sw_gunicorn/services/consumer.py	Adds consumer service issuing traced HTTP call into the Gunicorn provider.
tests/plugin/web/sw_gunicorn/services/init.py	Package marker for test services.
tests/plugin/web/sw_gunicorn/expected.data.yml	Expected cross-process trace snapshot for Gunicorn prefork scenario.
tests/plugin/web/sw_gunicorn/docker-compose.yml	Defines Gunicorn master/worker setup plus an asyncio-enhancement rejection case.
tests/plugin/web/sw_gunicorn/init.py	Package marker for the new plugin test.
tests/plugin/web/sw_fork_support/test_fork_support.py	Adds explicit os.fork() regression coverage under experimental fork support.
tests/plugin/web/sw_fork_support/services/app.py	Implements parent/child forked services and endpoints used by the fork-support test.
tests/plugin/web/sw_fork_support/services/init.py	Package marker for test services.
tests/plugin/web/sw_fork_support/expected.data.yml	Expected span/segment snapshot for fork-support scenario.
tests/plugin/web/sw_fork_support/docker-compose.yml	Runs fork-support scenario over HTTP reporter for determinism.
tests/plugin/web/sw_fork_support/init.py	Package marker for the new plugin test.
tests/plugin/http/sw_websockets/services/consumer.py	Updates test consumer to import/connect via websockets’ new asyncio API when available.
tests/plugin/http/sw_websockets/docker-compose.yml	Pins uvicorn<0.50 in test harness to avoid provider crashes with older websockets.
skywalking/plugins/sw_websockets.py	Instruments both legacy and new asyncio websockets client handshake paths.
skywalking/plugins/sw_grpc.py	Replaces aio channel factory rebinding with a Channel subclass to satisfy grpcio 1.83 type checks.
skywalking/config.py	Updates fork-support config commentary to reflect Gunicorn prefork behavior.
skywalking/bootstrap/loader/sitecustomize.py	Starts “instrumentation-only” in Gunicorn master and rejects asyncio enhancement under prefork.
skywalking/bootstrap/cli/utility/runner.py	Updates Gunicorn prefork messaging to reflect master instrumentation-only behavior.
skywalking/agent/init.py	Adds prefork-master start path, reporting/no-op guard, and fork-hook registration hardening.
pyproject.toml	Raises grpcio and grpcio-tools dependency floor to >= 1.83.
poetry.lock	Updates locked versions for grpcio/grpcio-tools and transitive dependencies (e.g., protobuf).
Makefile	Pins grpcio-tools for codegen in lockstep with the runtime grpcio floor.
docs/en/setup/Plugins.md	Updates websockets support matrix and notes to include new asyncio API instrumentation.
docs/en/setup/faq/How-to-use-with-uwsgi.md	Clarifies master-process monitoring limitations for uWSGI and Gunicorn.
docs/en/setup/faq/How-to-use-with-gunicorn.md	Documents fork-safe Gunicorn behavior and grpcio >= 1.80 issue background.
docs/en/setup/Configuration.md	Updates fork-support configuration description to reflect prefork master/worker behavior.
docs/en/setup/CLI.md	Clarifies prefork CLI behavior and adds fork timing guidance.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.