[Provider plugins F4] Conformance harness, hostile fixtures, and agent CI provider mode

[AntoineToussaint]

Outcome

Create the proof mechanism for the provider foundation:

• a normative Codefly-owned provider API fixture server;
• a neutral provider exercising every lifecycle capability;
• hostile providers attempting to break trust boundaries;
• a real provider leaf-protocol mode in codefly agent ci;
• deterministic golden/cassette/state/output/receipt fixtures;
• Linux and macOS enforcement.

F4 is the P1 exit gate. Vendor work cannot start merely because F1-F3 compile.

Dependencies

• F1 contracts: [Provider plugins F1] Kind, protocol v0, manifest/artifact format, and generic contracts core#133
• F2 broker: [Provider plugins F2] Secure host broker, credential capabilities, response filtering, and cassettes core#134
• F3 coordinator: [Provider plugins F3] Coordinator, permissions, hardened spawn, state, projections, and CLI #196

F4 integrates the production implementations. It must not replace the broker,
PDP, sandbox, state, sink, or coordinator with permissive stubs.

Why this issue exists

Current codefly agent ci conformance is service-oriented: it generates a
service and runs workspace gates. A provider is a typed leaf protocol and needs
a new harness.

The design's strongest invariants are negative:

• an untrusted provider cannot read workspace/secrets;
• it cannot dial outside the broker;
• it cannot lie about principal/action/resource;
• it cannot spend a credential on an unplanned request;
• secret-bearing read/write responses do not reach the provider;
• partial/crash outcomes remain recoverable.

A well-behaved sample provider cannot prove those claims. A hostile fixture is
mandatory.

Repository ownership

Primary: codefly-dev/cli

The normative fixture server and providers may initially live in:

• CLI conformance testdata, or
• Core provider conformance test packages.

A separate provider-conformance repository is optional and must not block
this issue.

Use approved Codefly build/test commands. The provider agent must be built,
packaged, installed, and loaded through the same paths production uses.

Deliverables

1. Normative reference fixture server

Implement a loopback server with deterministic resources and fault injection:

• account identity/mode;
• resource list/retrieve/create/update/replace/delete/import;
• ownership metadata;
• cursor pagination with at least two pages;
• incomplete page/cursor failure;
• 429 and retry-after/rate metadata;
• idempotent POST and same-key replay;
• non-idempotent POST;
• one-time secret on mutation;
• secret-bearing read/list response;
• safe public value adjacent to secret fields;
• API version/create-only field;
• redirect responses;
• oversized/compressed/malformed/duplicate-key JSON;
• configurable timeout:
  • before request read;
  • after request accepted;
  • after mutation/before response;
• dropped connection after effect;
• deterministic request IDs/timestamps.

The fixture is a normative Codefly protocol test artifact, not a fake vendor
used as vendor acceptance evidence.

Expose safe inspection endpoints to assert:

• exact requests received;
• resources/effects;
• idempotency replay;
• no request after denial/incomplete observation;
• cleanup.

2. Neutral provider

Implement the complete provider-neutral lifecycle:

• information/catalog;
• good/bad offline validation;
• paginated observation;
• deterministic material observation;
• no-op/create/update/replace/delete/import/manual/blocked/output plans;
• one-action Apply;
• broker callbacks;
• checkpoints;
• public and captured-secret output;
• state upgrade;
• local and remote Doctor;
• idempotency/error/rate classification.

It must use no vendor-specific Core/CLI branch.

3. Hostile provider suite

Build explicit attack modes:

• catalog exceeds bound manifest;
• catalog changes after Plan;
• undeclared/unplanned action;
• broad or wrong resource/remote ID;
• wrong origin/method/path/query/body;
• caller Authorization/cookie/Host/proxy headers;
• URL userinfo;
• redirect;
• direct external dial;
• direct loopback dial outside broker;
• inherited environment/proxy/key/secret reads;
• workspace/configuration/~/.codefly/other-provider state reads;
• workspace writes and symlink tricks;
• permission callback principal spoof;
• callback action/resource lies;
• reuse handle across binding/purpose/action;
• capture retarget/broaden/downgrade;
• raw response/log/cassette access;
• secret-resolver/shell/curl/vendor-CLI subprocess;
• extra request before checkpoint acknowledgement;
• mutation from Observe/Doctor;
• output key/classification/purpose/browser/consumer downgrade;
• old agent reading newer state.

The test passes only when the attack fails at the intended enforcement point
and no side effect occurs.

4. codefly agent ci provider mode

Retain reusable source/build/package/audit/drift stages, then add provider
conformance:

1. validate agent and provider manifests;
2. build/package through Codefly;
3. verify manifest/artifact digest;
4. install/resolve/load through authenticated UDS and hardened spawn;
5. verify runtime catalog subset;
6. run offline RPCs with a broker-request counter fixed at zero;
7. Observe fixture through real F2 broker;
8. calculate/compare golden material observation and plan;
9. authorize/apply through real F3 coordinator/PDP;
10. capture secret and project public+secret output;
11. verify state/checkpoints/signed receipt sequence;
12. rerun and require no diff;
13. exercise drift and update;
14. exercise replace;
15. import exact resource;
16. disconnect/retain;
17. destroy exact owned resource;
18. upgrade state;
19. run hostile suite;
20. verify cleanup.

JSON report must name every gate, artifact digest, platform, and safe
correlation ID.

5. Golden artifacts

Commit reviewed deterministic:

• manifests/catalogs;
• desired/observed/material plans;
• request descriptors;
• sanitized broker cassettes;
• state before/after each lifecycle action;
• public/secret output metadata;
• signed receipt sequences;
• diagnostics and JSON output;
• version-upgrade fixtures.

No artifact contains raw secrets, credentials, DSNs, emails, domains, or
unsafe bodies. Seed poison values and hard-fail if they appear.

6. Platform matrix

Run provider hostile/conformance on:

• Linux/amd64;
• macOS/arm64.

Prove per platform:

• no direct egress;
• no TCP fallback;
• scrubbed environment;
• no workspace/secret/state reads;
• no writes;
• no cross-provider socket/state access;
• no arbitrary subprocess;
• cancellation/crash cleanup.

Provider execution is not supported on a platform until this matrix passes.

7. Fault and recovery matrix

Inject and verify:

• cancellation before send;
• crash after send/before response;
• response after mutation but before provider checkpoint;
• secret capture success then state/output failure;
• sink failure after remote effect;
• state commit then projection failure;
• projection success then receipt finalization failure;
• two concurrent CLI applies;
• stale approval/observation/endpoint/artifact;
• revocation between actions;
• remote manual drift/deletion;
• agent upgrade during Plan/Apply;
• idempotency window active/expired;
• incomplete pagination;
• policy/callback/backend failure.

Expected terminal state and recovery must be deterministic:

• FAILED with no effect;
• PARTIAL with known effects;
• UNCERTAIN with bounded safe next action;
• no blind retry.

Required tests

Pure/golden

• schema/canonical/digest;
• permission truth table;
• hostile request derivation;
• state/upgrade;
• writer/sink;
• receipts;
• JSON/exits.

Broker replay

• every network scenario uses F2 production code;
• replay never falls back live;
• record/replay policy identical;
• poisoned secrets absent;
• response filtering handles read-array secrets.

Full conformance

• all lifecycle steps and adversarial attempts;
• production admission missing-layer matrix;
• Gate A local;
• Gate B production-observe policy with read-only broker;
• Gate C remains rejected.

Acceptance criteria

• Neutral provider completes:
  Validate -> Observe -> Plan -> Apply one idempotent resource ->
  capture/project public+secret -> signed receipts -> no-diff rerun -> drift
  update -> replace -> import -> disconnect/retain -> exact destroy.
• Offline RPCs make zero broker calls.
• Every hostile attack fails without remote/local/config/sink side effects.
• Poison credentials/secrets are absent from provider process inputs,
  environment, output, state, plan, logs, diagnostics, receipts, and
  cassettes.
• Crash/lost-response cases terminate honestly and reconcile without duplicate
  mutation.
• Permission revocation stops later requests/actions.
• Linux and macOS suites pass.
• codefly agent ci emits stable machine-readable evidence and nonzero exit
  on any failed invariant.
• F1-F3 are not considered complete until this gate is green.

Explicit non-goals

• Stripe/Sentry/Resend API qualification.
• Hand-written fake vendor clients as acceptance evidence.
• Live vendor calls in default CI.
• Production remote state/sink.
• Windows support.
• Creating a separate conformance repository unless independently useful.

Review coverage

F4 supplies the falsification mechanism for the complete F1-F3 security and
lifecycle invariant set, including the previously missing hostile-provider,
read-path-secret, crash, pagination, and macOS enforcement evidence.