To stay ahead of CVEs and compatibility with k8s. Also to avoid drifting too much, we want to do another update of our main agent to the latest datadog tag.
Acceptance criteria
- Have the main agent updated to the latest tag (7.80.4) by cherry-picking our changes onto the tag we chose as base.
- Make sure branding is still good (no references to datadog in the log, our STS_ environment variables still work, etc.)
- Make sure the Beest tests still work. An option during this work would be to move the Beest tests to a separate repo, to take those out of the big delta.
- Do proper QA/validation on the end result, showing our demo scenario still works.
Scope of this upgrade
Bring the StackState main agent from v7.78.2 (current baseline) to v7.80.4 (upstream tip). Covers 11 upstream releases published 2026-04-29 → 2026-07-01:
7.78.2 · 7.78.3 · 7.78.4 · 7.79.0 · 7.79.1 · 7.79.2 · 7.80.0 · 7.80.1 · 7.80.2 · 7.80.3 · 7.80.4.
The two feature-carrying releases are the .0 minor bumps (7.79.0 and 7.80.0); the rest are patch-level CVE / bug rollups.
Breaking changes to plan for
7.80.0 — Health Platform API change
ReportIssue(checkID, checkName string, report *IssueReport) → ReportIssue(IssueReport). The IssueReport struct now carries IssueID (unique instance id), IssueType (template id), and Source (integration name). Persistence file format bumped to v2 — existing <run_path>/health-platform/issues.json files are logged as incompatible and discarded on startup (no data migration). Every StackState call site that reports issues to the health platform must be updated. Telemetry tag {{health_platform.issues_detected
{health_check_id=...}
}} renamed to issue_type — audit any StackState monitors/dashboards that filter on it.
7.80.0 — APM trace-agent lazy startup (Linux)
Trace-agent now starts on-demand when data hits its listeners, not at agent boot. Opt-out via apm_config.socket_activation.enabled: false / DD_APM_SOCKET_ACTIVATION_ENABLED=false. Verify StackState integration tests / process supervision don't assume the trace-agent is running at startup.
7.79.0 — Docker SDK migration to moby/moby v29
github.com/docker/docker v28.5.2 → github.com/moby/moby v29 (api v1.54.1, client v0.4.0). Cascaded into a 7.79.2 fix: some daemons return non-CIDR DefaultAddressPools[].Base values in /info that the new strict netip.Prefix decoder rejects, breaking DockerUtil init → cascades to the Docker workloadmeta collector and Docker core check (missing container/image tags on all metrics, traces, logs). The 7.79.2 fix switches the connectivity probe to /ping. Must land ≥ 7.79.2 to avoid regressing StackState's Docker topology collection.
7.79.0 — Config implementation swap (nodetreemodel default)
The improved config implementation is now the default. Fallback: DD_CONF_NODETREEMODEL=viper or conf_nodetreemodel: viper in datadog.yaml. Behavior-change risk for any StackState code that reads unusual config patterns (env-var precedence, nil handling, unknown keys). Regression-test gate.
7.79.0 — system-probe discovery module on by default (Linux)
discovery.enabled: true becomes default; when discovery is the only enabled system-probe module, system-probe-lite is auto-selected. Any StackState process-topology assumptions about "discovery off" need re-baselining. Opt-out: discovery.enabled: false.
7.79.0 — NTP schema restore
{{ntp.offset
{source:intake}
}} renamed to ntp.intake_offset; source:ntp tag removed from ntp.offset — restores pre-7.77.0 single-series behavior. Watch StackState monitors that filter on the (now-removed) source: tag.
7.79.0 — OTLP receiver default bind change (documentation only)
OTLP defaults are now advertised as localhost:4317 (gRPC) / localhost:4318 (HTTP) — previously 0.0.0.0. Only affects docs; runtime forwarding of user-configured values unchanged.
7.80.0 — Cluster Agent kube-state-metrics 2.13 → 2.18
Allow/denylist syntax changes from Go regexp to ECMAScript regex. Any StackState-shipped metric_allowlist / metric_denylist patterns using Go-specific features (e.g. (?s)) must be rewritten.
7.79.0 — DDOT metric remapping moves to backend (beta gate on)
exporter.datadogexporter.DisableAllMetricRemapping feature-gate promoted to beta and enabled by default. Metric remapping now handled by the Datadog (StackState) backend, not the collector. Verify the StackState receiver handles or tolerates unremapped metric names, or leave the gate disabled via --feature-gates=-exporter.datadogexporter.DisableAllMetricRemapping.
7.80.0 — External-collector OCB support removed
Components previously consumable from an external OpenTelemetry Collector via OCB are removed from this repo — users should switch to the sibling components in opentelemetry-collector-contrib. If StackState ships a custom collector build referencing these components, update the reference paths.
Security / CVE inheritance
| Release |
CVEs / dependency bumps |
| 7.78.3 |
CVE-2026-39883 (go.opentelemetry.io/otel/sdk PATH hijack, BSD/Solaris only) → v1.43.0 |
| 7.78.4 |
CVE-2026-35469 (moby/spdystream SPDY/3 frame parser OOM) → v0.5.1 |
| 7.79.0 |
CVE-2026-34040 (High, 7.8) + CVE-2026-33997 (Med, 8.1) — Docker SDK → moby/moby v29 |
| 7.79.1 |
CVE-2026-42151 + CVE-2026-42154 — github.com/prometheus/prometheus → v0.311.4 |
| 7.79.2 |
CVE-2026-46680 — containerd v1.7.32 + pinned containerd/v2 v2.0.9 |
| 7.80.0 |
CVE-2026-6357 (embedded pip → 26.1.1); CVE-2022-26788 (Windows 1809/LTSC2019 base image migration off deprecated PowerShell) |
| 7.80.2 |
Cluster-Agent AppSec ingress-nginx confused-deputy (existed since 7.78.0) — mutator now enforces pod-namespace match |
Merging to 7.80.4 clears every one of these without needing per-CVE VEX entries.
Toolchain / embedded bumps to inherit
| Component |
7.78.2 |
7.80.4 |
| Go |
(pre-1.25.10) |
1.25.11 |
| Embedded Python (CPython) |
3.13.13 |
3.13.14 |
| pip |
(pre-26.1.1) |
26.1.1 |
| JMXFetch |
(pre-0.52.0) |
0.52.0 |
| OTel Collector core |
v1.53.0 (v0.147.0) |
v1.58.0 (v0.152.0) |
| OTel semantic conventions |
v1.38.0 |
v1.40.0 |
| Security Agent policies |
v0.78.0 |
v0.80.0 |
| kube-state-metrics |
v2.13 |
v2.18 |
Note: the CPython 3.13.13 → 3.13.14 bump lines up with several pkg/config CVE-scan findings that STAC has been mitigating via VEX; merging to 7.80.4 retires the current round.
New capabilities (surface for smoke / E2E after merge)
- nccl core check (7.80.0) — GPU NCCL collective metrics via Unix socket + libnccl-profiler-dd.so. Enable with gpu.nccl.enabled: true.
- datadog_csi_driver native Go check (7.79.0) — replaces the Python OpenMetrics integration; metric names/tags unchanged.
- kata core check (7.79.0) — Kata container runtime metrics.
- macOS DNS monitoring (7.79.0) via libpcap.
- macOS Cloud Network Monitoring (7.79.0) via BPF filters.
- agent experimental check-config / onboard commands (7.79.0) — 6-stage datadog.yaml validator, no running agent needed.
- multi_secret_backends in datadog.yaml (7.80.0) — multiple named secret backends selectable per handle via ENC[backendID;secretKey].
- DDOT k8sobjectsreceiver added to default manifest (7.80.0).
- Prometheus HTTP SD provider in Cluster Agent (7.80.0).
- Autoscaling profiles: Argo Rollouts target (7.80.0).
- kubernetes_state.endpointslice.* metrics (7.80.0).
- v3beta metrics intake shadow sampling (7.80.0; disabled in 7.80.2 due to zlib) — 0.1% shadow traffic to a v3 protocol validation endpoint on datadoghq.com only. Not sent to StackState org, but verify nothing leaks.
Notable bug fixes to inherit
- 7.79.0: config-package concurrent-map-write crash (Docker container_collect_all); deadlock after remote-config value cleared; IPv6 host:port bracketing across many call sites; agent status with HA Agent enabled.
- 7.80.0: system-probe network tracer panic (gateway lookup subnet cache); OTLP histogram panic (counts == bounds+1 invariant violation); C-memory leak in logs batch sender (resetBatch() not closing zstd StreamCompressor); NetFlow trailing-padding packet drops; check-instance-level secret decryption regression (present since 7.40).
- 7.80.2: container log tailer silently stopping on idle-timeout close; CIS Docker rules no longer evaluated on non-Docker CRI nodes (GKE COS false-positive fix).
- 7.80.3: workload-autoscaling CPU-limit-removal consistency in HA cluster-agent; PAR self-enrollment proxy support.
Related work
- STAC-24773 — omnibus → Bazel migration on the 7.78.2 baseline. Merged.
- STAC-25143 — post-24773 omnibus cleanup (OpenSSL/zlib/openscap → Bazel + orphan-recipe sweep). Merged.
Both of the above make this upgrade cleaner: fewer omnibus recipes to reconcile against upstream base-7.80.4 during the merge.
References
Subtasks
[Epic] STAC-25220
To stay ahead of CVEs and compatibility with k8s. Also to avoid drifting too much, we want to do another update of our main agent to the latest datadog tag.
Acceptance criteria
Scope of this upgrade
Bring the StackState main agent from v7.78.2 (current baseline) to v7.80.4 (upstream tip). Covers 11 upstream releases published 2026-04-29 → 2026-07-01:
7.78.2 · 7.78.3 · 7.78.4 · 7.79.0 · 7.79.1 · 7.79.2 · 7.80.0 · 7.80.1 · 7.80.2 · 7.80.3 · 7.80.4.
The two feature-carrying releases are the .0 minor bumps (7.79.0 and 7.80.0); the rest are patch-level CVE / bug rollups.
Breaking changes to plan for
ReportIssue(checkID, checkName string, report *IssueReport) → ReportIssue(IssueReport). The IssueReport struct now carries IssueID (unique instance id), IssueType (template id), and Source (integration name). Persistence file format bumped to v2 — existing <run_path>/health-platform/issues.json files are logged as incompatible and discarded on startup (no data migration). Every StackState call site that reports issues to the health platform must be updated. Telemetry tag {{health_platform.issues_detected
{health_check_id=...}}} renamed to issue_type — audit any StackState monitors/dashboards that filter on it.
Trace-agent now starts on-demand when data hits its listeners, not at agent boot. Opt-out via apm_config.socket_activation.enabled: false / DD_APM_SOCKET_ACTIVATION_ENABLED=false. Verify StackState integration tests / process supervision don't assume the trace-agent is running at startup.
github.com/docker/docker v28.5.2 → github.com/moby/moby v29 (api v1.54.1, client v0.4.0). Cascaded into a 7.79.2 fix: some daemons return non-CIDR DefaultAddressPools[].Base values in /info that the new strict netip.Prefix decoder rejects, breaking DockerUtil init → cascades to the Docker workloadmeta collector and Docker core check (missing container/image tags on all metrics, traces, logs). The 7.79.2 fix switches the connectivity probe to /ping. Must land ≥ 7.79.2 to avoid regressing StackState's Docker topology collection.
The improved config implementation is now the default. Fallback: DD_CONF_NODETREEMODEL=viper or conf_nodetreemodel: viper in datadog.yaml. Behavior-change risk for any StackState code that reads unusual config patterns (env-var precedence, nil handling, unknown keys). Regression-test gate.
discovery.enabled: true becomes default; when discovery is the only enabled system-probe module, system-probe-lite is auto-selected. Any StackState process-topology assumptions about "discovery off" need re-baselining. Opt-out: discovery.enabled: false.
{{ntp.offset
{source:intake}}} renamed to ntp.intake_offset; source:ntp tag removed from ntp.offset — restores pre-7.77.0 single-series behavior. Watch StackState monitors that filter on the (now-removed) source: tag.
OTLP defaults are now advertised as localhost:4317 (gRPC) / localhost:4318 (HTTP) — previously 0.0.0.0. Only affects docs; runtime forwarding of user-configured values unchanged.
Allow/denylist syntax changes from Go regexp to ECMAScript regex. Any StackState-shipped metric_allowlist / metric_denylist patterns using Go-specific features (e.g. (?s)) must be rewritten.
exporter.datadogexporter.DisableAllMetricRemapping feature-gate promoted to beta and enabled by default. Metric remapping now handled by the Datadog (StackState) backend, not the collector. Verify the StackState receiver handles or tolerates unremapped metric names, or leave the gate disabled via --feature-gates=-exporter.datadogexporter.DisableAllMetricRemapping.
Components previously consumable from an external OpenTelemetry Collector via OCB are removed from this repo — users should switch to the sibling components in opentelemetry-collector-contrib. If StackState ships a custom collector build referencing these components, update the reference paths.
Security / CVE inheritance
Merging to 7.80.4 clears every one of these without needing per-CVE VEX entries.
Toolchain / embedded bumps to inherit
Note: the CPython 3.13.13 → 3.13.14 bump lines up with several pkg/config CVE-scan findings that STAC has been mitigating via VEX; merging to 7.80.4 retires the current round.
New capabilities (surface for smoke / E2E after merge)
Notable bug fixes to inherit
Related work
Both of the above make this upgrade cleaner: fewer omnibus recipes to reconcile against upstream base-7.80.4 during the merge.
References
Subtasks