Dynatrace health check misreads 403s as 404s and drops PROCESS_GROUP_INSTANCE events

[LouisParkin]

Found while investigating Rabobank's Dynatrace connectivity report (https://github.com/StackVista/stackstate/issues/578). Both are independent of that customer's gateway problem and will keep causing silent data loss after it is fixed.

1. "404" in str(e) matches hex inside entity IDs, silently dropping events

dynatrace_health.py:237 classifies a failed entity lookup as "entity not found" with a substring test:

if "404" in str(e) or "not found" in str(e).lower():

The exception message embeds the full request URL, so any entity whose hex ID contains 404 matches. PROCESS_GROUP_INSTANCE-7091B9883B404E8E does.

The misclassification is not just cosmetic. It adds the type to problematic_entity_types, and dynatrace_health.py:224 then skips every remaining PROCESS_GROUP_INSTANCE event for the rest of the run:

if entity_type == 'PROCESS_GROUP_INSTANCE' and entity_type in problematic_entity_types:
    continue

Evidence from the customer PRD log (97h, 1165 health runs): 1165 entity errors carry 404 in the entity ID and appear on the client ERROR side, while zero of them reach the INFO handler at :247 — the ERROR/INFO delta is exactly 1165 (12286 vs 11121). In all 1165 runs, PGI entity calls stop dead at that entity while other entity types keep being queried.

Any HTTP error on such an entity triggers this, so it is reachable independently of the 403.

Fix: branch on the actual status code rather than the message text. The client already knows it — worth surfacing it on the exception (or a typed exception) instead of re-deriving it from a string. The PROCESS_GROUP_INSTANCE-only special case at :224 also looks like it wants to be type-agnostic.

2. Failed entity lookups are never negatively cached

_get_entity_definition (dynatrace_health.py:330-342) caches only on success, so a persistently failing entity is re-fetched for every referencing event, on every run, indefinitely.

Same log: 625 distinct entities produced 12286 requests (~20x), one entity hit 4051 times over four days. Each failure writes two log lines, which is what made the logs unreadable (TEST was 92% traceback).

Fix: cache the failure alongside the success, ideally with a short TTL so a recovered entity is picked up again.

Two smaller things noticed nearby

• A 403 gets neither the refresh-and-retry nor the diagnostics that a 401 gets. dynatrace_client.py:107 tells a 401 to "Verify token validity and required API v2 scopes (entities.read, events.read, eventTypes.read)", but 403 — the actual insufficient-scope signal — falls through to the generic branch at :116. Applying that message to 403 would have identified the customer's outage from a single log line.
• dynatrace_topology.py:237 aborts the whole topology run on the first failing component type, so one blocked entity selector kills all eight. In the customer's PRD it died on process and never attempted host/service/application/queue/custom-device or the v1 synthetic endpoint. Per-type error handling would degrade instead of going dark, and start_snapshot() is currently left without a matching stop_snapshot() on failure.

[LouisParkin]

Scope after Bram's review on StackVista/stackstate#578:

• Bug 1 (403 misread as 404) — confirmed, in scope.
• Bug 2 (negative caching) — dropped. Persistent negative caching risks pinning a failure after the environment has recovered.
• New: when a non-200 response has no error key, fall back to logging a truncated slice of the response body, so a proxy sitting in front of Dynatrace can tell us something. Needs a length cap and care not to echo anything sensitive from the body into logs.

Retitling isn't needed — the title still describes the remaining work. The 403 scope-hint message and the per-component-type topology resilience noted above are unaddressed either way; the response-body fallback partly covers the first.