Parse observeFullEvaluationData and hash targeting_key in flagevaluations events

[vjfridge]

Background

Feature-flag evaluations are reported to Datadog so users can see how their flags behaved in production. Today, every evaluation carries the raw targeting key (the identifier of the subject being evaluated; often a user email or user ID) and the full evaluation context. Those fields can contain personally identifiable information (PII).

Motivation & High level changes

This PR lets the server decide, per environment, whether that raw data may be observed via a new boolean on the UFC. It is the Java piece of the cross-SDK "Protecting PII in flagevaluations" initiative. dd-trace-java is the pilot SDK; the same pattern will be copied to the other SDKs afterward.

This directly resolves the privacy concern raised on the parent PR #11639, where @AlexeyKuznetsov-DD flagged that flag-evaluation reporting was default-on and shipped the raw targeting_key (user IDs / emails) plus the full evaluation context in clear text, and asked for hashing / opt-in / an explicit privacy sign-off. The no-PII path is now the default.

Server says observeFullEvaluationData	targeting key	evaluation context
true	sent as-is (raw)	included (raw)
false or absent (privacy-preserving default)	SHA-256 hashed, sha256_<hex>	omitted entirely

"Absent" is treated exactly like false, so an older/cached config can never accidentally leak raw values.

How it works

• New field observeFullEvaluationData on the downloaded flag-configuration (UFC) model (top-level boolean, defaults to false when missing).
• Consent is captured at evaluation time, on the evaluation thread, and carried on the event. DDEvaluator snapshots the active UFC's observeFullEvaluationData once per evaluation and stamps it onto the ProviderEvaluation metadata; the OpenFeature hook (FlagEvalLoggingHook) reads that metadata value and stores it as FlagEvalEvent.observeFullEvaluationData. Aggregation and serialization then read that per-event snapshot and never consult the live gateway — so the hashed-vs-raw decision is pinned to the configuration active when the evaluation happened.
  • Why not read it at drain/flush? CURRENT_CONFIG can be overwritten by a later Remote Config update between when an evaluation happened and when the batch is drained and flushed. Reading the gateway at flush would retroactively apply a different environment's consent to already-collected evaluations. (This was a real bug caught by system-tests: a targeting key came back hashed even though the active UFC said true.) Snapshotting on the evaluation thread closes that window in both directions.
  • Bucket separation. observeFullEvaluationData is part of the aggregation bucket key (FullKey / DegradedKey), so events with different consent values never merge into the same bucket. Consent-off and consent-on evaluations for the same subject land in distinct buckets and are emitted with their respective policies. An AND-fold on the bucket's consent field is kept as defense-in-depth in case the key ever drifts from the field.
• When hashing, the targeting key is run through the shared ULeb128Encoder.hashTargetingKey helper and prefixed with sha256_; the context field is dropped so it's absent from the JSON (not null, not {}).
• Evaluation logic is unchanged.

SHA-256 hash

The hash is unsalted SHA-256 over the raw UTF-8 bytes (no trimming/case/Unicode normalization), emitted as lowercase hex. Unsalted is a deliberate cross-SDK contract requirement: every SDK must produce the identical digest for a given targeting key so hashed values line up across languages and against the backend. It matches the canonical test vector shared across all SDKs:

"jane.doe@datadoghq.com" -> sha256_b4698f9b6d186781fa8dc59e533578fa2d8379a46b1cf6db85cda6aa9c99e51b

Performance

Consent is read once per evaluation on the hook path — a single AtomicReference.get snapshotted onto the event (no allocation, no per-span hot path). Hashing runs at flush cadence, bounded by the number of distinct evaluation buckets, and uses a ThreadLocal<MessageDigest> (no per-call MessageDigest.getInstance).

Testing

• Hash helper matches the canonical PII vector; preserves whitespace/case; is lowercase 64-char hex.
• Config parsing covers true, false (as a @ValueSource matrix), absent (defaults to false), and explicit null (rejected / fail-closed).
• Serialization covers both raw and hashed paths, asserting on the raw wire bytes that the hashed value is present, the raw PII value never appears, and no per-event context is emitted.
• Consent-capture at evaluation time: hook tests assert the event's consent value is read from the evaluation metadata stamped by DDEvaluator at evaluation time (true, false, absent → false), and that the hook ignores the live gateway even when it disagrees with the metadata (ignoresGatewayConsentEvenWhenItDisagreesWithMetadata). FlagEvalEvent tests cover the new field and the fail-closed default on the convenience constructors.
• Consent-timing regression guards (both directions): a bucket whose event was snapshotted with consent off stays hashed even if a later RC update turns the gateway on before drain/flush; and symmetrically, an event snapshotted with consent on stays raw even if the gateway is later turned off — proving neither aggregation nor flush consults the gateway.
• Bucket separation: mixed-consent evaluations for the same subject land in distinct buckets (mixedConsentEvaluationsForSameSubjectLandInDistinctBuckets), and same-consent evaluations merge as before (sameConsentEvaluationsForSameSubjectMergeIntoOneBucket).

🤖 Generated with Claude Code

[vjfridge]

system-tests validation (L3) — Java SDK ✅

Ran the three new observeFullEvaluationData PII-protection tests from system-tests#7316 locally against this branch (java@1.65.0-SNAPSHOT+81af153ca7).

All three pass:

Test	Result
Test_FFE_EVP_Flagevaluation_ObserveFullData_Absent_Hashed	✅ PASS
Test_FFE_EVP_Flagevaluation_ObserveFullData_False_Hashed	✅ PASS
Test_FFE_EVP_Flagevaluation_ObserveFullData_True_Unhashed	✅ PASS

36 passed, 8 skipped, 2704 deselected, 1 xfailed, 3 xpassed in 301.86s

Weblog: spring-boot | Scenario: FEATURE_FLAGGING_AND_EXPERIMENTATION

Note: an earlier build (d65c79f266) of this PR failed True_Unhashed — the SDK was always hashing regardless of the flag value. Root cause: FlagEvaluationWriterImpl.buildEventList() read observeFullEvaluationData from CURRENT_CONFIG at flush time, which could be overwritten by a subsequent RC update before the 10s flush fired. The fix in this PR (capturing the value per-bucket at enqueue time with a privacy-preserving fold) resolves it.

[vjfridge]

ffe-dogfooding validation (L2) - Java SDK ✅

I ran this branch's dd-java-agent + dd-openfeature build through our OpenFeature dogfooding harness against a staging UFC, exercising all three behaviors of the PII-protection contract:

Protected — observeFullEvaluationData = false (default):

• targeting_key = sha256_ + 64-char lowercase-hex SHA-256 digest (71 chars total)
• context.evaluation omitted
• consistent across every emitted flagevaluation event

Full — observeFullEvaluationData = true (opt-in):

• targeting_key raw (verbatim subject id)
• context.evaluation present with the full evaluation context

Kill switch — DD_FLAGGING_EVALUATION_COUNTS_ENABLED = false:

• zero flagevaluation events emitted regardless of observeFullEvaluationData (SDK logs Flag evaluation EVP writer disabled)

Notes:

• Pointing the app at environments with observeFullEvaluationData false vs true produced exactly the expected payload shapes.
• Hashing was applied even to FLAG_NOT_FOUND (runtime-default) evaluations — the privacy posture is enforced independent of evaluation outcome. 👍
• The SDK posts to the singular /api/v2/flagevaluation EVP route, as expected for the flagevaluation track.

Validation harness changes: ffe-dogfooding#100 — adds the per-SDK PII posture dashboard badge and fixes the singular /api/v2/flagevaluation intake route.

Verdict: the Java SDK is performing as expected for the PII-protection contract — default hashing, opt-in full data, and kill switch all behave correctly. Cross-SDK hash equality is covered in the companion system-tests work.

[datadog-prod-us1-5]

More details

The event-time snapshot, conservative bucket fold, and serialization gates preserve the no-PII default while retaining raw data only for explicitly consented evaluations. No additional tests recommended: production telemetry is unavailable and the PR already covers the relevant consent transitions and wire shape.

Was this helpful? React 👍 or 👎

_{📊 Validated against 3 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit e0e3632 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e0e3632b8c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[leoromanovsky]

Two conditions to resolve please.

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.84 s	14.71 s	[-0.1%; +1.8%] (no difference)
startup:insecure-bank:tracing:Agent	13.64 s	13.70 s	[-1.3%; +0.4%] (no difference)
startup:petclinic:appsec:Agent	17.49 s	17.41 s	[-0.5%; +1.4%] (no difference)
startup:petclinic:iast:Agent	17.43 s	17.52 s	[-1.5%; +0.5%] (no difference)
startup:petclinic:profiling:Agent	17.49 s	16.79 s	[-0.3%; +8.6%] (no difference)
startup:petclinic:sca:Agent	17.00 s	17.31 s	[-6.1%; +2.5%] (no difference)
startup:petclinic:tracing:Agent	16.58 s	16.73 s	[-2.0%; +0.3%] (no difference)

Commit: 42e013e1 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[vjfridge]

System-test validation

Validated this PR against the tests/ffe/test_flag_eval_evp.py suite in system-tests, which has end-to-end coverage of the PII protection contract.

Branch tested: vickie/FFL-2790-protect-pii-with-observeFullEvaluationData (this branch merged onto leo.romanovsky/ffl-2446-evp-flagevaluation-java, commit f2265372ad)
Weblog: spring-boot, java@1.65.0-SNAPSHOT+f2265372ad
Command: TEST_LIBRARY=java ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION tests/ffe/test_flag_eval_evp.py
Result: ✅ 11/11 passed

The two tests that directly exercise the PII contract:

• Test_FFE_EVP_Flagevaluation_ObserveFullData_Absent_Hashed — when UFC has no observeFullEvaluationData field, the targeting key must be SHA-256 hashed and context.evaluation must be omitted. ✅
• Test_FFE_EVP_Flagevaluation_ObserveFullData_False_Hashed — same contract when observeFullEvaluationData=false is explicit. ✅
• Test_FFE_EVP_Flagevaluation_ObserveFullData_True_Unhashed — when observeFullEvaluationData=true, raw targeting key and full evaluation context are passed through. ✅

Without this PR, both hashing tests fail: the raw email jane.doe@datadoghq.com is emitted instead of sha256_b4698f9b.... With this PR, all 11 tests pass, confirming the no-PII-by-default behavior is correct end-to-end.