feat(openfeature): emit server-side EVP flagevaluation - #11639
feat(openfeature): emit server-side EVP flagevaluation#11639leoromanovsky wants to merge 112 commits into
Conversation
|
🎯 Code Coverage (details) 🔗 Commit SHA: dd5b057 | Docs | Datadog PR Page | Give us feedback! |
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3d4244f8ae
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
83ac4c4 to
81ed6f1
Compare
The flag-evaluation writer posts to whatever EVP proxy endpoint the Agent advertises, exactly like the exposure writer, and disables response compression so the request is valid on both /evp_proxy/v2/ and /evp_proxy/v4/. Nothing in production ever pinned an endpoint: preferredEvpProxyEndpoint was always null, and DDAgentFeaturesDiscovery.supportsEvpProxyEndpoint() had no caller outside the branch it gated. Drop the parameter, the discovery helper, and the endpoint set it read - whose name also misdescribed its contents, since it held every endpoint the Agent advertises, not just the EVP proxy ones. createBackendApi keeps its responseCompression overload, which the flag-evaluation writer does use, and keeps coverage for the no-EVP-proxy and disabled-compression paths. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 248aac7f93
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A single unserializable value in a customer's evaluation context (for example Double.NaN, which Moshi rejects) would poison the aggregator: the encode path threw, the finally branch skipped `aggregator.clear()` because `aggregatesWereEncoded` stayed false, and every subsequent flush re-encoded and re-threw the same bucket forever. Clear the aggregator in the finally branch regardless of whether encoding succeeded. On success, this preserves the existing behavior. On failure, it discards the poisoned bucket so later flushes recover. Reported by Codex on the current head. Adds a regression test with Double.NaN in the context. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
FeatureFlaggingSystem.stop() flips the enqueue-enabled gate before FlagEvaluationWriterImpl.close() runs. An in-flight producer that resolved the writer reference before the gate flip could reach enqueue(), see the gate false while closed=false, and drop the event without incrementing DROP_REASON_CLOSED. Shutdown loss disappeared from telemetry. Rename countClosedDropIfClosed() to countClosedDrop() and remove the inner guard. The early-exit path in enqueue() only fires when isClosedOrEnqueueDisabled() is true, so we can attribute the drop to DROP_REASON_CLOSED unconditionally. Reported by Codex on the current head. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
OkHttp's BridgeInterceptor adds a transparent Accept-Encoding: gzip whenever the caller does not set the header. EvpProxyApi previously skipped adding the header when responseCompression=false, so the BridgeInterceptor added gzip anyway and the flag had no wire effect. The flagevaluation track was documented as "response compression disabled to match the merged Go behavior" but negotiated gzip on the wire. Set Accept-Encoding: identity explicitly when responseCompression=false to preempt the BridgeInterceptor. The v=true path is unchanged. Reported by Codex on the current head. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
System-test validation update Ran the EVP flagevaluation system-test suite locally against two branches to characterize the current state and the PII fix: Branch: The 2 failures are the PII hashing tests — the raw targeting key ( Branch: The two PII failures are resolved by #12042. That PR needs to land alongside this one for the full contract to hold. |
The Java hand-off queue (65,536) was 16x wider than the shared RFC target of 4,096 events, and the evaluation-context snapshot depth (32) was 8x deeper than the RFC target of 4. Every other merged SDK uses the RFC numbers: dd-trace-go queue=4,096 depth=4 dd-trace-rb queue=4,096 depth=4 dd-trace-py queue=4,096 depth=4 dd-trace-js queue=4,096 depth=4 libdatadog depth=4 (sidecar coalescer, no app-thread queue) Bring the Java hot path in line so a full queue holds at most 4,096 pre-aggregated events and a caller-owned Value tree is bounded at 4 nesting levels, matching cross-SDK behavior and shrinking the pre-queue capture footprint by ~128x on paper. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…queue guard The hot path previously called snapshotValues to deep-copy the caller's EvaluationContext (bounded only by depth and cycles) and deferred flatten and prune to the worker. That let a wide, long-string, or long-key context sit un-pruned in the hand-off queue, and let the app thread pay the full snapshot cost even when the queue was already full and the event would be dropped. DDEvaluator.copyPrunedContext replaces snapshotValues + flattenValues + FlagEvaluationAggregator.pruneContext on the EVP hot path. It walks the caller-owned context once and caps every retained-size dimension inline: MAX_CONTEXT_FIELDS = 256 top-level fields kept MAX_KEY_LENGTH = 256 chars per key (closes prior gap) MAX_VALUE_LENGTH = 256 chars per string value MAX_LIST_ELEMENTS = 256 elements walked per list MAX_STRUCTURE_PROPERTIES = 256 properties walked per structure MAX_SNAPSHOT_DEPTH = 4 nesting depth (cross-SDK RFC) Every limit is a named constant so it can be tuned in isolation. Work on the hot path is now proportional to what is retained, never to what the caller supplied. FlagEvaluationWriter gains hasCapacityForEnqueue()/countPreQueueOverflow() so FlagEvalLoggingHook can short-circuit when the queue is saturated, counting the drop as queue_overflow without doing any context-copy work. FlagEvaluationAggregator.pruneContext becomes a passthrough. Aggregator tests assert the new store-what-you-get contract; the pruning contract moves to DDEvaluatorTest.copyPrunedContext coverage. Cross-SDK note: other SDKs still prune on the worker. This change is a Java-specific "do no harm" hardening; the same design is a candidate to fan out. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ed deduplicated reason tag copyPrunedContext now returns a CopyResult carrying the pruned attrs map and an optional truncatedReason string (sorted comma-separated cap names, e.g. "max_key_length,max_value_length"). The hook calls w.countContextTruncated(reason) when any cap fires, and the writer batches counts per unique reason string in a ConcurrentHashMap, draining them at each 10s flush cycle as flagevaluation.context.truncated metrics with "reason:..." tags. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
FlagEvalEvent: drop supplier constructor, attrsSupplier field, and contextAttributes() — all callers now pass pre-pruned attrs directly. FlagEvaluationAggregator: read event.attrs directly instead of going through pruneContext(event.contextAttributes()); delete pruneContext() which was a no-op passthrough. FlagEvaluationWriterImpl: remove pruneContext/canonicalContextKey delegation methods, MAX_CONTEXT_FIELDS/MAX_FIELD_LENGTH constants, and DROP_REASON_CONTEXT_ERROR — the supplier-throw path that triggered context_error is gone so the constant and its metric are unreachable. Tests: delete enqueueDoesNotResolveContextBeforeBuffering, contextMaterializationFailureDropsSingleEvent, publicConstructorAndContextHelpersDelegateToSharedImplementations, and aggregatorPruneContextIsPassthrough; update remaining callsites to use event.attrs directly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…riter FlagEvalEvent: fix class javadoc that still described the removed lazy supplier path. FlagEvaluationWriterImpl: remove six unreferenced constant re-exports (EVAL_SCALE_FULL_BUCKET_TARGET, EVAL_SCALE_PER_FLAG_BUCKET_TARGET, EVAL_SCALE_DEGRADED_BUCKET_TARGET, GLOBAL_CAP, PER_FLAG_CAP, DEGRADED_CAP) — all delegated to FlagEvaluationAggregator but had zero callers on the writer side. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Flatten nested try/finally in FeatureFlaggingSystem.stop() to per-resource try/catch for readability - Use a single evalTimestampMs for both startAt/endAt gating and the metadata timestamp so the logged eval time matches the allocation selection decision - Rename metadata key dd.eval.timestamp_ms -> __dd_eval_timestamp_ms to signal internal-only intent; update FlagEvalLoggingHook reader and tests in lockstep - Move the isEmpty/droppedQueueOverflow guard into shouldFlush() so flushIfNecessary() is a single-responsibility delegate Co-Authored-By: Claude <noreply@anthropic.com>
…Error only The OpenFeature SDK's executeAfterAllHooks already wraps finallyAfter calls in a catch(Exception) and logs them. Our Exception catch was redundant. LinkageError is kept because the SDK catches Exception, not Throwable, leaving classloading failures unguarded. Co-Authored-By: Claude <noreply@anthropic.com>
…t logging At shutdown there is nowhere to surface the error usefully, and each writer already logs internally on failure. Debug-logging from the catch blocks was misleading noise. True swallow (Exception ignored) matches the intent of the original suggestion. Co-Authored-By: Claude <noreply@anthropic.com>
…ingSystem.stop() Replaces four identical try/catch blocks with a single closeQuietly helper. SpanEnrichmentWriter gains AutoCloseable (it already had a close() method) so all four resources share the same call site. Co-Authored-By: Claude <noreply@anthropic.com>
…tionAggregator EVAL_SCALE_ prefix was ambiguous — it read as a runtime scale factor rather than design-time sizing assumptions. Split into two named groups: - EXPECTED_* for the design assumptions (flag count, users per flag, etc.) - *_SIZING_BASIS for the derived intermediate values - Inline comments on GLOBAL_CAP and DEGRADED_CAP explain they are the nearest powers of two above the respective sizing bases. Co-Authored-By: Claude <noreply@anthropic.com>
The multi-pass loop with Thread.yield() between passes was a heuristic for catching producers mid-enqueue during shutdown. A single poll loop already drains everything in the queue at that point. For events that race past the drain, close() sweeps the queue after joining the worker and counts any remainder as an observable drop — making extra passes redundant. Removed SHUTDOWN_DRAIN_PASSES constant and the loop. Co-Authored-By: Claude <noreply@anthropic.com>
The lazy Supplier overload and contextAttributes() accessor were removed in 29b974d, breaking compileTestJava. Drop the two tests that covered the deleted lazy path and remove the contextAttributes() assertions from the remaining tests. Co-Authored-By: Claude <noreply@anthropic.com>
…verload The lazy Supplier constructor was removed in 29b974d; pass attrs directly. Also rename nextLazyEvent -> nextEvent in FlagEvaluationHotPathBenchmark. Co-Authored-By: Claude <noreply@anthropic.com>
Commit 29b974d removed the lazy-supplier path and took two tests with it, dropping FlagEvaluationWriterImpl branch/instruction coverage below the 90% Jacoco threshold on Java 8. Add two replacement tests: - scoConstructorCreatesUsableWriter: exercises the public SCO constructor - countContextTruncatedAccumulatesPerReason: exercises countContextTruncated Co-Authored-By: Claude <noreply@anthropic.com>
The pre-queue guard methods were only exercised through mocks in the hook test, leaving FlagEvaluationWriterImpl branch coverage at 0.844 and failing jacocoTestCoverageVerification (threshold 0.9). Add a direct test that observes both branches of hasCapacityForEnqueue and the countPreQueueOverflow counter surfacing as a queue_overflow drop metric. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
sabrenner
left a comment
There was a problem hiding this comment.
llmobs span mapper changes lgtm - i believe this follows our node.js & python limits as well
…ions events (#12042) * Parse observeFullEvaluationData and hash targeting_key in flagevaluations events Adds the top-level observeFullEvaluationData boolean to the UFC model, plumbs it through to the EVP flagevaluation event serializer, and gates PII handling on it: when the flag is absent/false the targeting key is SHA-256 hashed (sha256_<hex>) and the raw evaluation context is omitted from the wire; when true the raw targeting key and context are emitted. Environment: Datadog workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Extract hashed targeting key prefix into a named constant Replace the inline "sha256_" literal with a documented HASHED_TARGETING_KEY_PREFIX constant describing the cross-SDK wire contract for privacy-preserving hashed targeting keys. Environment: Datadog workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Test observeFullEvaluationData parsing edge cases Parameterize the true/false config-parsing assertions with @valuesource and add a test locking in the fail-closed behaviour for an explicit JSON null: malformed config is rejected so full evaluation data is never observed off the back of it. Environment: Datadog workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Capture observeFullEvaluationData per bucket at aggregation time The flush-time read of FeatureFlaggingGateway.isObserveFullEvaluationDataEnabled() was a TOCTOU bug: CURRENT_CONFIG could be overwritten by a later RC update between when an evaluation happened and when the batch flushed, so events could be emitted under the wrong environment's consent (the system test observed a targeting key hashed even though the active UFC said observeFullEvaluationData=true). Capture consent when the evaluation is folded into its EvalBucket instead. On merge the value is folded with AND, so any no-consent evaluation in a bucket's lifetime sinks the whole bucket to hashed/omitted (fail-closed). buildEventList now reads bucket.observeFullEvaluationData rather than the gateway. The gateway accessor is retained; it is read at aggregation time. Adds a writer-level regression guard (a bucket aggregated under consent-off stays hashed even if the gateway later reports consent-on) plus aggregator fold tests, and an end-to-end parse->dispatch->flush test. Environment: Datadog workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Capture observeFullEvaluationData consent at evaluation time Snapshot the PII consent flag on the evaluation thread (in the OpenFeature hook) and carry it on FlagEvalEvent, instead of reading the gateway when the event is aggregated/flushed. This pins the hashed-vs-raw decision to the configuration active at evaluation time, closing a one-directional leak window where a later Remote Config update could retroactively apply a different environment's consent to already-collected evaluations. Aggregation and flush now read event.observeFullEvaluationData and never consult the gateway; the AND-fold across a bucket's evaluations is unchanged (any no-consent evaluation sinks the bucket to hashed/omitted). Environment: Datadog workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Bind observeFullEvaluationData consent to the evaluator's configuration Address PR #12042 review feedback (Codex P1, leoromanovsky, dd-oleksii): the FlagEvalLoggingHook was reading observeFullEvaluationData from FeatureFlaggingGateway.isObserveFullEvaluationDataEnabled() at hook-fire time, which races against a Remote Config swap of CURRENT_CONFIG that happens after DDEvaluator.evaluate() captured its own ServerConfiguration reference. That race can retroactively mark an evaluation performed without consent as consented and leak the raw targeting key / context. DDEvaluator now stamps the boolean directly from the ServerConfiguration it used, onto every ProviderEvaluation via ImmutableMetadata under key "dd.observe_full_evaluation_data". The hook reads consent from that metadata and no longer queries the gateway. Missing metadata (PROVIDER_NOT_READY or a non-DD provider) → false, the privacy-preserving default. The gateway's isObserveFullEvaluationDataEnabled() accessor is removed since its only real caller was the hook and re-adding it would re-open the race. Adds regression tests: hook honours consent metadata (true/false/absent) and ignores a gateway value that disagrees; evaluator stamps the correct boolean on the FLAG_NOT_FOUND path and omits metadata when it holds no config. Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Pass the boolean, not the ServerConfiguration, into error/resolveVariant Follow-up to the previous commit: the private error() and resolveVariant() helpers only ever read one field off the ServerConfiguration (observeFullEvaluationData), so pass the boolean directly instead of the whole config. Keeps the internal API narrow and removes the incidental coupling these helpers had to the UFC. While here, PROVIDER_NOT_READY now stamps consent as the privacy-preserving false rather than omitting the metadata. Same on-the-wire outcome the hook would have produced, but the invariant "every DD-produced evaluation carries dd.observe_full_evaluation_data" is now unconditional, which is easier to reason about. The two error() overloads collapse to one (the (String) null casts at call sites disappear along with them). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Drop the dd. prefix on the evaluation-metadata consent key The key is only ever read by FlagEvalLoggingHook one line later — it never lands on the wire, so it doesn't need the "dd." namespacing that "dd.eval.timestamp_ms" has (that key is re-emitted onto spans). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Trim verbose comments around the observeFullEvaluationData plumbing The race-vs-CURRENT_CONFIG backstory is captured in the previous commits' messages; the code only needs the forward-looking invariants (metadata is source of truth, missing key = false, DD-produced evaluations always stamp). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Skip evaluation context when aggregating consent-off evaluations Address PR #12042 review from leoromanovsky (escalated Codex P2 → P1): on the protected path (observeFullEvaluationData=false) the serializer drops the evaluation context, but the aggregator was still running it through pruneContext + canonicalContextKey and keying every full-tier bucket on it. A high-cardinality field on the evaluation context (request_id, timestamp, correlation id) would fragment buckets that emit byte-identical wire rows, blow out PER_FLAG_CAP (10k) inside one flush window, and force subsequent evaluations into the degraded tier — which drops the targeting key entirely. On the protected path aggregate() now uses ctxKey="" and stores prunedAttrs=null, so different contexts for the same subject collapse into one bucket. The targeting key stays in the aggregation identity, so different subjects still hash to different buckets. The consent-on path is unchanged. Regression tests: protected path collapses differing contexts for one subject; protected path still separates distinct subjects; full path still splits on context. Existing tests that exercise pruneContext / context-differentiation were updated to use consent=on (that's the code path they actually cover). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Skip evaluation-context capture on the hook hot path when consent is off Companion to the aggregator fix: with observeFullEvaluationData=false the evaluation context is dropped on emit and no longer influences aggregation, so there is no reason to snapshot it on the evaluation thread. The hook now branches on consent up front — the protected path enqueues an event with an empty materialized attrs map (no map copy of the OpenFeature context, no Supplier<Map> allocation, no lambda instance), while the consent-on path is unchanged. Grep confirms the only production consumer of FlagEvalEvent.contextAttributes / FlagEvalEvent.attrs is FlagEvaluationAggregator.aggregate, which already skips them on the protected path. Regression test: mutating the EvaluationContext after finallyAfter returns still yields empty attrs on the enqueued event — proves the hook never snapshotted it. Two existing tests that exercise the snapshot mechanism were switched to pass consent-on metadata (that's the code path they cover). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Include observeFullEvaluationData in the aggregation bucket key Bucket keys should cover every dimension the emitter will branch on. The serializer branches on observeFullEvaluationData (hashes the targeting key and drops the context when off), so two evaluations that differ only in consent produce different wire rows and must not share a bucket. Before this change they could: same subject, same flag, same empty context would land under the same FullKey regardless of consent, and the AND-fold would silently downgrade a consent-on evaluation to the protected wire shape because a nearby consent-off event merged into its bucket first. No PII leak (fail-closed direction), but arrival-order-dependent semantics and a lost raw-context row. Add observeFullEvaluationData to FullKey / DegradedKey (equals + hashCode). The AND-fold on bucket.observeFullEvaluationData stays as defensive belt- and-suspenders; every event merging into a bucket now carries the matching consent value by construction. Regression test: two events identical except for consent land in two full- tier buckets, one consent-on and one consent-off. Updated the previous "fold to false on mixed consent" test to reflect the new invariant. Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add consent metadata to ProviderTest flag-eval-logging hook route test The test asserts that context attributes flow through the logging hook, but the mock metadata omitted the observe-full-evaluation-data flag, so the hook took the privacy-preserving path and dropped context. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Redact error messages when observeFullEvaluationData is off Exception messages from the evaluator's outer catch blocks (NumberFormatException, generic Exception) can echo raw evaluation-context values verbatim — for example a GT rule on "id" with a PII-shaped targeting key produced error.message="For input string: \"jane.doe@...\"" on the wire regardless of consent, defeating the PR's own PII guard. Drop the message at DDEvaluator.error() when consent is off, and add a hook-layer fallback that substitutes ErrorCode.name() so operators keep a stable signal (e.g. "TYPE_MISMATCH") even when a third-party provider hands us a raw message. Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Exercise every consent-stamp code path in DDEvaluatorTest The only observeFullEvaluationData assertions were on error paths (FLAG_NOT_FOUND, PROVIDER_NOT_READY), leaving the success-path stamp in resolveVariant and the DISABLED/DEFAULT stamps in consentMetadata uncovered — line 448 could be deleted or hardcoded to either value and every existing test would still pass. Add symmetric consent-on/consent-off tests for each of resolveVariant, DISABLED, and DEFAULT so any mutation (delete / hardcode true / hardcode false) flips at least one assertion. Rename the previously misleading …OnSuccess test to reflect what it actually exercises (FLAG_NOT_FOUND error via error()). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Drop consent from DegradedKey to reclaim effective DEGRADED_CAP Two degraded buckets differing only in observeFullEvaluationData emit byte-identical wire JSON — the degraded serializer (fromBucket with isFullTier=false) drops the targeting key and context regardless of consent — so the consent dimension in DegradedKey halved effective DEGRADED_CAP for zero wire fidelity gain. FullKey correctly keeps consent (the full-tier serializer branches on it for raw-vs-hashed targeting key and context inclusion). Mixed-consent events now merge into one degraded bucket. The AND-fold on EvalBucket.observeFullEvaluationData still runs and collapses to false whenever any consent-off event lands in a mixed bucket; benign because the value has no downstream effect for degraded rows. Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Tolerate malformed observeFullEvaluationData in UFC parse Before this change ServerConfiguration.observeFullEvaluationData was a primitive boolean — Moshi's reflective adapter rejected the entire UFC whenever the JSON value was null or wrong-typed. Agentless swallows the IOException at DEBUG, so a pod starting after a malformed message had no last-known-good, stranded every flag on PROVIDER_NOT_READY, and served defaults forever. Fail-closed on privacy shouldn't cascade into fail-closed on availability. Box the field to Boolean so null tolerates naturally, register a LenientBooleanAdapter that maps wrong-typed values to null as well, and read via Boolean.TRUE.equals(...) at the DDEvaluator so null falls to the privacy-preserving default. The lenient adapter only intercepts Boolean (not primitive boolean), so mandatory fields like Flag.enabled keep their strict parse; the only other Boolean it touches is Allocation.doLog, which is already read as `!= null && doLog`. Reversed the earlier RejectsExplicitNull test — it had locked in the buggy behaviour — into a family of tolerance tests for null / stringified / numeric. Added a DDEvaluator test that a config with a null consent field evaluates without NPE and stamps the privacy-preserving default. Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Set observeFullEvaluationData=true for the NaN-poison flush test The consent-off short-circuit in FlagEvaluationEvent.fromBucket drops the raw context before Moshi encodes it, so a NaN in the attrs never reaches the encoder and the flush succeeds. That defeated the intent of encodeFailureClearsAggregatorSoLaterFlushesRecover, which must observe a real encode failure to prove the aggregator is cleared. Co-Authored-By: Claude <noreply@anthropic.com> * Cover LenientBooleanAdapter read-only and qualifier paths The per-class JaCoCo gate (0.9 minimum, gradle/jacoco.gradle) failed on the new adapter: toJson was never invoked (20/25 instructions) and the factory's !annotations.isEmpty() short-circuit never evaluated true (3/4 branches). Neither path is reachable through the parse-driven tests in JsonApiUfcResponseParserTest. Mirror the tests the sibling FlagMapAdapter and DateAdapter already have. The primitive-boolean assertion documents the guard that keeps this leniency off mandatory fields like Flag.enabled. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
📢 FYI I merged #12042 into this branch so we have PII protected when we merge to
|
| } | ||
|
|
||
| private static void appendLengthDelimited(final StringBuilder sb, final String s) { | ||
| sb.append(String.format("%08x", (long) s.length())); |
There was a problem hiding this comment.
String.format() in a loop is very GC heavy and not recommended.
Consider to have code like:
String hexLength = Integer.toHexString(s.length());
sb.append("00000000", 0, 8 - hexLength.length());
sb.append(hexLength);
sb.append(s);
Or even move this `format with leading zeroes to small utility function.
| public int hashCode() { | ||
| return Objects.hash( | ||
| flagKey, | ||
| variant, | ||
| allocationKey, | ||
| runtimeDefaultUsed, | ||
| errorMessage, | ||
| targetingKey, | ||
| contextKey); | ||
| } | ||
| } |
There was a problem hiding this comment.
Consider to use our hashing utility here:
import static datadog.trace.util.HashingUtils.addToHash;
import static datadog.trace.util.HashingUtils.hash;
...
@Override
public int hashCode() {
int result = hash(flagKey, variant, allocationKey);
result = addToHash(result, runtimeDefaultUsed);
result = addToHash(result, errorMessage);
result = addToHash(result, targetingKey);
return addToHash(result, contextKey);
}
This avoids not needed array allocation, and in some situation not needed boxing/unboxing.
| public int hashCode() { | ||
| return Objects.hash(flagKey, variant, allocationKey, runtimeDefaultUsed, errorMessage); | ||
| } |
There was a problem hiding this comment.
Same here, refactor to HashingUtils
| final Set<Object> seen = Collections.newSetFromMap(new IdentityHashMap<>()); | ||
| final int[] reasonMask = {0}; | ||
| for (final String key : keys) { | ||
| if (out.size() >= MAX_CONTEXT_FIELDS) { |
There was a problem hiding this comment.
Optional, but probably worth to guard agains.
What if we have a lot of keys and most of them will violate limitations?
In this case condition out.size() will be false and we have to iterate over all keys.
Probably that is intended, but maybe worth to detect such cases and have warning in logs (make sure not to flood logs though). Like imagine that you have 1000 keys and 999 first of them rejected and only one last processed.
🟢 NOTE TO REVIEWERS
I've chosen to keep this PR on the larger side in terms of "lines of code" as a test. The commits are deliberately layered in a narrative style. Each one is equivalent to what we would normally do with a "stacked" PR, but this preserves the overall view of the feature. Please review one commit at a time or all together.
If this review mechanism is not satisfactory, please let me know!
Stack Position
You are here: the second Java layer, stacked directly on #11892. It adds aggregate
flagevaluationEVP independently of whether UFC arrived through agentless HTTP or Agent Remote Configuration.flowchart LR subgraph JAVA["dd-trace-java"] J1["JAVA-01 · #11892<br/>Agentless + RC sources"] --> J2["JAVA-02 · #11639<br/>Aggregate evaluation EVP"] end subgraph SYSTEM["system-tests"] ST1["ST-01 · #7298<br/>Mock agentless backend"] --> ST2["ST-02 · #7299<br/>Side-effect contracts"] ST2 --> STM1["ST-M01 · #7300<br/>Enable Java configuration"] ST2 --> NEXT["Next drafts<br/>Enable Java side effects"] end subgraph DOGFOOD["ffe-dogfooding"] DOG0["DOG-00 · #92<br/>Agentless evaluation baseline"] --> DOG1["DOG-01 · #93<br/>Side-effect conduit"] end J1 --> STM1 J1 --> DOG0 J2 --> NEXT J2 --> DOG1 STM1 --> GREEN["Java proof<br/>both sources × side effects"] NEXT --> GREEN DOG1 --> GREEN classDef current fill:#fcbf49,stroke:#8a5a00,stroke-width:3px,color:#111; class J2 current;Motivation
Customers need consistent server-side feature-flag evaluation visibility across supported runtimes so rollout behavior can be correlated with application behavior in APM and Event Platform. This Java contribution adds that server-side
flagevaluationsignal for Java OpenFeature evaluations while preserving the existing OTelfeature_flag.evaluationspath and the existing exposure telemetry path.High Priority Changes and Decisions
These are the design points I would want reviewed most closely.
EVP routing uses the Agent-advertised proxy prefix, not a hard-coded v2/v3/v4 path. The SDK keeps the track route as
/api/v2/flagevaluation, but builds it under the proxy prefix discovered from the Agent. In current staging dogfooding that resolves to/evp_proxy/v4/api/v2/flagevaluation.Flagevaluation reuses the existing Event Platform publisher path. This keeps delivery aligned with existing Agent discovery, headers, lifecycle, and compression controls instead of adding a Java-specific HTTP writer. Response compression is disabled for this track to match the merged Go behavior.
The OpenFeature hook runs inline, and the context snapshot — not scalar extraction — dominates its cost.
finallyAfterextracts scalar evaluation metadata, then deep-copies the caller's evaluation context viaDDEvaluator.snapshotValues(a freshValueper attribute, recursiveArrayList/ImmutableStructurecopies for nested lists and structures, plus anIdentityHashMapfor cycle detection), then does a non-blocking enqueue. Only flattening, aggregation, and posting are deferred to the worker thread. This is the main correctness/performance boundary.To be explicit, since review flagged this: the hook's javadoc previously documented its contract as doing "ONLY cheap scalar extraction," and this section previously described the hook as one that "only captures and enqueues." Both understated the inline cost. Measured on the new
FlagEvalHookHotPathBenchmark, scalar extraction plus enqueue costs ~75 ns, while the inline snapshot ranges from ~119 ns (nothing to copy) to ~9.9 µs (10 nested structures × 10 fields) — the snapshot is 91–99.5% of the hook's inline cost once a context carries attributes, and inline work remains 21–47% of total per-evaluation work even after deferral. The javadoc claim has been corrected in code and is quantified below.The copy itself is unavoidable:
EvaluationContextis caller-owned and mutable, so its values must be captured before the event is handed to the writer thread. Callers passing large or deeply nested evaluation contexts should expect single-digit-microsecond inline cost per evaluation.The worker emits the existing batched flagevaluation contract. One flush produces a
FlagEvaluationsRequestwith top-levelcontextand aflagEvaluationsarray; this does not use a separate/batchedflagevaluationsroute.Aggregation keys are limited to schema-visible fields. The aggregate dimensions are flag key, variant key, allocation key, runtime-default state, error message, targeting key, and pruned context. OpenFeature
reasonis intentionally not a hidden aggregate key because it is not serialized to the worker contract.targeting_keyis the single identity field for the event. The hook removes duplicatetargetingKeyfromcontext.evaluationso the same identity is not encoded twice.Cardinality/backpressure behavior is intentionally lossy but counted. The writer uses full-fidelity buckets first, degraded buckets without targeting key/context after cap pressure, then counted drops if both tiers or payload limits are exhausted.
Event time and send time are separate. Each aggregate row preserves
first_evaluationandlast_evaluationbounds, while the payloadtimestampis the flush time.Other Changes
flagevaluationpath behindDD_FLAGGING_EVALUATION_COUNTS_ENABLEDwhile leaving the existing OTelfeature_flag.evaluationshook in place.FlagEvalHookHotPathBenchmark(feature-flagging-api) — the true evaluation-thread cost offinallyAfter, including the context deep copy, across flat/nested/list context shapes. This is new in response to review feedback.FlagEvaluationHotPathBenchmark(feature-flagging-lib) — writer queue mechanics and worker-thread aggregation.evalThreadCapturemethod is renamedwriterEnqueueto stop implying it covered the hook's inline cost. The hook lives infeature-flagging-apiand its OpenFeature context types are not on the lib's classpath, which is why the inline cost has to be measured in a second module.Commit Guide
LOC is rename-aware
git diff-tree -M --numstatfor each commit against its parent.571938b6c74949ed7ee44aa06476b7ac126fb44a7a43658da64ec77cae036fa7ade3e4986a9af6bef1c37aba16115a407b76cc3b21c0c231d7cb5116FlagEvaluationsRequestpayloads, split oversized bodies, degrade oversized rows, and count drops.e43f93acd017d77854205f88cadeb2597e97d10b58df936165acc6e2194096df04f1beedb264e7d890ed6cb556c73cafdb465b1456b3fbValidation Evidence
Hot-Path Cost (JMH)
./gradlew :products:feature-flagging:feature-flagging-api:jmh -PjmhIncludes=FlagEvalHookHotPathBenchmark(JDK 11.0.31, aarch64, 3×2s warmup / 5×1s measurement, single fork — relative magnitudes are the point, not absolute numbers.)
hookFinallyAfter(inline, ns/op)contextSnapshot(inline copy, ns/op)deferredFlatten(worker, ns/op)flat/0attrsflat/10attrsflat/100attrsnested/10structs_10fieldslist/10lists_10itemsReadings:
hookFinallyAfterminuscontextSnapshotatflat/0attrs). Everything above that is the context copy.flat/100attrs,nested/10structs_10fields, andlist/10lists_10itemsall carry 100 leaf values, but inline cost spans 3.0–9.9 µs. Nested structures are worst (~1.6× flat) because each one allocates an innerHashMapplus anImmutableStructurewrapper on top of the leaf copies; lists are cheapest (~0.5× flat) because 100 leaves sit under only 10 top-level map entries.flat/0attrsspends ~119 ns snapshotting a context with nothing to copy, because the cycle-detectionIdentityHashMapis allocated unconditionally. Tracked as a follow-up optimization (alias immutable scalarValues, allocate the cycle set lazily) rather than changed here —snapshotValuesis shared with the already-shipped exposure path, so it is out of scope for this PR.¹
contextSnapshotmeasuring marginally abovehookFinallyAfterfor the list shape is fork-to-fork variance on what is effectively the same work, not a negative-cost hook; the two are within ~3.5% of each other.Local Test Gates
origin/master(ac29db2316)::communication:test:products:feature-flagging:feature-flagging-api:test:products:feature-flagging:feature-flagging-agent:test:products:feature-flagging:feature-flagging-lib:test:products:feature-flagging:feature-flagging-lib:jmhClassesBackendApiFactoryTest,DDAgentFeaturesDiscoveryTestDDEvaluatorTest,ProviderTest,FlagEvalLoggingHookTestFeatureFlaggingSystemTestFeatureFlagEvpPublisherTest,FlagEvaluationAggregatorTest,FlagEvaluationPayloadsTest,FlagEvaluationWriterImplTest./gradlew :products:feature-flagging:feature-flagging-lib:test :products:feature-flagging:feature-flagging-lib:jacocoTestReport :products:feature-flagging:feature-flagging-lib:jacocoTestCoverageVerification./gradlew spotlessApply./gradlew spotlessCheck./gradlew :products:feature-flagging:feature-flagging-lib:spotlessCheck./gradlew :communication:forbiddenApis :dd-trace-core:forbiddenApis :internal-api:forbiddenApis :telemetry:forbiddenApis :products:feature-flagging:feature-flagging-api:forbiddenApis :products:feature-flagging:feature-flagging-agent:forbiddenApis :products:feature-flagging:feature-flagging-lib:forbiddenApisgit diff --checkpassed.Dogfooding App
ffe-dogfoodingJava artifacts from this localdd-trace-javastack withscripts/prepare-local-java.sh.dd-openfeatureanddd-java-agentartifacts plus the real backend EVP path.PROVIDER_READY.ffe-dogfooding-string-flagthrough the Java dogfooding app 15 times total: 5 evaluations for each targeting key:java-restack4-20260702T042247Z-alphajava-restack4-20260702T042247Z-bravojava-restack4-20260702T042247Z-charlievariant_1, allocationallocation-override-392dd7c149f8, servicejava, and evaluation reasonSTATIC.http://datadog-agent:8126/evp_proxy/v4/api/v2/flagevaluation, both returning202.Staging End-To-End
eventplatform.system.track(TRACK => 'flagevaluation')returned 3 aggregated rows for the exact targeting keys above.flag.key=ffe-dogfooding-string-flagvariant.key=variant_1allocation.key=allocation-override-392dd7c149f8evaluation_count=5System Tests
6b7aa4273d:TEST_LIBRARY=java ./run.sh +v FEATURE_FLAGGING_AND_EXPERIMENTATION tests/ffe/test_flag_eval_evp.py8 passed in 80.08s(Library: java@1.64.0-SNAPSHOT+6b7aa4273d,Weblog variant: spring-boot).Integration Addendum
JAVA-01 is the direct PR base and is preserved as an ancestor through signed merge commit
d5b90bad24.FeatureFlaggingSystemnow selects the configured source first, then starts exposure and aggregate-evaluation writers independently for agentless, Remote Configuration, and reserved offline source modes.Validation Evidence
dd-java-agent,dd-trace-api, anddd-openfeature: pass.CI Packaging Decision
Any custom Java system-test build must install all three artifacts. Supplying only
dd-java-agentanddd-trace-apileaves the weblog on the publisheddd-openfeatureimplementation and does not exercise the PR head.