Add agentless Feature Flagging configuration source by leoromanovsky · Pull Request #11892 · DataDog/dd-trace-java

leoromanovsky · 2026-07-08T17:33:56Z

Motivation

Java Feature Flagging needs a tracer-side configuration-source split so SDKs can fetch UFC from an agentless HTTP backend while preserving the existing Agent Remote Configuration path.

Changes

Adds DD_FEATURE_FLAGS_CONFIGURATION_SOURCE with agentless, remote_config, and reserved offline; default is agentless.
Adds agentless endpoint selection through DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL.
Adds agentless polling controls: DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_POLL_INTERVAL_SECONDS and DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_REQUEST_TIMEOUT_SECONDS.
Adds AgentlessConfigurationSource, an HTTP UFC poller used for both the default Datadog-managed endpoint and a configured custom HTTP endpoint.
Keeps remote_config on the existing RemoteConfigServiceImpl / Agent RC path.
Handles DD-API-KEY, accepted-200-only ETags/304, malformed UFC rejection, last-known-good preservation, and no overlapping polls.
Retries timeout/429/5xx failures twice per poll cycle with jittered delays derived from the poll period: clamp(P/6, 2s, 10s) then clamp(P/3, 5s, 30s).
Cancels in-flight HTTP and pending retry work during shutdown, suppresses post-shutdown retries, and makes subsystem startup idempotent.
Rejects nonpositive poll intervals and request timeouts by warning and using the documented defaults.
Removes the proposed DD_FEATURE_FLAGS_ENABLED and extra-header env from this PR.

Decisions

No new provider kill switch. This keeps the existing provider bootstrap behavior through DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED; changing enablement is out of scope for this PR.

Custom HTTP delivery remains under agentless. The source mode describes direct HTTP UFC delivery without the Datadog Agent. With no base URL, the SDK derives the first-party Datadog endpoint as https://api.<site>/api/v2/feature-flagging/config/server-distribution?dd_env=<env>. Setting DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL chooses a system-test, dogfood, or operator-managed HTTP endpoint without introducing a fourth source mode. A bare host uses the standard server-distribution path; a URL with a path is used as the exact UFC endpoint.

Remote Configuration remains a distinct mode. It is Agent-mediated and uses the RC protocol's security, signing, targeting, capability, and subscription lifecycle. Those semantics are materially different from direct HTTP polling, so remote_config remains a peer source mode rather than another agentless endpoint choice.

No offline factory yet. offline remains a reserved configuration-source value. This PR does not expose a startup-bytes API or offline factory because that API still needs design work.

Source selection stays SDK-local. DD_FEATURE_FLAGS_CONFIGURATION_SOURCE decides which local source implementation starts. It is not sent to the backend.

Initialization Modes

Solid connectors are implemented in this PR. The dashed offline connector previews startup-provided UFC bytes.

flowchart TD
    Gate{DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED}
    Disabled[Feature Flagging subsystem not started]
    System[FeatureFlaggingSystem.start]
    Select{DD_FEATURE_FLAGS_CONFIGURATION_SOURCE}
    Exposure[ExposureWriter starts for every enabled mode]

    subgraph AgentlessMode[agentless mode - current default]
        AgentlessSource[AgentlessConfigurationSource]
        OperationalDefaults[SDK-owned operation<br/>30s poll default; 2s request timeout default<br/>ETag; retry; last-known-good; no overlap]
        EndpointChoice{agentless base URL set?}
        DatadogManaged[Datadog-managed agentless<br/>first-party; CDN-backed]
        CustomHttp[Custom HTTP endpoint<br/>system tests; dogfood; operator backend]

        AgentlessSource --- OperationalDefaults
        AgentlessSource -- polls --> EndpointChoice
        EndpointChoice -- no --> DatadogManaged
        EndpointChoice -- yes --> CustomHttp
    end

    RemoteConfig[remote_config<br/>explicit opt-in; Agent-mediated]
    RemoteConfigSource[RemoteConfigServiceImpl]
    AgentRc[Datadog Agent Remote Configuration<br/>RC security; signing; targeting; subscriptions]

    OfflineReserved[offline today<br/>reserved; no configuration service]
    OfflineSource[Later: OfflineConfigurationSource<br/>customer UFC JSON bytes at startup; no network]

    Ufc[Shared UFC deserialize and evaluate pipeline]
    Gateway[FeatureFlaggingGateway]
    Provider[OpenFeature provider]

    Gate -- false --> Disabled
    Gate -- true --> System
    System --> Select
    System --> Exposure

    Select -- unset or agentless --> AgentlessSource
    DatadogManaged --> Ufc
    CustomHttp --> Ufc

    Select -- remote_config --> RemoteConfig
    RemoteConfig --> RemoteConfigSource
    RemoteConfigSource -- subscribes --> AgentRc
    RemoteConfigSource --> Ufc

    Select -- offline today --> OfflineReserved
    OfflineReserved -. later .-> OfflineSource
    OfflineSource -. startup UFC bytes .-> Ufc

    Ufc --> Gateway
    Gateway --> Provider

Customer Usage Example

Default Datadog-managed agentless delivery:

export DD_API_KEY=<datadog-api-key>
export DD_SITE=datadoghq.com
export DD_ENV=prod

# Optional because agentless is the default configuration source.
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=agentless
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_POLL_INTERVAL_SECONDS=30
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_REQUEST_TIMEOUT_SECONDS=2

java -javaagent:/path/to/dd-java-agent.jar -jar app.jar

Custom HTTP delivery keeps agentless mode and changes only the endpoint:

export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=agentless
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL=https://flags.example.com/ufc

java -javaagent:/path/to/dd-java-agent.jar -jar app.jar

Explicit Agent Remote Configuration delivery still uses the existing Agent path:

export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=remote_config
export DD_REMOTE_CONFIGURATION_ENABLED=true

java -javaagent:/path/to/dd-java-agent.jar -jar app.jar

Verification

./gradlew :products:feature-flagging:feature-flagging-agent:test :products:feature-flagging:feature-flagging-lib:test --no-daemon
result: BUILD SUCCESSFUL

./gradlew :products:feature-flagging:feature-flagging-agent:jacocoTestCoverageVerification :products:feature-flagging:feature-flagging-lib:jacocoTestCoverageVerification :products:feature-flagging:feature-flagging-agent:spotlessCheck :products:feature-flagging:feature-flagging-lib:spotlessCheck --no-daemon
result: BUILD SUCCESSFUL

./gradlew :dd-smoke-tests:openfeature:cleanTest :dd-smoke-tests:openfeature:test --tests datadog.smoketest.springboot.OpenFeatureProviderSmokeTest --no-daemon
result: BUILD SUCCESSFUL

git diff --check
result: clean

System Test Evidence

I built dd-java-agent, dd-trace-api, and dd-openfeature from the rewritten
head f9e70e67c4, rebuilt the Java Spring Boot weblog image with those local
artifacts, and verified the JAR inside the image matched the local agent JAR.
I then ran the Java manifest-enabled agentless scenario from the companion
system-tests draft:

TEST_LIBRARY=java ./run.sh +v \
  FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS \
  tests/ffe/test_agentless_configuration.py

Scenario: FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS
Library: java@1.65.0-SNAPSHOT+f9e70e67c4
Weblog variant: spring-boot
collected 1 item
tests/ffe/test_agentless_configuration.py .                        [100%]
1 passed

This exercises the Java manifest row proposed by the companion draft against
the exact code currently published in this PR.

Next Steps

Review the draft Java manifest activation:
DataDog/system-tests#7300.
Keep that PR draft until this Java behavior is available to system-tests CI.
After this PR lands, rerun the manifest activation in Java development CI and
merge the activation only when that gate is green.

dd-octo-sts · 2026-07-08T17:57:28Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.09 s	13.97 s	[+0.1%; +1.6%] (maybe worse)
startup:insecure-bank:tracing:Agent	13.03 s	13.06 s	[-1.0%; +0.6%] (no difference)
startup:petclinic:appsec:Agent	16.97 s	16.62 s	[+1.1%; +3.1%] (significantly worse)
startup:petclinic:iast:Agent	16.94 s	16.96 s	[-0.9%; +0.7%] (no difference)
startup:petclinic:profiling:Agent	16.66 s	16.77 s	[-1.7%; +0.4%] (no difference)
startup:petclinic:sca:Agent	17.07 s	16.90 s	[+0.1%; +1.9%] (maybe worse)
startup:petclinic:tracing:Agent	15.68 s	16.00 s	[-6.2%; +2.1%] (no difference)

Commit: 1c4a2234 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

dd-octo-sts · 2026-07-12T12:10:36Z

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

datadog-datadog-prod-us1-2

All retry, ETag, shutdown, and idempotency paths look correct. One real gap: apply() silently returns false for 401/403 with no log above DEBUG — users with a wrong or missing DD_API_KEY have no way to diagnose why feature flags never load. Fixed by separating the auth-failure branch in apply() with a LOGGER.warn(...) call.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit f9e70e6 · What is Autotest? · Any feedback? Reach out in #autotest}

…figuration-source

leoromanovsky · 2026-07-24T05:48:01Z

Review update

This update applies the decisions from this review and from the cross-SDK review.

Changes

The agent does not start Agentless polling during tracer startup. It waits until application code initializes the OpenFeature provider. The first poll cycle completes during provider activation. Later polls use a fixed delay.
An invalid or unsupported configuration source does not cause an exception. The agent disables the provider and writes a warning.
The request timeout default is now 5 seconds.
Agentless requests use the shared OkHttpUtils.sendWithRetries and HttpRetryPolicy implementation.
The client makes a maximum of three attempts. It uses the same retry ranges and jitter as the Node.js implementation.
A custom endpoint can use HTTP for local development. The client does not send the Datadog API key to a custom endpoint.
The client sends the Datadog API key only to the default Datadog HTTPS endpoint.
HTTP 401 and 403 responses write an authentication warning.

System-test evidence

I built these artifacts from Java commit 6df48d1cfd:

dd-java-agent-1.65.0-SNAPSHOT.jar
dd-trace-api-1.65.0-SNAPSHOT.jar
dd-openfeature-1.65.0-SNAPSHOT.jar

I used system-tests main at 03f0c053c. I removed the Java missing_feature waiver locally. I rebuilt the parametric image with the three local artifacts. I then ran the complete configuration-source test file.

PYTEST_XDIST_AUTO_NUM_WORKERS=1 TEST_LIBRARY=java \
  ./run.sh PARAMETRIC --skip-parametric-build \
  tests/parametric/test_ffe/test_configuration_sources.py

29 passed in 290.19s

The affected Java tests and formatting checks also passed:

BUILD SUCCESSFUL in 29s
298 actionable tasks: 7 executed, 291 up-to-date

Node.js alignment

Unsupported sources disable the provider in Java and Node.js.
Agentless polling starts when application code initializes the provider in Java and Node.js.
Both clients use three attempts, the same retry ranges, and 20 percent jitter. See Java and Node.js.
Both clients allow custom HTTP endpoints and omit the API key for those endpoints. See Java and Node.js.
Both default endpoints use HTTPS. See Java and Node.js.
Both clients use a 5-second request timeout by default. See Java and Node.js.

The Node.js links use the current head of #9482. That PR includes the endpoint behavior from merged #9481.

…figuration-source

scheduledPollContinuesAfterListenerRuntimeException waited on FakeClient.calls, which is incremented when a request starts rather than when it completes. The barrier therefore released as soon as the second poll began, letting the assertion race the poll thread that applies the configuration and notifies the listener. Wait on a CountDownLatch counted down by the listener itself, so the second notification is guaranteed to have happened before the assertions run. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: vickie.fridge <vickie.fridge@datadoghq.com>

This reverts commit 29c1fd4.

…-sdk-contract

dd-octo-sts · 2026-07-28T19:37:57Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-07-28T19:38:01Z

View all feedbacks in Devflow UI.

2026-07-28 19:38:00 UTC ℹ️ Start processing command /merge

2026-07-28 19:38:05 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-07-28 21:12:45 UTC ℹ️ MergeQueue: This merge request was merged

Agent.shutdown() stopped Feature Flagging twice: once through a no-arg shutdownFeatureFlagging() helper added on this branch, and once through shutdownFeatureFlagging(AGENT_CLASSLOADER), which reached master independently in #11892. Both resolve to FeatureFlaggingSystem.stop(), since shutdown() was only an alias for it, so the second call did nothing but repeat a reflective class load and re-log the stopped message. Drop this branch's helper, its call site, and the shutdown() alias, and point the tests at stop(). Agent.java now matches master, and the surviving path is the one covered by AgentFeatureFlaggingLifecycleTest. The enqueue gate that motivated the original helper is unaffected: it lives in stop(), which the remaining call site still reaches. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…-java Master received the agentless configuration source via the squashed #11892 while this branch already carried the same work through its own merge, so the overlapping changes arrived by two routes. Conflict resolution, all resolving to the branch side after confirming master contributed nothing unique in each file: - AgentThreadFactory / FeatureFlaggingConfig: both sides appended adjacent constants; kept both sets. - FeatureFlaggingSystem and its test: master's copies are the branch's minus the flag-evaluation writer, so the branch side is a superset. The offline configuration-source test exists on both sides in different positions; kept one copy. - FeatureFlaggingGatewayTest: git merged this cleanly but both sides had independently added an identical testProviderActivationListener, giving a duplicate method that failed compilation. Removed the copy. Verified: test sources compile for every module both sides touched, and the feature-flagging, communication, agent-bootstrap, internal-api and dd-trace-api suites pass. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Agent.shutdown() ran stopTelemetry() before shutdownFeatureFlagging(), so the drop and degradation counters the flag-evaluation writer queues during its final flush had no telemetry worker left to drain them: TelemetryRunnable emits queued metrics from flushPendingTelemetryData(), which only runs while the telemetry thread is still alive. This branch originally called shutdownFeatureFlagging() from above stopTelemetry(). Deduplicating the two shutdown paths in 9e63356 kept the call site that reached master through #11892, which sits after stopTelemetry(), and moved the ordering as a side effect. Restore it. Costs up to the writer's 5s join before telemetry stops; total agent shutdown stays bounded. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This comment has been minimized.

Sign in to view

leoromanovsky commented Jul 8, 2026

View reviewed changes

leoromanovsky commented Jul 9, 2026

View reviewed changes

Comment thread internal-api/src/main/java/datadog/trace/api/Config.java Outdated

leoromanovsky commented Jul 9, 2026

View reviewed changes

Comment thread ...ging/feature-flagging-agent/src/main/java/com/datadog/featureflag/FeatureFlaggingSystem.java Outdated

leoromanovsky mentioned this pull request Jul 11, 2026

test(ffe): pin end-to-end scenario to remote config DataDog/system-tests#7297

Merged

leoromanovsky force-pushed the leo.romanovsky/ffl-2693-java-agentless-configuration-source branch from 1858f32 to 82bb199 Compare July 11, 2026 04:01

leoromanovsky mentioned this pull request Jul 11, 2026

feat(openfeature): emit server-side EVP flagevaluation #11639

Open

leoromanovsky added 4 commits July 11, 2026 21:32

Define Feature Flagging configuration source contract

b8bbd26

Add Datadog-managed agentless UFC polling

4baefdb

Support custom agentless UFC endpoints

d378350

Select the Feature Flagging configuration source

f9e70e6

leoromanovsky force-pushed the leo.romanovsky/ffl-2693-java-agentless-configuration-source branch from be3f8fc to f9e70e6 Compare July 12, 2026 01:37

leoromanovsky commented Jul 12, 2026

View reviewed changes

leoromanovsky marked this pull request as ready for review July 12, 2026 12:10

leoromanovsky requested review from a team as code owners July 12, 2026 12:10

leoromanovsky requested review from AlexeyKuznetsov-DD, manuel-alvarez-alvarez, pavlokhrebto and typotter and removed request for a team July 12, 2026 12:10

leoromanovsky requested review from sarahchen6 and removed request for manuel-alvarez-alvarez July 12, 2026 12:11

leoromanovsky added comp: openfeature OpenFeature type: feature Enhancements and improvements labels Jul 12, 2026

datadog-datadog-prod-us1-2 Bot reviewed Jul 12, 2026

View reviewed changes

Comment thread ...feature-flagging-lib/src/main/java/com/datadog/featureflag/AgentlessConfigurationSource.java

sarahchen6 reviewed Jul 20, 2026

View reviewed changes

Comment thread ...feature-flagging-lib/src/main/java/com/datadog/featureflag/AgentlessConfigurationSource.java Outdated

vjfridge and others added 7 commits July 21, 2026 22:29

Merge branch 'master' into leo.romanovsky/ffl-2693-java-agentless-con…

c0575ac

…figuration-source

Merge branch 'master' into leo.romanovsky/ffl-2693-java-agentless-con…

3ae9cf0

…figuration-source

fix(feature-flags): align configuration source semantics

eaa99ce

feat(feature-flags): delay agentless polling until provider use

2c0ee6c

feat(communication): support mapped retried HTTP calls

bd0e347

fix(feature-flags): align agentless HTTP behavior

cfaca97

fix(feature-flags): complete initial poll during activation

6df48d1

sarahchen6 reviewed Jul 24, 2026

View reviewed changes

Comment thread dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Agent.java Outdated

leoromanovsky and others added 3 commits July 24, 2026 15:05

chore(feature-flags): remove unused agent feature

0e14ef5

Merge branch 'master' into leo.romanovsky/ffl-2693-java-agentless-con…

5ce7fa3

…figuration-source

dd-octo-sts Bot added the tag: ai generated Largely based on code generated by an AI or LLM label Jul 24, 2026

leoromanovsky added 5 commits July 24, 2026 16:04

test(feature-flags): cover agentless lifecycle branches

ad7d626

chore(agent): raise jar size budget to 34 MiB

29c1fd4

fix(feature-flags): document retry policy confinement

1c4a223

Revert "chore(agent): raise jar size budget to 34 MiB"

40c9e0f

This reverts commit 29c1fd4.

Merge remote-tracking branch 'origin/master' into agent/pr11892-cross…

f8ee701

…-sdk-contract

mcculls requested a review from mhlidd July 28, 2026 08:46

leoromanovsky added this pull request to the merge queue Jul 28, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 28, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit 29efc06 into master Jul 28, 2026
587 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the leo.romanovsky/ffl-2693-java-agentless-configuration-source branch July 28, 2026 21:12

github-actions Bot added this to the 1.65.0 milestone Jul 28, 2026

Uh oh!

Conversation

leoromanovsky commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Decisions

Initialization Modes

Customer Usage Example

Verification

System Test Evidence

Next Steps

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dd-octo-sts Bot commented Jul 12, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

leoromanovsky commented Jul 24, 2026

Review update

Changes

System-test evidence

Node.js alignment

Uh oh!

Uh oh!

dd-octo-sts Bot commented Jul 28, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

leoromanovsky commented Jul 8, 2026 •

edited

Loading

dd-octo-sts Bot commented Jul 8, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Jul 28, 2026 •

edited

Loading