Avoid retaining invalid VM method pointers

[jbachorik]

What does this PR do?:

Avoid retaining a raw VMMethod* after HotSpot stack walking has already returned JMETHODID_NOT_WALKABLE.

Motivation:

JMETHODID_NOT_WALKABLE means the walker could not establish that the VMMethod* is safe to use as a jmethodID source. Previously, the five-argument fillFrame converted that rejection into a raw-pointer
frame, allowing the pointer to survive in CallTraceStorage until a later dump.

Additional Notes:

At dump time, HotspotSupport::resolve could dereference the retained VMMethod* and synthesize a jmethodID from metadata HotSpot had already torn down or reused. Lookup::fillJavaMethodInfo could then issue
JVMTI calls on that stale or garbage identity. This is a strong match for the observed jvmti_GetClassSignature crash on releases where the known upstream two-call class-unload race is fixed.

The fix keeps JMETHODID_NOT_WALKABLE in the captured frame and normalizes it to the shared unknown method before any JVMTI call. The intentional nullptr raw-pointer fallback is unchanged.

How to test the change?:

• ./.claude/commands/build-and-summarize spotlessApply
• Linux fastdebug JDK 11.0.32: :ddprof-lib:gtestDebug_hotspotMethodId_ut
• Linux fastdebug JDK 11.0.32: :ddprof-test:testDebug -Pskip-gtest -Ptests=JavaProfilerTest.rawMethodResolutionDuringRetransformationStress

The focused native test forces the rejected sentinel, verifies that it is not packed as a raw pointer, and verifies deferred lookup resolves it as the shared unknown method without reaching JVMTI.

For Datadog employees:

• If this PR touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a security review.
• This PR doesn't touch any of that.
• JIRA: PROF-15568

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[jbachorik]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e61602b3c6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[jbachorik]

Jira: PROF-15568

[dd-octo-sts]

CI Test Results

Run: #30559202741 | Commit: 8fc58ce | Duration: 15m 16s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK	glibc-aarch64/debug	glibc-amd64/debug	musl-aarch64/debug	musl-amd64/debug
8	-	✅	-	-
8-ibm	-	✅	-	-
8-j9	✅	✅	-	-
8-librca	-	-	✅	✅
8-orcl	-	✅	-	-
11	-	✅	-	-
11-j9	✅	✅	-	-
11-librca	-	-	✅	✅
17	✅	✅	-	-
17-graal	✅	✅	-	-
17-j9	✅	✅	-	-
17-librca	-	-	✅	✅
21	✅	✅	-	-
21-graal	✅	✅	-	-
21-librca	-	-	✅	✅
25	✅	✅	-	-
25-graal	✅	✅	-	-
25-librca	-	-	✅	✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0

---

Updated: 2026-07-30 16:16:23 UTC

[dd-octo-sts]

Benchmark Results (commit 323b1be)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128028156 Commit: 323b1bea3e0ccdb09593932bb95ceae5108baf00

⚠️ Significant outliers

• 🟢 fj-kmeans (JDK 21): runtime -4.6% (2760→2634 ms)
• 🔴 fj-kmeans (JDK 25): runtime +3.4% (2775→2868 ms)

Runtime details (per benchmark × JDK)

Benchmark	JDK	Latest	Dev	Δ (dev vs latest)	Issues L/D
akka-uct	21	✅ 10203 ms (21 iters)	✅ 10329 ms (21 iters)	≈ +1.2% (±11.3%)	— / —
akka-uct	25	✅ 8828 ms (24 iters)	✅ 8885 ms (24 iters)	≈ +0.6% (±10.1%)	— / —
finagle-chirper	21	✅ 6000 ms (33 iters)	✅ 5941 ms (33 iters)	≈ -1% (±24.5%)	⚠️ W:3 / ⚠️ W:3
finagle-chirper	25	✅ 5431 ms (36 iters)	✅ 5450 ms (36 iters)	≈ +0.3% (±24.1%)	⚠️ W:3 / ⚠️ W:3
fj-kmeans	21	✅ 2760 ms (68 iters)	✅ 2634 ms (72 iters)	🟢 -4.6%	— / —
fj-kmeans	25	✅ 2775 ms (68 iters)	✅ 2868 ms (66 iters)	🔴 +3.4%	— / —
future-genetic	21	✅ 2062 ms (90 iters)	✅ 2097 ms (88 iters)	≈ +1.7% (±2.7%)	— / —
future-genetic	25	✅ 2080 ms (89 iters)	✅ 2078 ms (89 iters)	≈ -0.1% (±2.7%)	— / —
naive-bayes	21	✅ 1298 ms (132 iters)	✅ 1267 ms (135 iters)	≈ -2.4% (±32.3%)	— / —
naive-bayes	25	✅ 1008 ms (170 iters)	✅ 1029 ms (166 iters)	≈ +2.1% (±32%)	— / —
reactors	21	✅ 16262 ms (15 iters)	✅ 16509 ms (15 iters)	≈ +1.5% (±9.1%)	— / —
reactors	25	✅ 18524 ms (15 iters)	✅ 18087 ms (15 iters)	≈ -2.4% (±6%)	— / —

Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark	JDK	Dropped rec	Dropped jvmti	Dropped trace	Skipped WC	AGCT fail	Unwind fail
akka-uct	21	✅ / ✅	✅ / ✅	3 / 3	1878 / 1894	✅ / ✅	✅ / ✅
akka-uct	25	✅ / ✅	✅ / ✅	4 / 5	2349 / 2241	✅ / ✅	✅ / ✅
finagle-chirper	21	✅ / ✅	✅ / ✅	2 / 4	8924 / 8610	✅ / ✅	✅ / ✅
finagle-chirper	25	✅ / ✅	✅ / ✅	1 / 2	8312 / 8222	✅ / ✅	✅ / ✅
fj-kmeans	21	✅ / ✅	✅ / ✅	1 / 1	1297 / 1293	✅ / ✅	✅ / ✅
fj-kmeans	25	✅ / ✅	✅ / ✅	1 / 1	1308 / 1298	✅ / ✅	✅ / ✅
future-genetic	21	✅ / ✅	✅ / ✅	3 / ✅	2944 / 2982	✅ / ✅	✅ / ✅
future-genetic	25	✅ / ✅	✅ / ✅	1 / 2	2902 / 2802	✅ / ✅	✅ / ✅
naive-bayes	21	✅ / ✅	✅ / ✅	4 / 4	3510 / 3516	✅ / ✅	✅ / ✅
naive-bayes	25	✅ / ✅	✅ / ✅	2 / 4	3492 / 3494	✅ / ✅	✅ / ✅
reactors	21	✅ / ✅	✅ / ✅	1 / 1	1516 / 1596	✅ / ✅	✅ / ✅
reactors	25	✅ / ✅	✅ / ✅	✅ / ✅	1884 / 1865	✅ / ✅	✅ / ✅

[kaahos]

It looks good to me, Thanks for the fix!