Skip to content

Fix race in event_tracer/repeat unit tests#139

Merged
alan-george-lk merged 2 commits into
mainfrom
bugfix/tracing_test
May 26, 2026
Merged

Fix race in event_tracer/repeat unit tests#139
alan-george-lk merged 2 commits into
mainfrom
bugfix/tracing_test

Conversation

@alan-george-lk
Copy link
Copy Markdown
Collaborator

@alan-george-lk alan-george-lk commented May 26, 2026

Context: occasionally the CI runs would hang indefinitely on EventTracer.* based tests. This smelled like a race condition.

Root cause:

Race identified in src/trace/event_tracer.cpp. The writer thread waits on a CV under g_mutex:

std::unique_lock<std::mutex> lock(g_mutex);
g_cv.wait(lock, [] { return !g_event_queue.empty() || g_shutdown_requested.load(); });

But StopTracing() was modifying g_shutdown_requested without holding the g_mutex.

g_shutdown_requested.store(true);   // outside the lock
g_cv.notify_one();
g_writer_thread.join();             // <-- blocks forever

Even though that individual variable is atomic, per the CPP docs on condition variables:

Even if the shared variable is atomic, it must be modified while owning the mutex to correctly publish the modification to the waiting thread.

Solution was to simply do just that in StopTracing().

Testing

Verified locally and in CI with 1000 repeats of all unit tests (single run prior).

Also updated this stage to run all unit tests 100 times, but keep GTest output minimal to avoid spam.

@alan-george-lk alan-george-lk changed the title Fix race in event_tracer Fix race in event_tracer/repeat unit tests May 26, 2026
@alan-george-lk alan-george-lk marked this pull request as ready for review May 26, 2026 18:17
g_shutdown_requested.store(true);
// Signal writer thread to shut down. The shutdown flag must be set while
// holding g_mutex so the writer thread cannot miss the notification
{
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice.

Copy link
Copy Markdown
Collaborator

@stephen-derosa stephen-derosa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM -- thank you!

Comment thread .github/workflows/tests.yml
@alan-george-lk alan-george-lk merged commit c194e76 into main May 26, 2026
28 of 29 checks passed
@alan-george-lk alan-george-lk deleted the bugfix/tracing_test branch May 26, 2026 21:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants