Skip to content

fix(spanner): prevent memory leak and thread blocking in transaction keep-alive - #13897

Open
olavloite wants to merge 1 commit into
mainfrom
spanner/connection-tx-keep-alive-fix
Open

fix(spanner): prevent memory leak and thread blocking in transaction keep-alive#13897
olavloite wants to merge 1 commit into
mainfrom
spanner/connection-tx-keep-alive-fix

Conversation

@olavloite

Copy link
Copy Markdown
Contributor
  • Enable setRemoveOnCancelPolicy(true) on KEEP_ALIVE_SERVICE so canceled tasks are immediately purged from DelayedWorkQueue.
  • Use a WeakReference<ReadWriteTransaction> in KeepAliveRunnable to prevent scheduled tasks from retaining strong references to transaction instances.
  • Use abortedLock.tryLock() in KeepAliveRunnable so the shared executor thread does not block when a transaction is active or retrying.
  • Remove duplicate maybeScheduleKeepAlivePing listener registration on keep-alive query completion.
  • Add unit tests in ReadWriteTransactionTest verifying task removal on cancel, weak reference retention, non-blocking lock handling, and single ping scheduling on completion.

@olavloite
olavloite requested review from a team as code owners July 27, 2026 09:10

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the transaction keep-alive mechanism in ReadWriteTransaction by using a ScheduledThreadPoolExecutor with removeOnCancelPolicy enabled, and converting KeepAliveRunnable into a static nested class with a WeakReference to prevent memory leaks. It also introduces lock-based synchronization using abortedLock to avoid sending keep-alive pings when the transaction is busy. Feedback on this PR highlights a critical issue where the keep-alive loop can silently terminate when the transaction is busy; because keepAliveFuture.isDone() is still false during execution, the synchronous call to reschedule the ping results in a no-op. A fix is suggested to reset keepAliveFuture to null before rescheduling.

@olavloite
olavloite force-pushed the spanner/connection-tx-keep-alive-fix branch from d5e0904 to 8c8c44d Compare July 28, 2026 07:52
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the keep-alive mechanism in ReadWriteTransaction to prevent memory leaks and improve task cancellation. It changes KeepAliveRunnable to a static class using a WeakReference to the transaction, replaces the scheduled executor with a ScheduledThreadPoolExecutor configured to remove cancelled tasks, and introduces a rescheduling mechanism when the transaction is busy. A critical issue was identified in the newly added test testKeepAliveRescheduledWhenLockBusy, where the use of a ReentrantLock on the same thread allows tryLock() to succeed due to reentrancy, thereby failing to test the actual rescheduling path. The lock should be acquired on a separate thread to correctly simulate a busy transaction.

@olavloite
olavloite force-pushed the spanner/connection-tx-keep-alive-fix branch 4 times, most recently from 8b5aaa6 to ef6529f Compare July 28, 2026 09:27
@olavloite
olavloite requested a review from sakthivelmanii July 28, 2026 09:37
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the keep-alive mechanism in ReadWriteTransaction to prevent memory leaks and improve task scheduling. Specifically, it changes KeepAliveRunnable to a static class holding a WeakReference to the transaction, replaces the scheduled executor with a ScheduledThreadPoolExecutor that removes cancelled tasks, and uses abortedLock.tryLock() to reschedule pings if the transaction is busy. Feedback suggests releasing abortedLock before calling maybeScheduleKeepAlivePing() in the exception handler of KeepAliveRunnable.run() to minimize lock scope and prevent potential deadlocks.

@olavloite
olavloite force-pushed the spanner/connection-tx-keep-alive-fix branch from ef6529f to 6d328ce Compare July 28, 2026 14:46
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the keep-alive mechanism in ReadWriteTransaction to use a ScheduledThreadPoolExecutor with a remove-on-cancel policy and updates KeepAliveRunnable to use a WeakReference to prevent memory leaks. It also introduces locking to reschedule pings when the transaction is busy and adds corresponding unit tests. The review feedback suggests improving the robustness of the unit tests by asserting the presence of the specific ScheduledFuture in the shared queue rather than asserting the exact queue size, which can cause flakiness during concurrent test execution.

@olavloite
olavloite force-pushed the spanner/connection-tx-keep-alive-fix branch from 6d328ce to e537ee3 Compare July 28, 2026 16:01
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the keep-alive mechanism in ReadWriteTransaction to prevent memory leaks and queue accumulation. It introduces a ScheduledThreadPoolExecutor with a task-removal-on-cancel policy, refactors KeepAliveRunnable to a static class using a WeakReference to the transaction, and implements non-blocking lock checks to reschedule pings when the transaction is busy. Comprehensive unit tests are also added. The reviewer suggests checking shouldPing() before scheduling a new keep-alive ping to avoid adding redundant tasks to the queue after a transaction has finished.

@olavloite
olavloite force-pushed the spanner/connection-tx-keep-alive-fix branch from e537ee3 to fff729d Compare July 28, 2026 16:31
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the keep-alive mechanism in ReadWriteTransaction to prevent memory leaks and improve concurrency handling. It replaces the single-thread scheduled executor with a ScheduledThreadPoolExecutor configured to remove cancelled tasks, wraps the transaction in a WeakReference inside KeepAliveRunnable, and uses abortedLock.tryLock() to avoid executing pings when the transaction is busy. The review feedback suggests adding @SuppressWarnings("FutureReturnValueIgnored") to prevent static analysis errors, and removing the redundant manual rescheduling logic (along with its associated test) when the lock is busy, as active transactions already schedule pings automatically.

@olavloite
olavloite force-pushed the spanner/connection-tx-keep-alive-fix branch from fff729d to 0afb1d9 Compare July 28, 2026 16:41
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the keep-alive mechanism in ReadWriteTransaction by replacing the ScheduledExecutorService with a ScheduledThreadPoolExecutor configured to remove cancelled tasks, and converting KeepAliveRunnable to a static class with a WeakReference to prevent memory leaks. It also introduces concurrency controls using abortedLock.tryLock() and adds comprehensive unit tests. The review feedback highlights a critical race condition where failing to acquire abortedLock can permanently terminate the keep-alive loop, and suggests rescheduling the ping when the lock is busy, along with updating the corresponding unit test to assert that a new future is scheduled.

@olavloite
olavloite force-pushed the spanner/connection-tx-keep-alive-fix branch from 0afb1d9 to f54989b Compare July 28, 2026 16:58
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the keep-alive mechanism in ReadWriteTransaction to prevent memory leaks and improve resource management. Specifically, it replaces the scheduled executor service with a ScheduledThreadPoolExecutor configured to remove tasks on cancellation, and converts KeepAliveRunnable into a static class that holds a WeakReference to the transaction. Additionally, it introduces robust locking and status checks during keep-alive scheduling and execution, accompanied by comprehensive unit tests covering these scenarios. There are no review comments, so I have no feedback to provide.

…keep-alive

- Enable `setRemoveOnCancelPolicy(true)` on `KEEP_ALIVE_SERVICE` so canceled tasks are immediately purged from `DelayedWorkQueue`.
- Use a `WeakReference<ReadWriteTransaction>` in `KeepAliveRunnable` to prevent scheduled tasks from retaining strong references to transaction instances.
- Use `abortedLock.tryLock()` in `KeepAliveRunnable` so the shared executor thread does not block when a transaction is active or retrying.
- Remove duplicate `maybeScheduleKeepAlivePing` listener registration on keep-alive query completion.
- Add unit tests in `ReadWriteTransactionTest` verifying task removal on cancel, weak reference retention, non-blocking lock handling, and single ping scheduling on completion.
@olavloite
olavloite force-pushed the spanner/connection-tx-keep-alive-fix branch from f54989b to a1f5720 Compare July 28, 2026 17:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant