Skip to content

[Pipe] Fix batch and async resource lifecycle leaks - #18582

Merged
jt2594838 merged 2 commits into
apache:masterfrom
Caideyipi:fix-pipe-batch-async-lifecycle
Sep 10, 2026
Merged

jt2594838 merged 2 commits into
apache:masterfrom
Caideyipi:fix-pipe-batch-async-lifecycle

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Make Pipe batch memory resizing non-blocking and roll back partially constructed batches when allocation fails.
  • Detach asynchronous batches atomically and release batch-owned memory after event ownership moves to handlers or retry queues.
  • Make async callbacks and retry bookkeeping idempotent to prevent duplicate queue entries and event reference leaks.
  • Clean up TsFile parser reservations, readers, buffers, public resources, and temporary batch files on close and failure paths.
  • Add regression coverage for memory pressure, parser close races, retry lifecycle, batch detachment, and TsFile cleanup.

Tests

  • mvn compile -pl iotdb-core/datanode -DskipTests
  • mvn test -pl iotdb-core/datanode -Dtest=PipeMemoryManagerResizeTest,TsFileInsertionEventParserTest,PipeSinkTest,PipeTabletEventTsFileBatchTest,PipeTransferBatchReqBuilderTest,PipeTransferTrackableHandlerTest,PipeTransferTsFileHandlerCleanupTest,IoTDBDataRegionAsyncSinkTest -DfailIfNoTests=false
  • mvn test-compile -pl iotdb-core/datanode -P with-zh-locale -DskipTests

The selected test suite ran 70 tests with 0 failures and 3 skipped.

@jt2594838
jt2594838 merged commit 6c2b747 into apache:master Sep 10, 2026
42 of 43 checks passed
@jt2594838
jt2594838 deleted the fix-pipe-batch-async-lifecycle branch September 10, 2026 06:15
jt2594838 pushed a commit that referenced this pull request Sep 11, 2026
* [Pipe] Fix batch and async resource lifecycle leaks (#18582)

* [Pipe] Fix batch and async resource lifecycle leaks

* Fix review feedback for pipe async lifecycle

(cherry picked from commit 6c2b747)

* [Pipe] Fix dev/1.3 backport test compatibility

* [Pipe] Fix air gap tsfile batch test data
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants