Skip to content

[Pipe] Restore sink progress under memory pressure - #18640

Open
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/pipe-sink-memory-self-lock
Open

Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/pipe-sink-memory-self-lock

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Restore sink-batch forward progress

#18582 changed the memory reservation owned by PipeTabletEventBatch from a normal PipeMemoryBlock to a PipeTabletMemoryBlock. This made sink-batch growth subject to the tablet data-structure admission thresholds. Once retained tablet memory crossed that sub-quota, a sink batch could no longer grow enough to transfer and acknowledge events, so the retained/floating memory could not be released either.

This PR restores the batch reservation to a normal pipe memory block. Tablet and TsFile allocations remain bounded by their data-structure thresholds, while sink batches can use the remaining non-floating memory to make downstream progress and release retained events.

Back off memory-timeout retries

PipeAbstractSinkSubtask previously retried MEMORY_TIMEOUT failures immediately while applying backoff to the other transient resource failures. The immediate retry loop could consume several CPU cores and flood logs while memory remained unavailable. Memory timeouts now use the existing non-report exception backoff after logging the transient condition.

Regression coverage

  • Verify that tablet batch payload memory increases total pipe memory without increasing tablet-memory accounting.
  • Verify that a sink subtask waits before retrying a memory-timeout failure.
  • PipeSleepIntervalTest: 4 tests passed.
  • PipeMemoryManagerResizeTest and PipeTransferBatchReqBuilderTest: 8 tests passed after isolated compilation against the module classpath.
  • Spotless, Checkstyle, and git diff --check passed for the changed files.

A full DataNode reactor compile was also attempted, but this Windows host exhausted native JVM memory while compiling the 2,983 DataNode sources. The focused DataNode classes and tests compile and pass independently.


This PR has:

  • been self-reviewed.
  • added comments explaining the why and intent where the behavior is not obvious.
  • added unit tests for the new code paths.

Key changed/added classes (or packages if there are too many classes) in this PR
  • PipeTabletEventBatch
  • PipeAbstractSinkSubtask
  • PipeTransferBatchReqBuilderTest
  • PipeSleepIntervalTest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant