Skip to content

[To dev/1.3] [Pipe] Restore sink progress under memory pressure (#18640) - #18649

Merged
jt2594838 merged 1 commit into
apache:dev/1.3from
Caideyipi:backport/e2ac222-dev-1.3
Sep 17, 2026
Merged

jt2594838 merged 1 commit into
apache:dev/1.3from
Caideyipi:backport/e2ac222-dev-1.3

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Backport #18640 to dev/1.3 by cherry-picking e2ac222.

Restore sink-batch forward progress

#18582 changed the memory reservation owned by PipeTabletEventBatch from a normal PipeMemoryBlock to a PipeTabletMemoryBlock. This made sink-batch growth subject to the tablet data-structure admission thresholds. Once retained tablet memory crossed that sub-quota, a sink batch could no longer grow enough to transfer and acknowledge events, so the retained/floating memory could not be released either.

This change restores the batch reservation to a normal pipe memory block. Tablet and TsFile allocations remain bounded by their data-structure thresholds, while sink batches can use the remaining non-floating memory to make downstream progress and release retained events.

Back off memory-timeout retries

PipeAbstractSinkSubtask previously retried MEMORY_TIMEOUT failures immediately while applying backoff to the other transient resource failures. Memory timeouts now use the existing non-report exception backoff after logging the transient condition.

The cherry-pick had one conflict in PipeAbstractSinkSubtask; the resolution preserves the dev/1.3 logging code and applies the upstream backoff behavior.

Regression coverage

  • mvn spotless:apply -pl iotdb-core/node-commons,iotdb-core/datanode: passed.
  • mvn test -pl iotdb-core/node-commons -Dtest=PipeSleepIntervalTest: passed (4 tests).
  • git diff origin/dev/1.3..HEAD --check: passed.
  • PipeTransferBatchReqBuilderTest was attempted, but the standalone DataNode build stopped before tests because the local 1.3.7-SNAPSHOT dependency artifacts are older than the current dev/1.3 sources (missing unrelated generated Thrift and cross-module APIs). The changed DataNode source itself produced no compiler error; CI will build it with reactor-consistent dependencies.

This PR has:

  • been self-reviewed.
  • added unit tests for the changed code paths.

Key changed/added classes
  • PipeTabletEventBatch
  • PipeAbstractSinkSubtask
  • PipeTransferBatchReqBuilderTest
  • PipeSleepIntervalTest

@jt2594838
jt2594838 merged commit 0c83c1b into apache:dev/1.3 Sep 17, 2026
16 of 17 checks passed
@jt2594838
jt2594838 deleted the backport/e2ac222-dev-1.3 branch September 17, 2026 03:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants