Skip to content

[Pipe] Add SHOW PIPE MEMORY for named memory blocks - #18575

Open
Caideyipi wants to merge 6 commits into
apache:masterfrom
Caideyipi:show-pipe-memory
Open

[Pipe] Add SHOW PIPE MEMORY for named memory blocks#18575
Caideyipi wants to merge 6 commits into
apache:masterfrom
Caideyipi:show-pipe-memory

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Description

Named Pipe memory blocks

  • Add an immutable name to every PipeMemoryBlock managed by PipeMemoryManager.
  • Propagate descriptive names through all Pipe memory allocation APIs and call sites.
  • Keep live zero-sized blocks visible until release, while using weak references so diagnostics do not retain otherwise unreachable zero-sized blocks.
  • Preserve one output row per block instance; blocks with the same name are not aggregated.

SHOW PIPE MEMORY and information_schema

  • Add SHOW PIPE MEMORY in the table SQL dialect.
  • Add information_schema.pipe_memory with:
    • name STRING TAG
    • memory_usage_in_bytes INT64 ATTRIBUTE
  • Expose floating memory as an independent FloatingMemory block.
  • Read the table from the local DataNode and require global system privilege.

Tests

  • Add parser and PipeMemoryManager unit coverage.
  • Update IoTDBDatabaseIT for table discovery, schema, permissions, SHOW/SELECT access, and system-table counts.
  • Verified:
    • mvn -o install -pl iotdb-core/node-commons,iotdb-core/relational-grammar -DskipTests
    • mvn -o test -pl iotdb-core/datanode -Dtest=ShowPipeMemoryTest,PipeMemoryManagerResizeTest,LoadTsFileMemoryManagerTest -Dcheckstyle.skip=true -Dspotless.check.skip=true
    • mvn checkstyle:check -pl iotdb-core/datanode
    • git diff --check

The integration test module could not be executed locally in offline mode because several required snapshot artifacts were unavailable.

This PR has:

  • been self-reviewed.
    • concurrent read and write
  • added comments explaining non-obvious lifecycle and accounting behavior.
  • added unit tests or modified existing tests to cover new code paths.
  • added integration tests.
Key changed/added classes
  • PipeMemoryBlock
  • PipeMemoryManager
  • InformationSchema
  • InformationSchemaContentSupplierFactory
  • RelationalSql.g4
  • AstBuilder
  • IoTDBDatabaseIT

jt2594838
jt2594838 reviewed Sep 8, 2026
View reviewed changes
Comment thread ...c/main/java/org/apache/iotdb/db/pipe/event/common/statement/PipeStatementInsertionEvent.java Outdated
Comment on lines +88 to +90
this.allocatedMemoryBlock =
PipeDataNodeResourceManager.memory().forceAllocateForTabletWithRetry(0);
PipeDataNodeResourceManager.memory()
.forceAllocateForTabletWithRetry(PipeStatementInsertionEvent.class.getSimpleName(), 0);

@jt2594838 jt2594838 Sep 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add some identifiers from the event?

@jt2594838 jt2594838 Sep 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same below.

@Caideyipi Caideyipi Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in 5c7cf74, with follow-up fixes in c0163e0. Event-owned blocks now record an assigner snapshot from the event. For EnrichedEvent, it uses coreReportMessage() and falls back to toString() when needed; the snapshot is bounded and does not retain the event object. The same handling is applied to the other event allocation sites.

@Caideyipi Caideyipi Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in 5c7cf74, with follow-up fixes in c0163e0. This allocation site uses the same bounded event assigner snapshot, so the block retains the event identifier without retaining the event object.

Comment thread ...g/apache/iotdb/db/pipe/event/common/tsfile/parser/query/TsFileInsertionEventQueryParser.java
Comment thread ...org/apache/iotdb/db/pipe/event/common/tsfile/parser/scan/TsFileInsertionEventScanParser.java Outdated
@Caideyipi
[Pipe] Add hierarchical memory diagnostics to SHOW PIPE MEMORY
5c7cf74
@Caideyipi
[Pipe] Fix memory diagnostic allocation behavior
c0163e0
@Caideyipi
Merge branch 'master' of https://github.com/apache/iotdb into show-pi…
0062fee 
…pe-memory

# Conflicts:
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/PipeTsFileInsertionEvent.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/memory/PipeMemoryManager.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFilePublicResource.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/sink/payload/evolvable/batch/PipeTabletEventBatch.java
jt2594838
jt2594838 approved these changes Sep 14, 2026
View reviewed changes
Comment thread iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/memory/PipeMemoryBlock.java Outdated
Comment on lines +59 to +60
private final Set<PipeMemoryBlock> children =
Collections.newSetFromMap(new java.util.WeakHashMap<>());

@jt2594838 jt2594838 Sep 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mind the import

@Caideyipi Caideyipi Sep 17, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in 656473e. Added the java.util.WeakHashMap import and removed the fully qualified reference.

Comment thread ...datanode/src/main/java/org/apache/iotdb/db/pipe/resource/memory/PipeMemoryBlockCategory.java Outdated
if (type == null) {
return OTHER;
}
switch (type) {

@jt2594838 jt2594838 Sep 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use an enhanced switch without a default branch.

@Caideyipi Caideyipi Sep 17, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in 656473e. Converted this to an exhaustive switch expression without a default branch, with NORMAL explicitly mapped to OTHER.

@Caideyipi
Merge branch 'master' of https://github.com/apache/iotdb into show-pi…
7f156a5 
…pe-memory

# Conflicts:
#	integration-test/src/test/java/org/apache/iotdb/relational/it/schema/IoTDBDatabaseIT.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/receiver/protocol/thrift/IoTDBDataNodeReceiver.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/sink/payload/evolvable/batch/PipeTabletEventBatch.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/InformationSchemaContentSupplierFactory.java
#	iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/table/InformationSchema.java
#	iotdb-core/relational-grammar/src/main/antlr4/org/apache/iotdb/db/relational/grammar/sql/RelationalSql.g4
@Caideyipi
[Pipe] Address SHOW PIPE MEMORY review feedback
656473e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jt2594838 jt2594838 jt2594838 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Caideyipi @jt2594838