Skip to content

[Feature] Add Pipe logical backup sink and import tooling - #18623

Open
Caideyipi wants to merge 3 commits into
masterfrom
feat/pipe-logical-backup-sink
Open

Caideyipi wants to merge 3 commits into
masterfrom
feat/pipe-logical-backup-sink

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add built-in logical-backup-sink / logical-backup-connector plugins that persist Pipe metadata and data events as ordered WAL-like .pwal segments
  • add CRC32C/SHA-256 integrity checks, atomic manifests, segment recovery/rotation, directory locking, append identity validation, and committed event groups
  • require DataRegion logical backup tasks to explicitly configure sink.resume=append (or connector.resume=append), with validation directing operators to use a shared backup directory so leader changes resume the same archive; SchemaRegion keeps its existing default
  • support tablet, TsFile, delete, schema plan/snapshot, heartbeat, and terminate records for tree and table models
  • provide inspect, verify, export, and import commands with safe ZIP handling, dry-run, schema-first replay, and replay request allowlisting
  • checkpoint imports after every request and accept the receiver idempotent-conflict status, so recovery can safely replay at most the request whose checkpoint write was interrupted
  • package Linux and Windows import/export wrappers in the CLI distribution and provide English/Chinese messages

Validation

  • node-commons logical-backup tests: 32 passed (LogicalBackupWriterTest, LogicalBackupArchiveReaderTest)
  • CLI tests: 7 passed (PipeLogicalBackupToolTest)
  • DataNode validation tests: 2 passed (LogicalBackupSinkTest)
  • English compilation/tests and Chinese-locale test-compile passed for the changed DataNode and CLI modules
  • Spotless, Checkstyle, and forbidden-apis checks passed for the changed modules

@codecov

codecov Bot commented Sep 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 69.62639% with 626 lines in your changes missing coverage. Please review.
✅ Project coverage is 42.96%. Comparing base (31621ef) to head (218dcb4).
⚠️ Report is 11 commits behind head on master.

Files with missing lines Patch % Lines
...sink/protocol/logicalbackup/LogicalBackupSink.java 14.63% 245 Missing ⚠️
.../apache/iotdb/tool/pipe/PipeLogicalBackupTool.java 64.45% 177 Missing ⚠️
...s/pipe/sink/logicalbackup/LogicalBackupWriter.java 87.93% 86 Missing ⚠️
...sink/logicalbackup/LogicalBackupSegmentReader.java 76.27% 56 Missing ⚠️
...sink/logicalbackup/LogicalBackupArchiveReader.java 84.18% 37 Missing ⚠️
...s/pipe/sink/logicalbackup/LogicalBackupRecord.java 57.14% 18 Missing ⚠️
...ugin/dataregion/PipeDataRegionSinkConstructor.java 66.66% 2 Missing ⚠️
.../schemaregion/PipeSchemaRegionSinkConstructor.java 66.66% 2 Missing ⚠️
...s/pipe/sink/logicalbackup/LogicalBackupFormat.java 71.42% 2 Missing ⚠️
.../builtin/sink/logicalbackup/LogicalBackupSink.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #18623      +/-   ##
============================================
+ Coverage     42.77%   42.96%   +0.19%     
  Complexity      442      442              
============================================
  Files          5443     5461      +18     
  Lines        394537   397485    +2948     
  Branches      51723    52142     +419     
============================================
+ Hits         168747   170770    +2023     
- Misses       225790   226715     +925     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Caideyipi Caideyipi changed the title [Feature] Add Pipe logical backup sink and restore tooling [Feature] Add Pipe logical backup sink and import tooling Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant