Skip to content

Fix oversized Load TsFile piece dispatch - #18627

Merged
jt2594838 merged 2 commits into
apache:masterfrom
Caideyipi:fix/load-tsfile-thrift-frame-slicing
Sep 18, 2026
Merged

jt2594838 merged 2 commits into
apache:masterfrom
Caideyipi:fix/load-tsfile-thrift-frame-slicing

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

Load TsFile dispatch could send a serialized LoadTsFilePieceNode larger than the receiver's dn_thrift_max_frame_size, causing the Thrift transport to reject the frame.

  • Discover each receiver's frame limit and split piece payloads using the smaller sender/receiver limit, reserving 1 KiB for the RPC envelope. Cache the remote limit for the dispatcher.
  • Reassemble slices in order with a buffer preallocated to the original payload size. Return specific errors for missing metadata and invalid slices.
  • Keep small requests on the existing wire format, including when an older receiver does not implement frame-limit discovery.
  • Clean up partial assemblies on timeout/rollback and reject EXECUTE while slices are incomplete.

Verification

  • LoadTsFileDispatcherImplTest: 7 tests passed, including differently sized sender/receiver transports, multiple replicas, limit caching, legacy fallback, and discovery failure.
  • LoadTsFilePieceNodeAssemblerTest: 7 tests passed, including invalid slice diagnostics and direct/read-only buffer reassembly.
  • DataNodeInternalRPCServiceImplLoadTest: 1 test passed, covering all six partial metadata combinations.
  • Full-reactor mvn test-compile -DskipTests passed for both EN and ZH locales.
  • DataNode Checkstyle/Spotless and EN/ZH message parity checks passed.
  • A local JDK 17 microbenchmark favored 8 KiB bulk copying over byte-by-byte writing for direct and read-only inputs; the bulk path is retained.

Sliced transfers require upgraded sender and receiver DataNodes; older receivers do not understand slice metadata.

@jt2594838
jt2594838 merged commit 7310b2b into apache:master Sep 18, 2026
39 checks passed
@Caideyipi
Caideyipi deleted the fix/load-tsfile-thrift-frame-slicing branch September 18, 2026 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants