[core][flink] Preserve file stats during primary-key compaction - #9131
Open
liujiayi771 wants to merge 1 commit into
Open
[core][flink] Preserve file stats during primary-key compaction#9131liujiayi771 wants to merge 1 commit into
liujiayi771 wants to merge 1 commit into
Conversation
Preserve file value stats across primary-key writer restore and compactor scans so metadata-only level upgrades do not permanently lose skipping metadata. Drop stats only when constructing DELETE manifest entries. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5.6-sol Co-Authored-By: Claude Code <noreply@anthropic.com>
liujiayi771
marked this pull request as ready for review
August 9, 2026 09:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fixes #7026.
When
manifest.delete-file-drop-statsis enabled, scan-leveldropStats()strips stats from every scanned manifest entry, not just DELETE entries. During primary-key compaction, metadata-only level upgrades can reuse the same data file for DELETE and ADD entries. If compactor or restore scans drop the stats first, the ADD entry permanently loses file value stats and subsequent reads lose data-skipping metadata.This change keeps stats on scan results and applies
copyWithoutStats()only while constructing DELETE manifest entries. ADD entries therefore preserve file stats, while DELETE entries still omit them according to the configured option.Tests
ASDF_MAVEN_VERSION=3.9.9 mvn -pl paimon-core -Pfast-build -DwildcardSuites=none -Dtest=TableWriteTest#testDropStatsOnlyForDeleteManifestEntries testASDF_MAVEN_VERSION=3.9.9 mvn -pl paimon-flink/paimon-flink-common -Pfast-build -Pflink1 -DwildcardSuites=none -Dtest=TableWriteCoordinatorTest#testScanPreservesStatsWhenDeleteManifestStatsAreDropped,CompactorSourceITCase#testStreamingReadPreservesStatsWhenDeleteManifestStatsAreDropped testgit diff --check github/master...HEAD