feat(functions-aggregate): make approx_distinct HLL precision configurable#23816
Open
sandeshkr419 wants to merge 2 commits into
Open
feat(functions-aggregate): make approx_distinct HLL precision configurable#23816sandeshkr419 wants to merge 2 commits into
sandeshkr419 wants to merge 2 commits into
Conversation
…rable HyperLogLog precision was hardcoded at p=14 (16 384 registers, 16 KiB per sketch). Lower precision reduces memory and inter-shard state size at the cost of higher estimation error (p=12 → 4 KiB, ~1.6% error; p=10 → 1 KiB, ~3.3%). Changes: - `HyperLogLog::with_precision(p)` constructs a sketch at any p in 4..=18; `new()` keeps p=14 as the default (no behaviour change). - `HyperLogLog::from_registers(vec)` replaces `new_with_registers([u8; 16384])` and infers p from the slice length (must be a power of two). - `register_for_hash` and `count_from_hashes` take a `p` argument. - `ApproxDistinct::with_precision(p)` / `HllGroupsAccumulator::with_precision(p)` / `HLLAccumulator::with_precision(p)` / `NumericHLLAccumulator::with_precision(p)` propagate the chosen precision through every accumulator path. - Wire format is unchanged: serialized state is still raw register bytes; the length encodes p implicitly (len == 1 << p), so merge_serialized / TryFrom accept any valid precision automatically. - All 167 existing unit tests pass; 9 new tests cover precision construction, accuracy at p=10/12, roundtrip serialization, cross-precision merge panic, and boundary validation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Resolves #23815
Rationale for this change
Adds runtime-configurable HLL precision to approx_distinct. Previously the register count was a compile-time constant (p=14, 16 KiB per sketch); now any precision between 4 to 18 is supported.
What changes are included in this PR?
Are these changes tested?
9 new tests cover: construction at non-default precisions, accuracy within expected error bounds at p=10 and p=12, roundtrip serialization at p=10/12/14, cross-precision merge panic (programming error guard), and boundary validation.
Are there any user-facing changes?
None for existing callers. Old ApproxDistinct::new() and HyperLogLog::new() continue to use p=14.