Skip to content

Commit efdf030

Browse files
dfa1claude
andcommitted
perf(fsst): count training candidates by code in flat arrays, not HashMaps (#395)
Counts still tallied candidates with two HashMaps even after #393's allocation fixes, paying a hash computation, bucket lookup, and equals() per bump. The Rust reference's Counter (spiraldb/fsst) counts by code in flat arrays with zero hashing and zero boxing. Ported the same shape: codes 0..254 are real trained-symbol codes (resolved through the current Compressor's own symbol table), codes 255..510 are escape pseudo-codes for literal bytes — a fixed, dense space regardless of how many symbols are actually trained in any one generation, so counts1/counts2 need no per-generation resizing. A BitSet per array tracks which slots this generation touched; only the touched bits are cleared between generations, not the (much larger) count arrays, mirroring Counter's own bitmap-gated design. CompressorBuilder now threads one Counts instance across all five generations of a train() call instead of allocating fresh ones. Matcher.longestMatch already returns a code (Matcher.codeOf); this also fixes the redundancy #393's finding #4 flagged, where that code was computed and then thrown away in favor of re-deriving an identity from packed bytes plus length. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P4ijFsGW1MHEcGiu26vNzi
1 parent d0e46c3 commit efdf030

2 files changed

Lines changed: 147 additions & 75 deletions

File tree

fsst/src/main/java/io/github/dfa1/vortex/fsst/CompressorBuilder.java

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,12 +61,15 @@ public Compressor train(byte[][] rows) {
6161
if (sample.chunkCount() == 0) {
6262
return compressor;
6363
}
64+
// Reused across every generation instead of allocating fresh count arrays five times over
65+
// (issue #395).
66+
TrainingGeneration.Counts counts = new TrainingGeneration.Counts();
6467
for (int gen = 0; gen < GENERATIONS; gen++) {
6568
boolean finalGeneration = gen == GENERATIONS - 1;
6669
int chunkLimit = sample.chunkCountForGeneration(gen);
6770
int fractionNumerator = Sample.SAMPLE_FRACTION_NUMERATORS[gen];
6871
List<Symbol> symbols = TrainingGeneration.run(
69-
compressor, sample, chunkLimit, fractionNumerator, finalGeneration);
72+
compressor, sample, chunkLimit, fractionNumerator, finalGeneration, counts);
7073
// Reuse compressor's own (now fully consumed) matcher arrays instead of allocating a
7174
// fresh ~288 KB pair of tables on every one of the five generations (issue #393 #7).
7275
compressor = Compressor.rebuild(symbols, compressor);

0 commit comments

Comments
 (0)