Refactor TSDataType switches to use TypeService by jt2594838 · Pull Request #18606 · apache/iotdb

jt2594838 · 2026-09-09T02:06:48Z

What is changed

Refactor TSDataType switch handling into TypeService implementations.
Add an ArchUnit check that rejects TSDataType switch usage outside TypeService.
Update Tablet decoding and AlignedTVList WAL handling for TsFile 2.4.1-SNAPSHOT.
Merge the latest master branch changes.

Validation

DataNode and related regression tests passed.
ArchUnit TSDataType switch checks passed without a baseline.
REST Chinese-locale test compilation passed.

Caideyipi

Reviewed the latest head 6ac266b. The previously reported max-value initialization, Type.setTo direction, Tablet binary wire-format, WAL size, localization, and architecture-check issues are fixed. Targeted regression tests pass, and all CI checks except SonarCloud pass; SonarCloud failed with Java heap space OOM during analysis. No remaining functional issues found.

JackieTien97

Several correctness and performance regressions remain at 6ac266b065f06b9bc6b16d31a4b58c66e08b90fb. The inline comments cover eight main findings and two lower-priority implementation-contract issues.

The comments include reproducible table SQL, executed class-level results, and measured performance data where available. Each performance comment identifies the measured versions and workload; the historical microbenchmark numbers are not presented as a fresh benchmark of this exact HEAD or as end-to-end SQL/RPC latency ratios. Current source and the resolved TsFile snapshot were checked against the affected paths.

The previously fixed binary serialization/WAL-size/DATE-split/MAX_BY/grouped-MAX/DATE-statistics/tablet-decoding findings are omitted. The earlier SELECT INTO numeric-widening allegation was disproved by the real planner/SQL path and is also omitted.

JackieTien97 · 2026-09-15T04:46:39Z

-                dataType));
-    }
+  private double getDoubleValue(Column column, int position) {
+    return column.getDouble(position);


[P1] Preserve native timestamp reads for TimeColumn inputs

The previous implementation read INT64/TIMESTAMP using column.getLong(position) and widened the result to double. column.getDouble(position) is not equivalent for the actual TimeColumn implementation in the pinned TsFile snapshot: it inherits the default method that throws UnsupportedOperationException.

This is reachable in table SQL. AbstractAggTableScanOperator.buildValueColumn returns inputRegion.getTimeColumn() for a TIME argument and places that same object in the aggregation block's valueColumns. Replacing the block's dedicated time slot with a placeholder does not convert this value-column object into a LongColumn. Table metadata accepts TIMESTAMP arguments for these functions.

Table-model reproducer using the values from the real-instance comparison:

CREATE DATABASE review18606_corr; USE review18606_corr; CREATE TABLE readings(device_id STRING TAG, d DOUBLE FIELD); INSERT INTO readings(time,device_id,d) VALUES (1,'neg',-8.5),(2,'neg',-2.75); FLUSH; SELECT corr(time,d),covar_pop(time,d),regr_slope(time,d) FROM readings WHERE device_id='neg';

The earlier BASE 25ab7941ce vs PR 9f4c0f0404 real-instance run returned [1.0, 1.4375, 0.17391304347826086] on BASE and 301: org.apache.tsfile.read.common.block.column.TimeColumn on the PR. Tree SQL and grouped variants reproduced the same failure. This getter and the relevant dependency remain unchanged at 6ac266b065; the TsFile artifact still resolves to 2.4.1-260909-20260909.035234-2.

Please preserve declared-type/native reads, or complete the concrete Column contracts before relying on the generic accessor. The equivalent correlation/covariance/regression changes in the tree, table and grouped implementations need the same treatment.

Fixed by upgrading tsfile.version to 2.4.1-260915-SNAPSHOT, which contains apache/tsfile#961 (8fdbc49bc80e). TimeColumn.getDouble(position) now delegates to the native long getter and widens the result, preserving region offsets. I tested the actual published 2.4.1-260915-20260915.102750-1 JARs against the old 260909 snapshot using the same IoTDB aggregation classes. Tree, table and grouped corr/covar_pop/regr_slope pass with direct, region, dictionary and RLE time columns. Your example returns [1.0, 1.4375, 0.17391304347826086]. Across both P1 reproductions, the old snapshot fails all 76 targeted checks and the new one passes all 76. These are class-level checks; I have not rerun the live-instance SQL reproduction.

JackieTien97 · 2026-09-15T04:46:39Z

+      nullCounts.increment(groupId);
+    } else {
+      countMap.compute(
+          column.getTsPrimitiveType(position), (key, count) -> count == null ? 1L : count + 1);


[P1] Preserve logical row access when constructing MODE keys

The old typed getters are not equivalent to getTsPrimitiveType(position) for two Column implementations used by this code:

TimeColumn does not implement this method and throws.

DictionaryColumn.getTsPrimitiveType currently passes the logical position straight to its dictionary, without applying getId(position) as its typed getters do. MODE can therefore count values that are not present in its logical input.

Table-model time-column reproducer:

CREATE DATABASE review18606_mode; USE review18606_mode; CREATE TABLE readings(device_id STRING TAG, d DOUBLE FIELD); INSERT INTO readings(time,device_id,d) VALUES (1,'neg',-8.5),(2,'neg',-2.75); FLUSH; SELECT substring(device_id,1,1),mode(time) FROM readings GROUP BY substring(device_id,1,1) ORDER BY 1;

In the earlier real-instance comparison (BASE 25ab7941ce, PR 9f4c0f0404), the matching group returned time=1 on BASE, while the PR threw the TimeColumn exception.

A separate executed class-level test covers the dictionary mapping:

Underlying values: [10, 11, 12, 100, 100, 100] Selected positions: [3, 4, 5] Logical input: [100, 100, 100] BASE MODE: 100 PR MODE: 10

The real DistinctGroupedAccumulator wrapper also reproduces the dictionary result with the corresponding selected mask. This is evidence for that concrete class-level path, not a claim that every MODE(DISTINCT ...) SQL plan fails. The call and relevant TsFile implementations remain unchanged at the current HEAD.

Please construct keys through declared-type native getters, or fix and validate both concrete Column implementations, including non-identity/repeated dictionary mappings.

Fixed by the same TsFile upgrade to 2.4.1-260915-SNAPSHOT (apache/tsfile#961). TimeColumn.getTsPrimitiveType now returns a TsLong from getLong(position), and DictionaryColumn.getTsPrimitiveType applies getId(position) before reading its dictionary. The actual published JAR passes the time-column MODE checks and non-identity/repeated mappings for all 10 supported types, including nested dictionaries, dictionary/ID offsets and the real DistinctGroupedAccumulator created through AccumulatorFactory with a selected mask. The [3,4,5] example now returns 100 instead of 10. The combined P1 probe changes from 0/76 to 76/76 passing checks. This validates the concrete accumulator paths, not an end-to-end SQL rerun.

JackieTien97 · 2026-09-15T04:46:39Z

+                  case INT32 -> (column, rowIndex) -> ((int[]) column)[rowIndex];
+                  case DATE ->
+                      (column, rowIndex) ->
+                          new DateTime(Date.valueOf(((LocalDate[]) column)[rowIndex]));


[P2] Do not pass java.sql.Date to Milo DateTime

Date here is java.sql.Date. Milo's DateTime(Date) constructor calls date.toInstant(), but java.sql.Date.toInstant() always throws UnsupportedOperationException. A normal DATE value therefore fails in the OPC UA Client/Server tablet path before it can be published.

Executed with the actual Milo dependency during the earlier review:

LocalDate date = LocalDate.of(2026, 9, 9); new DateTime(java.sql.Date.valueOf(date));

java.lang.UnsupportedOperationException at java.sql.Date.toInstant(Date.java:316) at org.eclipse.milo.opcua.stack.core.types.builtin.DateTime.<init>(DateTime.java:57)

The old java.util.Date.from(date.atStartOfDay(ZoneId.systemDefault()).toInstant()) expression succeeds for the same value. The conversion expression is still present in this HEAD. Please retain the java.util.Date/Instant conversion. This was verified at the actual conversion boundary; it was not a full OPC UA server integration test.

Restored java.util.Date.from(localDate.atStartOfDay(ZoneId.systemDefault()).toInstant()) before constructing Milo's DateTime. This preserves the previous time-zone conversion and avoids java.sql.Date.toInstant(). Added a regression at the actual Milo conversion boundary in TypeServicesTest; it passes with the upgraded TsFile dependency. This is not a full OPC UA server integration test.

JackieTien97 · 2026-09-15T04:46:40Z

+      final int columnIndex,
+      final long outputMinReportIntervalMilliseconds)
+      throws IOException {
+    if (rowValueUpdater == null) {


[P2] Refresh the cached row updater when a series changes type

AggregateProcessor retains this state by timeseries path, but rowValueUpdater is selected only for the first row's type. If the same path is recreated with a different type, later rows continue through the original getter. For example, an INT32 row followed by a DOUBLE row still calls PipeRow.getInt, which casts the DOUBLE column's double[] to int[] and fails before the window can handle the type transition.

Previously, AggregateProcessor dispatched on each row's actual type. TimeSeriesWindow explicitly handles a changed input type by purging the old window. Caching the getter indefinitely prevents reaching that existing behavior.

Executed dispatch-level reproduction with real PipeRow instances and the real TimeSeriesRuntimeState.updateWindows(..., Row, ...) entry point:

First row: INT32 [42] -> INT32 updater Next row: DOUBLE [2.5] -> ClassCastException: [D cannot be cast to [I Direct DOUBLE overload -> receives 2.5 normally

The typed overloads were instrumented to record dispatch; this test did not run the complete Pipe lifecycle. The state key, one-time initialization and primitive-array getters establish the failure path, and remain unchanged in this HEAD.

Please cache the associated data type together with the updater and refresh it on a type change, or dispatch using each row's actual type.

The cached updater now stores its associated TSDataType and is refreshed whenever the incoming row's type changes. This lets a recreated path reach the existing typed/window handling instead of invoking a stale primitive getter. TimeSeriesRuntimeStateTest exercises the real Row entry point with PipeRow values changing INT32 -> DOUBLE -> INT32 and verifies the typed dispatch and values. The test passes; it does not run the full Pipe lifecycle.

JackieTien97 · 2026-09-15T04:46:40Z

+                }
+              } catch (DateTimeParseException e) {
+                throw new IoTDBRuntimeException(
+                    "Year must be between 1000 and 9999.",


[P2] Retain the localized messages when moving the implementation

This catch previously used CalcMessages.EXCEPTION_YEAR_MUST_BETWEEN_1000_9999_8FBB94AA, but now emits a raw English literal. The same regression remains in the Cannot cast %s to %s type branches in this file and several SUM/AVG argument checks. These strings stay English in a Chinese-locale build even when zh-locale-compile passes.

Some of the earlier message-key feedback has been fixed, but these call sites still bypass the existing English/Chinese constants. Please restore the corresponding localized constants and keep any new messages in both locale files.

Restored the existing CalcMessages constants for the year-range and cast failures and the SUM/AVG argument checks, including grouped SUM/AVG. The new batch strategies also use the existing localized unsupported-type messages. Both English and Chinese affected-module builds were validated; no new message keys are needed for these fixes.

JackieTien97 · 2026-09-15T04:46:40Z

-            String.format(
-                CalcMessages.EXCEPTION_UNSUPPORTED_DATA_TYPE_LAST_ARG_37F52124, seriesDataType));
-    }
+    addInput(arguments[0], arguments[1], mask);


[P2] Keep the descending LAST/LAST_BY early-return path

The old switch invoked protected addIntInput/addLongInput/etc., so a LastDescAccumulator executed its overridden loop and returned after the first qualifying row. The factory still creates the descending subclass, but this new call enters a private superclass loop and bypasses those overrides. The equivalent LastByAccumulator change bypasses the LastByDescAccumulator optimization when canFinishAfterInit is true.

An executed BASE 25ab7941ce vs PR 9f4c0f0404 comparison instrumented the time Column on the same 1,024-row, descending, non-null block:

Accumulator BASE time.getLong calls PR calls Result

LastDescAccumulator 1 1,024 unchanged

LastByDescAccumulator 1 1,024 unchanged

These are measured getter counts, not latency ratios. The outer scan can still inspect hasFinalResult() after the call and skip subsequent blocks; it cannot recover the work already done inside this block. The private loops remain unchanged at the current HEAD.

Please preserve the type-specific/overridable batch entry point or explicitly carry the descending early-return condition into the shared loop. Ascending or otherwise non-finalizable inputs must continue scanning.

The shared LAST/LAST_BY loops now return as soon as hasFinalResult() becomes true. This carries the descending subclasses' termination condition into the shared path while keeping non-finalizable inputs scanning. LastDescendingInputTest checks that a descending block reads only one qualifying timestamp and that non-finalizable LAST_BY still scans the block; both tests pass.

JackieTien97 · 2026-09-15T04:46:40Z

+      AGGREGATION_NUMERIC_COLUMN_TO_DOUBLE_CONVERTER_SERVICE =
+          type ->
+              switch (type.getTypeEnum()) {
+                case INT32, INT64, FLOAT, DOUBLE -> ignored -> type::getDouble;


[P2] Preserve mixed-type throughput when consolidating SUM's input loops

Before the refactor, SUM selected a type-specific loop once per block; the inner loop called getInt, getLong, getFloat or getDouble directly. Now all four types use the same converter expression and a common inner loop invoking valueConverter.convert(column, position) for each value. The converter is already stored per accumulator, so this is not an allegation of a new allocation or switch for every row.

I measured the actual BASE/PR SumAccumulator implementations in separate JVMs, with four accumulators (INT32/INT64/FLOAT/DOUBLE) alternating at 1,024-row batch boundaries. Both used the same TsFile dependency to isolate the Java refactor. Each run warmed up for 20.48 million rows, then took five samples of 4.096 million rows using ThreadMXBean current-thread CPU time. Final values were consumed into a volatile checksum (3.2664E7 on both sides), then accumulators were reset.

Runtime / workload BASE ns/value PR ns/value

Corretto 17.0.5 arm64, mixed types 0.89–0.92 5.60–5.69

OpenJDK 24.0.2 arm64, mixed types 0.887–0.902 5.49–5.65

OpenJDK 24.0.2 arm64, INT32 only about 0.90 about 0.90

These measurements were taken during the earlier review against BASE 25ab7941ce and the PR implementation at 0b6c4eb8b6; they are not a new benchmark of 6ac266b065. The affected SUM loop and this conversion strategy are still unchanged. No extra per-value allocation was measured after warm-up. The shared polymorphic call path is a plausible JIT explanation, but I did not inspect generated assembly to establish the exact optimization lost.

This is an aggregation-kernel result, not evidence that end-to-end SQL is six times slower. Please retain/select type-specific batch strategies, or add an equivalent warmed mixed-type benchmark demonstrating that the shared loop preserves throughput. The new cached-service helper does not change this per-value path.

Moved SUM's add/remove loops into the strategies returned by AGGREGATION_NUMERIC_COLUMN_TO_DOUBLE_CONVERTER_SERVICE. The accumulator selects its strategy once, and each batch uses a native getter loop rather than the shared per-value converter. The loop starts with the existing sum so floating-point additions are not regrouped across batches; empty/all-null unsupported inputs retain deferred rejection. SumAccumulatorTest covers mixed types, masks/nulls, removal and reset. A warmed local Windows/JBR 21 microbenchmark with four numeric types alternating every 1,024 rows improved from 3.738 to 0.440 ns/value against 6ac266b065, with matching checksums. This is an aggregation-kernel result, not SQL throughput. I also benchmarked analogous AVG/grouped aggregates, extrema, rate, moment and cast loops separately and retained 23 of 30 additional candidates; the seven rejected candidates were restored.

JackieTien97 · 2026-09-15T04:46:40Z

+      return;
+    }
+    int middle = (from + to) >>> 1;
+    sortIndexes(index, scratch, from, middle, valueProvider);


[P2] Preserve the fast path for long ordered runs in large tablets

Previously the index array was Integer[] sorted with Arrays.sort(..., Comparator.comparingLong(...)), whose TimSort implementation recognizes existing ordered runs. This primitive merge sort reduces boxing, but unconditionally recurses, merges and copies at every level, even when the input is almost sorted. The normal insert path calls checkSorted first, so a fully sorted tablet skips sorting on both versions; a single inversion is enough to enter this more expensive path.

Executed benchmark of the real Session.sortTablet: 262,144 rows, one INT32 value column, 60 warm-up sorts followed by 21 samples (median shown). Tablet construction/cloning was outside the timing region; sorting timestamps/indices and reordering the value column were inside. Results were consumed through a volatile sink. BASE and PR ran in separate JVMs.

Input / runtime BASE ns/row PR ns/row PR / BASE

One inversion, Corretto 17.0.5 arm64 5.644 34.935 6.19x

One inversion, OpenJDK 24.0.2 arm64 5.628 29.466 5.24x

Reverse ordered, OpenJDK 24.0.2 arm64 5.745 28.368 4.94x

Random, OpenJDK 24.0.2 arm64 244.440 151.547 0.62x (improves)

For the one-inversion JDK 24 case, this is approximately 1.48 ms -> 7.72 ms per sort. The input was an increasing timestamp sequence with only its final pair out of order. Smaller 1,024-row cases did not show a stable regression.

The measurements compare BASE 25ab7941ce with PR 9f4c0f0404; Session.java and this sorting path are unchanged at the current HEAD. They measure local sort latency, not complete RPC write throughput.

Please keep the primitive-array benefit while restoring ordered-run handling or skipping unnecessary merges, with equal-timestamp stability preserved. Random inputs improve, so simply reverting every part of the primitive-array change would lose that benefit.

Kept primitive indexes and added ordered-run detection before recursion, skipped already ordered merges, and used direct reversal only for strictly descending inputs so duplicate timestamp order remains stable. Session regression tests cover ordering and duplicate stability. On the local Windows/JBR 21 benchmark (262,144 rows, one INT32 column, 60 warm-ups and 21 samples; construction outside timing), almost-sorted input improved from 27.271 to 6.378 ns/row and descending input from 34.324 to 3.748; random input was effectively unchanged at 168.075 vs 168.481. These compare 6ac266b065 with the fix and measure local sorting, not RPC throughput.

JackieTien97 · 2026-09-15T04:46:40Z

-            break;
-          default:
-            throw new UnSupportedDataTypeException(String.format(DATATYPE_UNSUPPORTED, dataType));
+        if (!Type.fromTsDataType(dataType).arrayEquals(this.columns[i], columns[i], rowCount)) {


[P3] Keep hashCode consistent with active-row equality

equals now compares column values only through rowCount, but hashCode still uses Arrays.deepHashCode(columns) over the complete backing arrays. Two otherwise identical nodes with the same active rows and different spare-capacity values can therefore be equal while having different hashes.

The earlier real-node comparison reproduced equals == true, different hash codes, and HashSet.contains == false for such a pair. BASE considered the nodes unequal because it compared the complete arrays.

Please make the hash cover the same active values as equality, or retain consistent full-array semantics in both methods. This is a Java equality/hash contract issue; I have not identified a production caller using these nodes as hash keys, so I am not claiming an observed write/deduplication failure.

Changed the column hash to consume exactly the first rowCount values from each original backing array, without copying. The native loops are selected through StorageEngine.ARRAY_PREFIX_HASHER_SERVICE, keeping the TSDataType switch inside TypeService as required by this PR's architecture rule. DATE hashes normalize LocalDate and integer representations consistently with equality. Regression tests cover equal active rows with different spare values/capacities and HashSet lookup, alongside the existing serialization tests.

JackieTien97 · 2026-09-15T04:46:40Z

+    try {
+      return TypeServices.NUMERIC_ROW_READER_SERVICE
+          .call(TypeServices.toReadType(row.getDataType(index)))
+          .read(row);


[P3] Pass the requested column index to the numeric reader

This public helper selects the type from row.getDataType(index), but NUMERIC_ROW_READER_SERVICE always reads column 0. The previous switch read row.getInt(index) / getLong(index) / etc. For a real RowImpl with two INT32 values [11, 22], the executed BASE/PR comparison of getValueAsDouble(row, 1) returned 22.0 before the refactor and 11.0 afterwards.

Please use the indexed numeric reader and pass index through. The current single-input Envelope transform no longer calls this helper, and I did not find another in-repository production caller, so this is a low-priority helper-contract regression rather than a demonstrated ordinary Envelope SQL failure.

Switched this helper to INDEXED_NUMERIC_ROW_READER_SERVICE and passed index to read(row, index). The regression test supplies two INT32 values [11, 22] and verifies that column 1 returns 22.0; the Envelope test suite passes. This fixes the public helper contract without claiming an ordinary Envelope SQL failure.

Caideyipi

I rechecked the current head (6ac266b) and confirmed that this PR still has correctness issues, so I cannot approve it yet:

TableCorrelationAccumulator, TableCovarianceAccumulator, and TableRegressionAccumulator now call Column.getDouble() for every numeric input. The pinned TsFile TimeColumn implements getLong() but inherits the unsupported getDouble(), so table queries using time/TIMESTAMP as an argument throw UnsupportedOperationException.
GroupedModeAccumulator now uses Column.getTsPrimitiveType(position). TimeColumn does not implement that accessor, and DictionaryColumn does not apply its logical id mapping in that method, so MODE can either throw or count the wrong dictionary value.
TypeServices.OPC_UA_TABLET_OBJECT_VALUE_GETTER_SERVICE passes java.sql.Date.valueOf(...) to Milo DateTime; Milo calls Date.toInstant(), which throws for java.sql.Date, breaking DATE OPC-UA tablet values.
TimeSeriesRuntimeState caches the row updater only once. If a timeseries is recreated with another type, subsequent rows still use the first typed getter and can fail with an array cast exception instead of reaching the existing type-change window handling.
InsertTabletNode.equals() compares typed columns only through rowCount, while hashCode() uses the complete backing arrays. Equal nodes can therefore have different hashes, violating the Java equality contract.

These are independent of the already fixed serialization and MAX/Type.setTo issues. Please address the correctness regressions before merging.

jt2594838 · 2026-09-16T06:48:10Z

@@ -146,7 +147,7 @@
        <thrift.version>0.24.0</thrift.version>
        <xz.version>1.9</xz.version>
        <zstd-jni.version>1.5.6-3</zstd-jni.version>
-        <tsfile.version>2.4.1-260909-SNAPSHOT</tsfile.version>
+        <tsfile.version>2.4.1-260915-SNAPSHOT</tsfile.version>


Upgrade to the published TsFile snapshot containing apache/tsfile#961 so the generic Column access used by this refactor supports TimeColumn and dictionary logical positions. Both P1 threads contain the old/new artifact verification and its class-level scope.

jt2594838 · 2026-09-16T06:48:12Z

-                CalcMessages.EXCEPTION_UNSUPPORTED_DATA_TYPE_AGGREGATION_AVG_ARG_4E63A3C3,
-                argumentDataType));
-    }
+    inputStrategy.addInput(this, arguments[0], mask);


Move AVG add/remove scans into native TypeService strategies and write sum/count back once per batch. Starting from the existing sum preserves floating-point order. NumericBatchAggregationTest covers masks, nulls, removal and ordering. Local mixed-type kernels improved 6.13–8.78x; fixed DOUBLE was approximately flat for full selection and improved with masks.

jt2594838 · 2026-09-16T06:48:13Z

  public GroupedSumAccumulator(TSDataType argumentDataType) {
    this.argumentDataType = argumentDataType;
+    Type type = Type.fromTsDataType(argumentDataType);
+    this.inputStrategy = TypeServices.GROUPED_SUM_INPUT_SERVICE.call(type);


Use a typed batch loop for grouped SUM, indexing groupIds by the logical selected position and preserving null/initialization behavior. NumericBatchAggregationTest covers masks and group placement. Local mixed-type kernels improved 3.11–3.56x, with a tradeoff: fixed-DOUBLE full/masked runs were about 3.5%/7.4% slower. These are warmed local microbenchmarks, not SQL throughput.

jt2594838 · 2026-09-16T06:48:15Z

  public GroupedAvgAccumulator(TSDataType argumentDataType) {
    this.argumentDataType = argumentDataType;
+    Type type = Type.fromTsDataType(argumentDataType);
+    this.inputStrategy = TypeServices.GROUPED_AVG_INPUT_SERVICE.call(type);


Use a typed batch loop for grouped AVG so input type dispatch is outside the row loop. Group IDs still use logical input positions, and count/sum updates preserve input order. NumericBatchAggregationTest verifies masks, group placement and floating-point order; local mixed-type kernels improved 3.08–3.38x, with fixed-DOUBLE performance flat or better.

jt2594838 · 2026-09-16T06:48:16Z

-    this.minResult = TsPrimitiveType.getByType(seriesDataType);
+    this.type = Type.fromTsDataType(seriesDataType);
+    this.minResult = type.getTsPrimitiveType();
+    this.valueUpdater = TypeServices.MIN_COLUMN_BATCH_UPDATER_SERVICE.call(type);


MIN now selects a TypeService strategy containing the native scan, removing the shared per-value updater dispatch. NumericBatchAggregationTest verifies masked/null inputs and the existing NaN, signed-zero and integer-boundary behavior. The local mixed-type benchmark improved; fixed-DOUBLE results were smaller or approximately flat.

jt2594838 · 2026-09-16T06:48:17Z

-    this.maxResult = TsPrimitiveType.getByType(seriesDataType);
+    this.type = Type.fromTsDataType(seriesDataType);
+    this.maxResult = type.getTsPrimitiveType();
+    this.valueUpdater = TypeServices.MAX_COLUMN_BATCH_UPDATER_SERVICE.call(type);


MAX now selects a TypeService strategy containing the native scan, removing the shared per-value updater dispatch. NumericBatchAggregationTest verifies masked/null inputs and the existing NaN, signed-zero and integer-boundary behavior. The local mixed-type benchmark improved; fixed-DOUBLE results were smaller or approximately flat.

jt2594838 · 2026-09-16T06:48:19Z

-    this.extremeResult = TsPrimitiveType.getByType(seriesDataType);
+    this.type = Type.fromTsDataType(seriesDataType);
+    this.extremeResult = type.getTsPrimitiveType();
+    this.valueUpdater = TypeServices.EXTREME_COLUMN_BATCH_UPDATER_SERVICE.call(type);


EXTREME now selects a TypeService strategy containing the native scan, removing the shared per-value updater dispatch. NumericBatchAggregationTest verifies masked/null inputs and the existing NaN, signed-zero and integer-boundary behavior. The local mixed-type benchmark improved; fixed-DOUBLE results were smaller or approximately flat.

jt2594838 · 2026-09-16T06:48:30Z

      TSDataType seriesDataType, VarianceAccumulator.VarianceType varianceType) {
    this.seriesDataType = seriesDataType;
+    this.doubleValueConverter =
+        TypeServices.NUMERIC_BATCH_READER_SERVICE


Use the native numeric batch reader for grouped variance, retaining the existing recurrence callback and logical row order rather than changing the numerical algorithm. NumericMomentBatchTest covers moments/variance across tree, table and grouped implementations, including masked/null inputs. This candidate was retained after mixed-type and fixed-DOUBLE measurements; related candidates with measured regressions were restored.

jt2594838 · 2026-09-16T06:48:32Z

      TSDataType seriesDataType, CentralMomentAccumulator.MomentType momentType) {
    this.seriesDataType = seriesDataType;
+    this.doubleValueConverter =
+        TypeServices.NUMERIC_BATCH_READER_SERVICE


Use the native numeric batch reader for table central moment, retaining the existing recurrence callback and logical row order rather than changing the numerical algorithm. NumericMomentBatchTest covers moments/variance across tree, table and grouped implementations, including masked/null inputs. This candidate was retained after mixed-type and fixed-DOUBLE measurements; related candidates with measured regressions were restored.

jt2594838 · 2026-09-16T06:48:33Z

-      initializeOrValidateWindow(groupId, currentWindowStart, currentWindowEnd);
-      samples.add(groupId, time, value);
-    }
+    TypeServices.RATE_INPUT_SERVICE


GroupedNaiveDeltaAccumulator now dispatches to a native batch reader through RATE_INPUT_SERVICE. It preserves value validation before time/window validation and keeps the existing state update callback, including logical group positions where applicable. RateBatchInputTest exercises all 16 implementations with native and wrapped columns, masks/nulls and validation cases. This is one of the 12 retained rate candidates; the other four were restored after fixed-DOUBLE benchmarks showed regressions.

jt2594838 · 2026-09-16T06:48:35Z

-      initializeOrValidateWindow(groupId, currentWindowStart, currentWindowEnd);
-      samples.add(groupId, time, value);
-    }
+    TypeServices.RATE_INPUT_SERVICE


GroupedNaiveIncreaseAccumulator now dispatches to a native batch reader through RATE_INPUT_SERVICE. It preserves value validation before time/window validation and keeps the existing state update callback, including logical group positions where applicable. RateBatchInputTest exercises all 16 implementations with native and wrapped columns, masks/nulls and validation cases. This is one of the 12 retained rate candidates; the other four were restored after fixed-DOUBLE benchmarks showed regressions.

jt2594838 · 2026-09-16T06:48:36Z

-              arguments[1], position, RateFunctionType.IRATE, 2);
-      samples.add(groupId, time, value);
-    }
+    TypeServices.RATE_INPUT_SERVICE


GroupedNaiveIrateAccumulator now dispatches to a native batch reader through RATE_INPUT_SERVICE. It preserves value validation before time/window validation and keeps the existing state update callback, including logical group positions where applicable. RateBatchInputTest exercises all 16 implementations with native and wrapped columns, masks/nulls and validation cases. This is one of the 12 retained rate candidates; the other four were restored after fixed-DOUBLE benchmarks showed regressions.

jt2594838 · 2026-09-16T06:48:38Z

-      initializeOrValidateWindow(groupId, currentWindowStart, currentWindowEnd);
-      samples.add(groupId, time, value);
-    }
+    TypeServices.RATE_INPUT_SERVICE


GroupedNaiveRateAccumulator now dispatches to a native batch reader through RATE_INPUT_SERVICE. It preserves value validation before time/window validation and keeps the existing state update callback, including logical group positions where applicable. RateBatchInputTest exercises all 16 implementations with native and wrapped columns, masks/nulls and validation cases. This is one of the 12 retained rate candidates; the other four were restored after fixed-DOUBLE benchmarks showed regressions.

jt2594838 · 2026-09-16T06:48:39Z

-              arguments[1], position, RateFunctionType.IRATE, 2);
-      update(groupId, time, value);
-    }
+    TypeServices.RATE_INPUT_SERVICE


GroupedOrderedIrateAccumulator now dispatches to a native batch reader through RATE_INPUT_SERVICE. It preserves value validation before time/window validation and keeps the existing state update callback, including logical group positions where applicable. RateBatchInputTest exercises all 16 implementations with native and wrapped columns, masks/nulls and validation cases. This is one of the 12 retained rate candidates; the other four were restored after fixed-DOUBLE benchmarks showed regressions.

jt2594838 · 2026-09-16T06:48:41Z

-      initializeOrValidateWindow(currentWindowStart, currentWindowEnd);
-      samples.add(time, value);
-    }
+    TypeServices.RATE_INPUT_SERVICE


NaiveDeltaAccumulator now dispatches to a native batch reader through RATE_INPUT_SERVICE. It preserves value validation before time/window validation and keeps the existing state update callback, including logical group positions where applicable. RateBatchInputTest exercises all 16 implementations with native and wrapped columns, masks/nulls and validation cases. This is one of the 12 retained rate candidates; the other four were restored after fixed-DOUBLE benchmarks showed regressions.

jt2594838 · 2026-09-16T06:48:42Z

-      initializeOrValidateWindow(currentWindowStart, currentWindowEnd);
-      samples.add(time, value);
-    }
+    TypeServices.RATE_INPUT_SERVICE


NaiveIncreaseAccumulator now dispatches to a native batch reader through RATE_INPUT_SERVICE. It preserves value validation before time/window validation and keeps the existing state update callback, including logical group positions where applicable. RateBatchInputTest exercises all 16 implementations with native and wrapped columns, masks/nulls and validation cases. This is one of the 12 retained rate candidates; the other four were restored after fixed-DOUBLE benchmarks showed regressions.

jt2594838 · 2026-09-16T06:48:44Z

-              arguments[1], position, RateFunctionType.IRATE, 2);
-      samples.add(time, value);
-    }
+    TypeServices.RATE_INPUT_SERVICE


NaiveIrateAccumulator now dispatches to a native batch reader through RATE_INPUT_SERVICE. It preserves value validation before time/window validation and keeps the existing state update callback, including logical group positions where applicable. RateBatchInputTest exercises all 16 implementations with native and wrapped columns, masks/nulls and validation cases. This is one of the 12 retained rate candidates; the other four were restored after fixed-DOUBLE benchmarks showed regressions.

jt2594838 · 2026-09-16T06:48:46Z

-      initializeOrValidateWindow(currentWindowStart, currentWindowEnd);
-      samples.add(time, value);
-    }
+    TypeServices.RATE_INPUT_SERVICE


NaiveRateAccumulator now dispatches to a native batch reader through RATE_INPUT_SERVICE. It preserves value validation before time/window validation and keeps the existing state update callback, including logical group positions where applicable. RateBatchInputTest exercises all 16 implementations with native and wrapped columns, masks/nulls and validation cases. This is one of the 12 retained rate candidates; the other four were restored after fixed-DOUBLE benchmarks showed regressions.

jt2594838 · 2026-09-16T06:48:47Z

-      initializeOrValidateWindow(currentWindowStart, currentWindowEnd);
-      update(time, value);
-    }
+    TypeServices.RATE_INPUT_SERVICE


OrderedDeltaAccumulator now dispatches to a native batch reader through RATE_INPUT_SERVICE. It preserves value validation before time/window validation and keeps the existing state update callback, including logical group positions where applicable. RateBatchInputTest exercises all 16 implementations with native and wrapped columns, masks/nulls and validation cases. This is one of the 12 retained rate candidates; the other four were restored after fixed-DOUBLE benchmarks showed regressions.

jt2594838 · 2026-09-16T06:48:49Z

-      initializeOrValidateWindow(currentWindowStart, currentWindowEnd);
-      update(time, value);
-    }
+    TypeServices.RATE_INPUT_SERVICE


OrderedIncreaseAccumulator now dispatches to a native batch reader through RATE_INPUT_SERVICE. It preserves value validation before time/window validation and keeps the existing state update callback, including logical group positions where applicable. RateBatchInputTest exercises all 16 implementations with native and wrapped columns, masks/nulls and validation cases. This is one of the 12 retained rate candidates; the other four were restored after fixed-DOUBLE benchmarks showed regressions.

jt2594838 · 2026-09-16T06:48:51Z

-      initializeOrValidateWindow(currentWindowStart, currentWindowEnd);
-      update(time, value);
-    }
+    TypeServices.RATE_INPUT_SERVICE


OrderedRateAccumulator now dispatches to a native batch reader through RATE_INPUT_SERVICE. It preserves value validation before time/window validation and keeps the existing state update callback, including logical group positions where applicable. RateBatchInputTest exercises all 16 implementations with native and wrapped columns, masks/nulls and validation cases. This is one of the 12 retained rate candidates; the other four were restored after fixed-DOUBLE benchmarks showed regressions.

jt2594838 · 2026-09-16T06:48:52Z

    return value;
  }

+  public static void validateValue(double value, RateFunctionType functionType) {


Expose the existing finite/nonnegative value checks to the typed batch readers without changing exception order or messages. Keep readValue’s scalar checks inline: replacing them with a helper changed the scalar benchmark behavior on paths whose batch refactor was rejected. RateBatchInputTest covers these validation semantics.

jt2594838 · 2026-09-16T06:48:53Z

      Type returnType, ColumnTransformer childColumnTransformer, ZoneId zoneId) {
    super(returnType, childColumnTransformer);
    this.zoneId = zoneId;
+    this.numericCastBatch =


Select the numeric source/target batch strategy once for CAST/TRY_CAST. The 16 INT32/INT64/FLOAT/DOUBLE pairs use native loops; other types keep the existing conversion path. NumericCastBatchTest covers overflow, NaN, null/selection, region and RLE inputs, including TRY_CAST null results. Local mixed-type kernels improved 2.11–2.83x; fixed DOUBLE-to-DOUBLE was mostly smaller gains or flat.

jt2594838 · 2026-09-16T07:23:54Z

  public VarianceAccumulator(TSDataType seriesDataType, VarianceType varianceType) {
    this.seriesDataType = seriesDataType;
+    this.doubleValueConverter =
+        TypeServices.NUMERIC_BATCH_READER_SERVICE


Use the native numeric batch reader for tree variance, retaining the existing recurrence callback and logical row order rather than changing the numerical algorithm. NumericMomentBatchTest covers moments/variance across tree, table and grouped implementations, including masked/null inputs. This candidate was retained after mixed-type and fixed-DOUBLE measurements; related candidates with measured regressions were restored.

jt2594838 added 30 commits July 13, 2026 16:02

Add type services

d8aef23

remove switches

6f47103

remove switches

7414025

remove switches

bd99532

refactor calculateValueLength

e3ab985

refactor calculateCapacity

5e179c0

refactor calculateCapacity

f2925d0

refactor TabletExample

9312f56

refactor IoTDBDescriptor.getDefaultEncodingByType

6138d92

refactor estimateSingleValueArrayMemoryUsageInBytes

4000fe9

refactor estimateTsPrimitiveTypeValueMemoryUsageInBytes

9d76c8a

refactor parse

1a5ece5

fix compilation

308061a

refactor convertTsDataType2VariantType

1fa4125

refactor initResultTsBlock

16460a4

refactor getRowSkipPrefixNulls

f453368

refactor TwoSum

6af8fc2

refactor UDAFAvg

2a3c35f

refactor UDAFSum

cc63846

refactor encodeRecord

236b866

refactor fillMeasurementValueColumns

a4b417a

refactor copyValueList

88a6514

refactor transferTabletForPubSubModel

2bd4b6c

refactor PRIMITIVE_TYPE_VALUE_EXTRACTOR_SERVICE

253a4e0

refactor reorderValueListAndBitMap

308571b

refactor writeIntermediateToStream

3e250e4

refactor constructInsertTabletStatement

dfe6aec

refactor processColumn

706092b

refactor getColumnsBytes

83d027b

refactor addTimeValuePairToResult

46dfe11

Caideyipi approved these changes Sep 15, 2026

View reviewed changes

JackieTien97 requested changes Sep 15, 2026

View reviewed changes

Caideyipi requested changes Sep 15, 2026

View reviewed changes

jt2594838 added 2 commits September 16, 2026 10:12

tmp

4595213

Fix TypeService review regressions and batch numeric input loops

c4ec742

jt2594838 commented Sep 16, 2026

View reviewed changes

Merge branch 'master' into remove_swtich_type

19d97dd

Accumulator	BASE time.getLong calls	PR calls	Result
LastDescAccumulator	1	1,024	unchanged
LastByDescAccumulator	1	1,024	unchanged

Runtime / workload	BASE ns/value	PR ns/value
Corretto 17.0.5 arm64, mixed types	0.89–0.92	5.60–5.69
OpenJDK 24.0.2 arm64, mixed types	0.887–0.902	5.49–5.65
OpenJDK 24.0.2 arm64, INT32 only	about 0.90	about 0.90

Input / runtime	BASE ns/row	PR ns/row	PR / BASE
One inversion, Corretto 17.0.5 arm64	5.644	34.935	6.19x
One inversion, OpenJDK 24.0.2 arm64	5.628	29.466	5.24x
Reverse ordered, OpenJDK 24.0.2 arm64	5.745	28.368	4.94x
Random, OpenJDK 24.0.2 arm64	244.440	151.547	0.62x (improves)

Uh oh!

Conversation

jt2594838 commented Sep 9, 2026

What is changed

Validation

Uh oh!

Caideyipi left a comment

Choose a reason for hiding this comment

Uh oh!

JackieTien97 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Caideyipi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment