Skip to content

feat!: align integrations with IoTDB 2.0.11 and JDK 17 - #132

Open
CritasWang wants to merge 7 commits into
masterfrom
wx/iotdb-2.0.11-alignment
Open

CritasWang wants to merge 7 commits into
masterfrom
wx/iotdb-2.0.11-alignment

Conversation

@CritasWang

Copy link
Copy Markdown
Contributor

Description

Align the Java integrations with the released IoTDB 2.0.11 baseline (TsFile 2.4.0, Thrift 0.23.0) and move the reactor to JDK 17. Version properties in the root pom follow the iotdb-parent:2.0.11 values; iotdb-jdbc, iotdb-subscription and isession are now managed centrally.

Breaking changes

  • Every Java module requires JDK 17. CI covers JDK 17 and 21; Jenkins builds with JDK 17. The .java-9-and-above / .java-11-below / .java-16 profiles are gone.
  • MyBatis: TIME/TIMESTAMP columns map to Long (raw server-precision ticks), FLOAT to Float, DATE to LocalDate, BLOB to byte[]. Regenerate mapper interfaces and XML together; batchInsert is now a bounded default method over an internal batchInsertRows statement.
  • MyBatis-Plus example: IoTDBTableMapper exposes only operations the IoTDB table model supports (insert, selectList, selectByKey, deleteByKey, upsertFields, updateAttributes) through a restricted ISqlInjector.
  • Spring Boot starter: fetch-size defaults to the client's 5000 (was 1024); connection-timeout-in-ms defaults to 0 instead of failing startup when unset; sql-dialect is deprecated and both pool types are always exposed.

Spring Boot starter

IoTDBSessionPool becomes an @AutoConfiguration with @ConditionalOnMissingBean pool beans that are closed with the context. The compression flags follow the renamed 2.0.11 builder methods (enableThriftRpcCompaction for the tree pool, enableThriftCompression for the table pool). Defaults are documented in additional-spring-configuration-metadata.json and covered by an ApplicationContextRunner test. The same starter is validated against Spring Boot 4.1.1 / Spring 7.0.9 through the spring-boot4 profile.

MyBatis

New mybatis-support module with runtime adapters: IoTDBQueryInterceptor routes prepared queries through a single executeQuery() because the 2.0.11 driver's execute() returns true without populating getResultSet(); IoTDBLocalDateTypeHandler and IoTDBBlobTypeHandler use the driver paths that preserve NULL dates and arbitrary bytes. The generator plugin gains IoTDBKeyPlugin (nullable TAG components become IS NULL predicates; UPDATE statements are dropped with a warning because IoTDB UPDATE rejects time in the predicate and FIELD columns in SET) and a bounded, validating BatchInsertPlugin. A with-mybatis distribution profile ships the runtime jar next to the generator plugin.

Collector

IoTDBPushSource consumes record-format subscription messages through the 2.0.11 getRecordTabletIterator() API, validates the topic format and subscribes on the calling thread so that a task that cannot consume fails creation instead of staying registered while silently idle. PushSourceTask/PullSourceTask propagate start failures like SinkTask already did.

ThingsBoard, connectors, CI

The ThingsBoard module follows the reactor versions and its container ITs default to apache/iotdb:2.0.11-standalone; each IT now drops the databases it created so the shared container does not run out of region memory. TSFileConfigUtil covers the new setEncryptSalt setter. The CI compile-check matrix is JDK 17/21, adds the ThingsBoard and Grafana connector profiles, a Spring Boot 4 job, and a mybatis-integration job that runs the generator and MyBatis-Plus mappers against a real IoTDB 2.0.11 service container (bound to all interfaces and awaited with a real client session, not a TCP probe).

Validation performed locally

  • mvn clean verify -Pwith-springboot,with-all-connectors,with-examples,with-thingsboard,with-grafana-connector,with-mybatis on JDK 17 and JDK 21: all 40 modules pass (272 tests on JDK 21).
  • Both mybatis-integration matrix entries (iotdb-mybatis-it and iotdb-mybatis-it,spring-boot4) against an apache/iotdb:2.0.11-standalone container started like the CI service, ms precision.
  • Starter tests on Spring Boot 3.5.1 and 4.1.1; MyBatis-Plus example unit and live ITs on both Boot lines.
  • ThingsBoard -Piotdb-table-it Testcontainers suite: 58/58.
  • JDBC probes on 2.0.11 confirming the behaviors the adapters rely on (prepared execute() leaves getResultSet() null, update counts are -1, TTL=INF unquoted is rejected, grouped MAX of non-positive floats is fixed, SUM over INT64 returns DOUBLE).

Known limitations

  • On servers configured with us/ns time precision, a DELETE with a time predicate can report success without deleting the row on IoTDB 2.0.11; the ms suites pass and the READMEs document this server-side issue. The tests were not weakened.
  • The collector source's steady-state consumption path was not exercised against a live broker: the Apache 2.0.11 image has subscription disabled at compile time. Its failure path (missing topic, tsfile topic, subscription disabled) is what was verified.
  • The Spark 2.4 / Scala 2.11 modules compile but still need a runtime migration before deployment on JDK 17.

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster (tested against a standalone IoTDB 2.0.11 container).

Key changed/added classes (or packages if there are too many classes) in this PR
  • pom.xml (version baseline, JDK 17 enforcer), .github/workflows/compile-check.yml, Jenkinsfile
  • iotdb-spring-boot-starter: IoTDBSessionPool, IoTDBSessionProperties, IoTDBSessionPoolTest
  • mybatis-support (new): IoTDBQueryInterceptor, IoTDBLocalDateTypeHandler, IoTDBBlobTypeHandler
  • mybatis-generator: BatchInsertPlugin, IoTDBKeyPlugin, LombokPlugin
  • examples/mybatis-generator, examples/mybatisplus-generator (CodeGenerator, IoTDBTableMapper, IoTDBMybatisConfiguration, Velocity templates)
  • iotdb-collector: IoTDBPushSource, PipeRawTabletInsertionEvent, PushSourceTask, PullSourceTask
  • iotdb-thingsboard-table: IoTDBTableConfiguration, IoTDBTableTimeseriesDao, container ITs
  • distributions: with-mybatis profile and assembly

🤖 Generated with Claude Code

Align client dependencies and connector APIs with the released IoTDB
2.0.11 baseline. Add Spring Boot 4 compatibility and audit starter pool
configuration, including the default fetch size of 5000.

Add MyBatis query and type adapters, bounded batch generation, nullable
composite keys, and IoTDB-specific MyBatis-Plus operations and templates.
Update English documentation and the existing Chinese counterparts.

Validation: the full JDK 17 reactor, MyBatis offline and live ms suites,
and Spring Boot 3/4 compatibility checks on JDK 17/21 passed. Document
the upstream us/ns time-filtered DELETE issue without weakening tests.

BREAKING CHANGE: Java modules require JDK 17. Regenerate MyBatis mapper
interfaces and XML together; TIME/TIMESTAMP fields use Long, and the
MyBatis-Plus example exposes only IoTDB-supported mapper operations.
The service container only listened on its loopback because the image
defaults dn_rpc_address to 127.0.0.1, so the published port accepted the
TCP handshake and the JDBC session was reset. Bind it to all interfaces,
as the Testcontainers ITs already do, and replace the TCP probe with a
real client round trip through start-cli: the port is reachable several
seconds before the DataNode can open a session.
The MyBatis-Plus example had no parent since the Spring Boot starter
parent was dropped, so Spotless ran from a hand-copied configuration and
Checkstyle, RAT and the reactor's plugin management never touched it.
Inherit the examples parent and re-point the inherited Spring, Jackson
and Mockito management at the selected Boot line for both the default
Boot 3.5.1 build and the spring-boot4 profile; surefire moves to a
JUnit Platform 6 capable version for Boot 4.

The Spring Boot starter example now uses the reactor's google-java-format
1.28.0 / Spotless 2.44.5 instead of the pins the reactor moved away from.
IoTDB 2.0.11 UPDATE rejects time in its predicate and FIELD columns in
SET, so MBG's key-based UPDATE statements can never run. IoTDBKeyPlugin
now disables them with a generator warning instead of emitting SQL that
fails at runtime, and gains unit coverage for the nullable-TAG predicate
rewrite it already performed.

The mybatis-support runtime jar is required by every generated mapper
but was only reachable through mvn install; the new with-mybatis
distribution profile packages it next to the generator plugin.
The IoTDB push source opened its subscription on the worker thread, so a
missing topic, a tsfile-format topic, an unreachable broker or a server
without subscription support only produced one log line while the task
stayed registered and silently never delivered. Validate the topic
format and subscribe on the calling thread so start() reports the cause,
and let PushSourceTask/PullSourceTask propagate a start failure even
when closing the source succeeds, matching SinkTask and ProcessorTask.
The container ITs provisioned a fresh database per test and never
dropped it; on an 8 GB Docker host the DataNode ran out of region memory
after about a dozen databases and later tests timed out on their first
write. Track the databases each test creates and drop them afterwards.

apache/iotdb#18300 (grouped MAX of non-positive floats) ships in 2.0.11,
so the DAO comment no longer claims it is unreleased; the -MIN(-x)
projection stays for older servers. Remaining 2.0.8 references in the
README and migration guide now name the verified 2.0.11 behavior.
The starter README notes that connection-timeout-in-ms now defaults to 0
instead of failing startup with a NullPointerException when unset. The
compile.version property in the Flink and Hive connector poms had no
consumer.
}

private void requireRecordFormatTopic() throws Exception {
try (final SubscriptionTreeSession session = new SubscriptionTreeSession(host, port)) {
Comment on lines +212 to +213
new XMLMapperBuilder(
new StringReader(xml), configuration, "mapper.xml", configuration.getSqlFragments())
Comment on lines +118 to +122
new XMLMapperBuilder(
new StringReader(new DefaultXmlFormatter().getFormattedContent(document)),
configuration,
"mapper.xml",
configuration.getSqlFragments())

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Aligns Java integrations with IoTDB 2.0.11 and JDK 17, while adding MyBatis runtime/generator support and updating starter, collector, ThingsBoard, and CI integrations.

Changes:

  • Centralizes IoTDB 2.0.11 dependencies and JDK 17 requirements.
  • Adds MyBatis handlers, interceptor, generator plugins, and examples.
  • Updates Spring Boot starter defaults, collector subscription handling, and ThingsBoard integration tests.
File summaries
File Description
README.md Updated as part of this pull request.
README-zh.md Updated as part of this pull request.
mybatis-support/src/test/java/org/apache/iotdb/mybatis/type/IoTDBTypeHandlerTest.java Updated as part of this pull request.
mybatis-support/src/test/java/org/apache/iotdb/mybatis/IoTDBQueryInterceptorTest.java Updated as part of this pull request.
mybatis-support/src/main/java/org/apache/iotdb/mybatis/type/IoTDBLocalDateTypeHandler.java Updated as part of this pull request.
mybatis-support/src/main/java/org/apache/iotdb/mybatis/type/IoTDBBlobTypeHandler.java Updated as part of this pull request.
mybatis-support/src/main/java/org/apache/iotdb/mybatis/IoTDBQueryInterceptor.java Updated as part of this pull request.
mybatis-support/README.md Updated as part of this pull request.
mybatis-support/pom.xml Updated as part of this pull request.
mybatis-generator/src/test/java/org/apache/iotdb/mybatis/plugin/IoTDBKeyPluginTest.java Updated as part of this pull request.
mybatis-generator/src/main/resources/generatorConfig.xml Updated as part of this pull request.
mybatis-generator/src/main/java/org/apache/iotdb/mybatis/plugin/LombokPlugin.java Updated as part of this pull request.
mybatis-generator/src/main/java/org/apache/iotdb/mybatis/plugin/IoTDBKeyPlugin.java Updated as part of this pull request.
mybatis-generator/src/main/java/org/apache/iotdb/mybatis/plugin/BatchInsertPlugin.java Updated as part of this pull request.
mybatis-generator/README.md Updated as part of this pull request.
mybatis-generator/README-zh.md Updated as part of this pull request.
mybatis-generator/pom.xml Updated as part of this pull request.
Jenkinsfile Updated as part of this pull request.
iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableTtlIT.java Updated as part of this pull request.
iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableTimeseriesDaoTest.java Updated as part of this pull request.
iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableTimeseriesDaoIT.java Updated as part of this pull request.
iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableTimeseriesAggregationIT.java Updated as part of this pull request.
iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableLatestDaoIT.java Updated as part of this pull request.
iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableIngestionBenchmarkIT.java Updated as part of this pull request.
iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAttributesDaoIT.java Updated as part of this pull request.
iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableTimeseriesDao.java Updated as part of this pull request.
iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableConfiguration.java Updated as part of this pull request.
iotdb-thingsboard-table/README.md Updated as part of this pull request.
iotdb-thingsboard-table/pom.xml Updated as part of this pull request.
iotdb-thingsboard-table/docs/user-guide.md Updated as part of this pull request.
iotdb-thingsboard-table/docs/migration-guide.md Updated as part of this pull request.
iotdb-thingsboard-table/docs/benchmarks/README.md Updated as part of this pull request.
iotdb-thingsboard-table/docker-compose.test.yml Updated as part of this pull request.
iotdb-thingsboard-table/docker-compose.bench.yml Updated as part of this pull request.
iotdb-thingsboard-table/CI-NOTES.md Updated as part of this pull request.
iotdb-spring-boot-starter/src/main/resources/META-INF/additional-spring-configuration-metadata.json Updated as part of this pull request.
iotdb-spring-boot-starter/src/main/java/org/apache/iotdb/session/IoTDBSessionPool.java Updated as part of this pull request.
iotdb-spring-boot-starter/src/main/java/org/apache/iotdb/config/IoTDBSessionProperties.java Updated as part of this pull request.
iotdb-spring-boot-starter/README.md Updated as part of this pull request.
iotdb-spring-boot-starter/pom.xml Updated as part of this pull request.
iotdb-collector/collector-core/src/main/java/org/apache/iotdb/collector/runtime/task/source/push/PushSourceTask.java Updated as part of this pull request.
iotdb-collector/collector-core/src/main/java/org/apache/iotdb/collector/runtime/task/source/pull/PullSourceTask.java Updated as part of this pull request.
iotdb-collector/collector-core/src/main/java/org/apache/iotdb/collector/plugin/builtin/source/iotdb/IoTDBPushSource.java Updated as part of this pull request.
iotdb-collector/collector-core/src/main/java/org/apache/iotdb/collector/plugin/builtin/sink/event/PipeRawTabletInsertionEvent.java Updated as part of this pull request.
iotdb-collector/collector-core/pom.xml Updated as part of this pull request.
examples/rocketmq/readme.md Updated as part of this pull request.
examples/rabbitmq/readme.md Updated as part of this pull request.
examples/pom.xml Updated as part of this pull request.
examples/mybatisplus-generator/src/test/java/org/apache/iotdb/CodeGeneratorTest.java Updated as part of this pull request.
examples/mybatisplus-generator/src/test/java/org/apache/iotdb/ApplicationTest.java Updated as part of this pull request.
examples/mybatisplus-generator/src/main/resources/templates/iotdb-mapper.xml.vm Updated as part of this pull request.
examples/mybatisplus-generator/src/main/resources/templates/iotdb-mapper.java.vm Updated as part of this pull request.
examples/mybatisplus-generator/src/main/resources/templates/iotdb-entity.java.vm Updated as part of this pull request.
examples/mybatisplus-generator/src/main/resources/mappers/Table2Mapper.xml Updated as part of this pull request.
examples/mybatisplus-generator/src/main/resources/mappers/Table1Mapper.xml Updated as part of this pull request.
examples/mybatisplus-generator/src/main/resources/application.yml Updated as part of this pull request.
examples/mybatisplus-generator/src/main/java/org/apache/iotdb/xml/Table2Mapper.xml Updated as part of this pull request.
examples/mybatisplus-generator/src/main/java/org/apache/iotdb/xml/Table1Mapper.xml Updated as part of this pull request.
examples/mybatisplus-generator/src/main/java/org/apache/iotdb/service/Table2Service.java Updated as part of this pull request.
examples/mybatisplus-generator/src/main/java/org/apache/iotdb/service/Table1Service.java Updated as part of this pull request.
examples/mybatisplus-generator/src/main/java/org/apache/iotdb/service/impl/Table2ServiceImpl.java Updated as part of this pull request.
examples/mybatisplus-generator/src/main/java/org/apache/iotdb/service/impl/Table1ServiceImpl.java Updated as part of this pull request.
examples/mybatisplus-generator/src/main/java/org/apache/iotdb/mapper/Table2Mapper.java Updated as part of this pull request.
examples/mybatisplus-generator/src/main/java/org/apache/iotdb/mapper/Table1Mapper.java Updated as part of this pull request.
examples/mybatisplus-generator/src/main/java/org/apache/iotdb/mapper/IoTDBTableMapper.java Updated as part of this pull request.
examples/mybatisplus-generator/src/main/java/org/apache/iotdb/Main.java Updated as part of this pull request.
examples/mybatisplus-generator/src/main/java/org/apache/iotdb/entity/Table2.java Updated as part of this pull request.
examples/mybatisplus-generator/src/main/java/org/apache/iotdb/entity/Table1.java Updated as part of this pull request.
examples/mybatisplus-generator/src/main/java/org/apache/iotdb/config/IoTDBMybatisConfiguration.java Updated as part of this pull request.
examples/mybatisplus-generator/pom.xml Updated as part of this pull request.
examples/mybatis-generator/src/test/java/org/apache/iotdb/mybatis/MapperConfigurationTest.java Updated as part of this pull request.
examples/mybatis-generator/src/main/resources/mybatis-config.xml Updated as part of this pull request.
examples/mybatis-generator/src/main/resources/generatorConfigByExample.xml Updated as part of this pull request.
examples/mybatis-generator/src/main/resources/generatorConfig.xml Updated as part of this pull request.
examples/mybatis-generator/src/main/java/org/apache/iotdb/mybatis/plugin/xml/MixMapper.xml Updated as part of this pull request.
examples/mybatis-generator/src/main/java/org/apache/iotdb/mybatis/plugin/model/Mix.java Updated as part of this pull request.
examples/mybatis-generator/src/main/java/org/apache/iotdb/mybatis/plugin/mapper/MixMapper.java Updated as part of this pull request.
examples/mybatis-generator/src/main/java/org/apache/iotdb/mybatis/Main.java Updated as part of this pull request.
examples/mybatis-generator/README.md Updated as part of this pull request.
examples/mybatis-generator/pom.xml Updated as part of this pull request.
examples/kafka/readme.md Updated as part of this pull request.
examples/iotdb-spring-boot-start/src/main/resources/application.properties Updated as part of this pull request.
examples/iotdb-spring-boot-start/src/main/java/org/apache/iotdb/iotdbspringbootstartexample/service/IoTDBService.java Updated as part of this pull request.
examples/iotdb-spring-boot-start/README.md Updated as part of this pull request.
examples/iotdb-spring-boot-start/pom.xml Updated as part of this pull request.
distributions/src/assembly/mybatis-generator-plugin.xml Updated as part of this pull request.
distributions/pom.xml Updated as part of this pull request.
connectors/spark-iotdb-table-connector/pom.xml Updated as part of this pull request.
connectors/spark-iotdb-connector/pom.xml Updated as part of this pull request.
connectors/hive-connector/src/main/java/org/apache/iotdb/hive/TsFileSerDe.java Updated as part of this pull request.
connectors/hive-connector/pom.xml Updated as part of this pull request.
connectors/grafana-plugin/docker-compose.yaml Updated as part of this pull request.
connectors/grafana-connector/pom.xml Updated as part of this pull request.
connectors/flink-tsfile-connector/src/test/java/org/apache/iotdb/flink/util/TSFileConfigUtilCompletenessTest.java Updated as part of this pull request.
connectors/flink-tsfile-connector/src/main/java/org/apache/iotdb/flink/tsfile/util/TSFileConfigUtil.java Updated as part of this pull request.
connectors/flink-iotdb-connector/pom.xml Updated as part of this pull request.
.github/workflows/compile-check.yml Updated as part of this pull request.
Review details

Suppressed comments (2)

iotdb-collector/collector-core/src/main/java/org/apache/iotdb/collector/plugin/builtin/source/iotdb/IoTDBPushSource.java:132

  • This does not actually require the topic's format to equal TopicConstant.FORMAT_RECORD_HANDLER_VALUE; it only rejects a string containing tsfilehandler. Any other non-record format/legacy representation passes startup, but the worker below rejects its message type asynchronously, so the task can be created and then silently stop instead of failing creation as intended. Parse the serialized topic attributes and require an exact record-format value (while explicitly rejecting all other formats).
    mybatis-generator/src/main/resources/generatorConfig.xml:33
  • The default generator configuration still omits the jdbcType.FLOAT override. IoTDBJavaTypeResolver only replaces TIMESTAMP and otherwise inherits MBG's default FLOAT mapping, so users running this checked-in config will continue to generate Double fields instead of the new Float contract. Add the same FLOAT override already present in the example configurations.
  • Files reviewed: 102/104 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

}
// Like SinkTask/ProcessorTask: a source that cannot start must fail task creation
// instead of being swallowed when its cleanup succeeds.
throw e;
}
// Like SinkTask/ProcessorTask: a source that cannot start must fail task creation
// instead of being swallowed when its cleanup succeeds.
throw e;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants