Skip to content

Fix flaky IoTDBAutoRegionGroupExtension2IT during DataNode shutdown - #18642

Open
CRZbulabula wants to merge 1 commit into
apache:masterfrom
CRZbulabula:fix-flaky-auto-region-group-it
Open

CRZbulabula wants to merge 1 commit into
apache:masterfrom
CRZbulabula:fix-flaky-auto-region-group-it

Conversation

@CRZbulabula

Copy link
Copy Markdown
Contributor

Description

IoTDBAutoRegionGroupExtension2IT verifies that DataRegionGroups can still be extended while one DataNode is unavailable.

The test currently stops a DataNode gracefully and waits until the ConfigNode reports it as Unknown. During graceful shutdown, however, the DataNode temporarily changes its status to ReadOnly. An in-flight heartbeat sampled during this window may update the ConfigNode after the test has already observed the Unknown status.

When this happens, RegionBalancer excludes the ReadOnly DataNode and sees only two available candidates for a replication factor of three. The partition request then returns NO_ENOUGH_DATANODE (1003) instead of SUCCESS_STATUS, making the test fail intermittently.

Changes

  • Stop the unavailable DataNode forcibly in IoTDBAutoRegionGroupExtension2IT.
  • Continue waiting for the ConfigNode to observe the DataNode as Unknown before creating partitions.

A forced stop matches the unavailable-node scenario that this test is intended to cover and avoids the transient ReadOnly state introduced by graceful shutdown. This change does not modify production node-status or allocation semantics.

Tests

mvn clean verify \
  -DskipUTs \
  -Dit.test=IoTDBAutoRegionGroupExtension2IT \
  -DfailIfNoTests=false \
  -Dfailsafe.failIfNoSpecifiedTests=false \
  -pl integration-test -am \
  -PClusterIT \
  -P with-integration-tests

Result: 1 test run, 0 failures, 0 errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant