Add read-only HA replication checks - #43
Draft
viliakov wants to merge 4 commits into
Draft
Conversation
Inspect HDFS, Elasticsearch, Kafka and ClickHouse through namespace-scoped Kubernetes discovery and fixed read-only exec queries. Require complete evidence, reject observations whose Kubernetes membership changes, and support bounded waits with a stable healthy period and JSON output. Keep replication checks independent of backup configuration. Include dedicated Job RBAC and a SUSE BCI packaging example. Document that this is sampled replication evidence, not a node-removal safety certificate; live cluster validation remains outstanding. Tracking: StackVista/stackstate#501
Use chart name and component labels instead of requiring a Helm release. Distinguish the HDFS NameNode from its SecondaryNameNode even though both use a namenode container. Clear the inherited broker JMX settings for Kafka CLI invocations while preserving authentication options, and alias the ClickHouse queue subquery. ClickHouse retains last_queue_update_exception after successful queue updates; report it as historical context instead of blocking recovered replicas. Current coordination errors and backlog still fail. Verified against StorageReplicatedMergeTree::queueUpdatingTask in ClickHouse v26.5.1.882-stable. Full Go tests and golangci-lint pass. Read-only validation in stackstate-nightly reports HDFS, Elasticsearch and ClickHouse healthy; Kafka correctly flags three perf-test topics with one assigned replica. Added namespace/component discovery, SecondaryNameNode, JMX isolation and recovered-exception regression coverage.
Discover the namespace's ZooKeeper ensemble using chart labels and query every member with read-only mntr. Require at least three voters, matching configured voting counts, one leader and all expected synchronized followers. Recheck the leader after sampling to detect elections without depending on Kubernetes readiness. Enable the check by default and document the supported chart layout and sampled-maintenance boundary. Cover incomplete membership, lag, elections, unavailable metrics and malformed responses. Full Go tests and golangci-lint pass; nightly ZooKeeper passed a live stable-period wait with three voters.
…tion Timestamp progress and final table output, and expose the elapsed/remaining stability period, resets and completion. Keep the default 30-second requirement after the first completed healthy observation. Shorten the next polling delay when the stability deadline comes before the configured interval. Discard interrupted observations, preserve the last completed report and stop launching queries after cancellation. Return nonzero with an unknown overall result without flooding output with canceled request URLs. Full Go tests and golangci-lint pass. Virtual-time tests cover the default stability period, resets, long intervals and cancellation. Live Beest validation exited successfully after three healthy rounds (37.4 seconds of observed stability); a separate SIGINT check retained the last completed report and exited nonzero without query URLs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
sts-backup replication checkfor HDFS, Elasticsearch, Kafka, ClickHouse and ZooKeeper. It discovers one installation per namespace, returns table/JSON results and fails for degraded or unverified replication. Includes a Kubernetes Job/RBAC example and SUSE BCI container recipe.Wait mode shows timestamps and stability progress. Cancellation retains the last completed observation while returning nonzero. No release name or backup configuration is required.
This reports replication status and does not certify node-removal safety. Controlled recovery validation remains outstanding.
Tracking: https://github.com/StackVista/stackstate/issues/501
Validation: full Go tests and golangci-lint pass. Live Beest validation completed the default stability wait successfully; SIGINT retained the last completed report without aborted query URLs. ZooKeeper stable-period validation also passed against nightly.