K8s based service discovery#994
Conversation
…ement database health check endpoints
…uild processes, and improve database initialization scripts.
… and Dockerfile templates
…r topic ownership consistency during membership churn
…etirement state and sync.Once shutdown logic
|
@or-else requesting your review for this |
|
I'll try to review this weekend. |
There was a problem hiding this comment.
Pull request overview
Adds Kubernetes-native cluster membership discovery (EndpointSlice watch) and readiness/liveness health endpoints so Tinode can run as a StatefulSet and automatically reconcile cluster peers without a static cluster_config.nodes list.
Changes:
- Add Kubernetes discovery backend + reconciler to dynamically add/remove peers and rebuild the ring from EndpointSlices.
- Add
/livezand/readyzendpoints backed by new store/DBCheckHealth()plumbing across adapters. - Add K8s/Minikube deployment manifests/docs and update Go module dependencies for
client-go.
Reviewed changes
Copilot reviewed 38 out of 40 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| server/tinode.conf | Documents new cluster_config.discovery configuration. |
| server/store/store_health_test.go | Unit tests for Store.CheckHealth() behavior. |
| server/store/store.go | Adds CheckHealth() to store interface/implementation. |
| server/store/mock_store/mock_store.go | Updates gomock store interface to include CheckHealth(). |
| server/main.go | Registers health handlers and minor comment fix. |
| server/hub.go | Avoids nil deref by using a single owner/proxy snapshot from cluster. |
| server/http_health_test.go | Tests /livez and /readyz handlers and HEAD behavior. |
| server/http.go | Adds health endpoints + readiness check against persistent store. |
| server/db/rethinkdb/tests/rethink_test.go | Adds build tags for rethinkdb tests. |
| server/db/rethinkdb/adapter.go | Implements RethinkDB CheckHealth(). |
| server/db/postgres/tests/postgres_test.go | Adds build tags for postgres tests. |
| server/db/postgres/adapter.go | Implements Postgres CheckHealth() via Ping. |
| server/db/mysql/tests/mysql_test.go | Adds build tags for mysql tests. |
| server/db/mysql/adapter.go | Implements MySQL CheckHealth() via PingContext. |
| server/db/mongodb/tests/mongo_test.go | Adds build tags for mongodb tests. |
| server/db/mongodb/adapter.go | Implements MongoDB CheckHealth() via Ping. |
| server/db/common/common.go | Adds shared HealthCheckTimeout constant. |
| server/db/adapter.go | Extends DB adapter interface with CheckHealth(). |
| server/cluster_test.go | Adds tests for peer add/remove, snapshot apply, worker ID parsing, concurrency. |
| server/cluster_leader.go | Uses ring signature accessor compatible with locked ring updates. |
| server/cluster_discovery_test.go | Unit tests for EndpointSlice-based discovery using fake clientset. |
| server/cluster_discovery.go | Implements discovery abstraction (static + Kubernetes informer backend). |
| server/cluster.go | Adds discovery mode, reconciler, runtime peer retirement, ring locking helpers, k8s worker ID derivation. |
| go.sum | Updates dependency checksums (adds k8s + related transitive deps). |
| go.mod | Adds k8s.io/* dependencies and updates several module versions / Go version directive. |
| docs/cluster-peer-retirement-options.md | Design notes for safe peer retirement under churn. |
| docker/tinode/entrypoint-k8s.sh | K8s-specific entrypoint wrapper (POD_NAME validation, stdout/stderr behavior). |
| docker/tinode/Dockerfile.k8s | Adds a workspace-built Tinode image for Minikube/local K8s testing. |
| docker/tinode/Dockerfile | Copies in entrypoint-k8s.sh for use as a command override. |
| docker/local-k8s/README.postgres-minikube.md | Local Minikube validation guide for k8s discovery. |
| docker/local-k8s/15-postgres.yaml | Local Postgres deployment/service manifest for Minikube testing. |
| docker/k8s/README.md | Reference docs for StatefulSet + EndpointSlice discovery deployment. |
| docker/k8s/30-statefulset.yaml | StatefulSet manifest with probes, POD_NAME env, and preStop delay. |
| docker/k8s/20-configmap.yaml | ConfigMap enabling cluster_config.discovery.mode = kubernetes. |
| docker/k8s/10-service.yaml | Headless Service exposing ws/grpc/cluster ports for EndpointSlices. |
| docker/k8s/00-rbac.yaml | Namespace-scoped RBAC for watching EndpointSlices. |
| docker-compose.dev.yml | Adds a dev postgres compose file. |
| README.md | Links to Kubernetes manifests and Minikube guide. |
| INSTALL.md | Adds Kubernetes deployment instructions and constraints. |
| .gitignore | Ignores local editor/tooling directories. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| func k8sDefaultListenAddr() string { | ||
| return ":" + strconv.Itoa(clusterBaseListenPort) |
|
@kaustubhbabar5 while waiting for my review, please address copilot comments. Thanks! |
…very error handling, and enable configurable RPC ports
…or database initialization scripts
Abeabenut
left a comment
There was a problem hiding this comment.
kaustubhbabar5:k8s_based_service_discovery
Kubernetes Discovery PR Summary (#316)
Summary
Adds Kubernetes-native cluster discovery for Tinode. Users can now run Tinode as
a
StatefulSetand let pods discover peers automatically from the headlessService
EndpointSlice, instead of maintaining a staticcluster_config.nodeslist.How To Use
Apply the provided manifests:
Enable Kubernetes discovery in
tinode.conf:Scale the cluster with Kubernetes:
Notes
StatefulSet;Deploymentis not supported because Tinode derivesstable worker IDs from pod ordinals.
POD_NAMEdownward API environment variable.cluster.nodesandfailoverconfig are ignored in Kubernetes discovery mode.TotalClusterNodesin stats.