fix(kubescape): recover from a failed client init without a pod restart - #78
Open
slashben wants to merge 1 commit into
Open
fix(kubescape): recover from a failed client init without a pod restart#78slashben wants to merge 1 commit into
slashben wants to merge 1 commit into
Conversation
The provider built its Kubernetes clients once, in RegisterTools. If that
failed, initError was set and all ten handlers short-circuited on it for
the lifetime of the process:
tool := NewKubescapeTool(kubeconfig) // sets initError, never retried
...
if k.initError != nil { return toolErr.ToMCPResult(), nil }
Nothing cleared or retried it. In-cluster this is reachable on ordinary
start ordering -- the pod comes up before the API server is reachable, or
before its RBAC has been applied -- and the result is that every Kubescape
tool stays dead until someone restarts the pod. The cluster recovers; the
provider does not.
Clients are now built on first use and cached. A failed attempt is reported
without retrying for 30s and retried after that, so the provider heals
itself once the cluster is reachable, while a loop of failing tool calls
cannot hammer the API server -- each attempt can block for a full dial
timeout.
NewKubescapeTool no longer fails, so a broken config now surfaces on the
first tool call instead of at registration. That is the point of the change,
but it does move the signal: kubescape_check_health remains the way to ask
whether the installation is reachable.
Also fixes the ordering it exposed: the client check ran before argument
validation, so with no cluster a call missing a required argument reported
**Error**: failed to create kubernetes config: ... no configuration has
been provided
instead of "manifest_name parameter is required". Argument validation needs
no cluster, so it now runs first in the five handlers that have required
arguments.
Construction and the clock are injected so the tests cover what matters
without a cluster and without sleeping: a build that fails once and then
succeeds proves recovery, and a second call inside the interval proves the
failure is not retried. Both fail against the previous code.
Docs-exempt: bug fix; no existing doc describes provider client lifecycle
Signed-off-by: Ben Hirschberg <ben@armosec.io>
Signed-off-by: Ben <ben@armosec.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The kubescape provider built its Kubernetes clients once at registration; any failure disabled all ten tools until the pod was restarted. Clients are now built lazily and retried, and argument validation runs before the client check.
What's wrong
The provider builds its Kubernetes clients once, at registration. If that fails,
initErroris set and all ten handlers short-circuit on it — forever:Nothing clears it, nothing retries. In-cluster this happens on ordinary start ordering — the pod comes up before the API server is reachable, or before its RBAC is applied. The cluster recovers seconds later; the provider stays dead until someone restarts the pod, with nothing in the error saying so.
The fix
Clients are built on first use and cached. A failure is reported without retrying for 30 s, then retried — so the provider heals itself, while a loop of failing calls can't hammer the API server (each attempt can block for a full dial timeout).
Also fixed: the client check ran before argument validation, so with no cluster a call missing
manifest_namereportedfailed to create kubernetes config: no configuration has been providedinstead of the missing argument. Validation needs no cluster, so it now runs first in the five handlers with required arguments.Risk / compatibility
NewKubescapeToolcan no longer fail — a broken config surfaces at the first tool call instead of at registration. That's the intent, but it moves the signal;kubescape_check_healthis still the way to ask if the install is reachable.pkg/kubescape, and no other provider used this pattern.The 30 s interval is the one number worth arguing about — it trades recovery latency against load on an already-unhealthy API server. I rejected retrying on every call because during an outage every call would pay a full dial timeout.
Testing
go test ./pkg/kubescape/...(59),-race,go vet,golangci-lint, andgo test -tags=test ./pkg/... ./internal/...(738) all pass. No cluster needed;test/e2eneeds kind and was not run.Client construction and the clock are injected so the two behaviours that matter are testable without a cluster or a sleep: a build that fails once then succeeds proves recovery, and a second call inside the interval proves no re-attempt. The ordering test asserts the builder is never called — I confirmed it fails by reverting the ordering in one handler.
Relationship to the other PRs
Independent of #75/#76/#77; all four branch from
main. Touches the same file as #76/#77, so whichever lands later needs a trivial rebase.Ticket
None.
AI-skills: superpowers:brainstorming,superpowers:test-driven-development,armosec-shared-rules:agent-dispatch-policy | cmds: /compact