Skip to content

fix(kubescape): recover from a failed client init without a pod restart - #78

Open
slashben wants to merge 1 commit into
kagent-dev:mainfrom
slashben:fix/kubescape-client-reinit
Open

fix(kubescape): recover from a failed client init without a pod restart#78
slashben wants to merge 1 commit into
kagent-dev:mainfrom
slashben:fix/kubescape-client-reinit

Conversation

@slashben

@slashben slashben commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Summary

The kubescape provider built its Kubernetes clients once at registration; any failure disabled all ten tools until the pod was restarted. Clients are now built lazily and retried, and argument validation runs before the client check.

What's wrong

The provider builds its Kubernetes clients once, at registration. If that fails, initError is set and all ten handlers short-circuit on it — forever:

tool := NewKubescapeTool(kubeconfig)        // on failure, sets initError
...
if k.initError != nil { return toolErr.ToMCPResult(), nil }

Nothing clears it, nothing retries. In-cluster this happens on ordinary start ordering — the pod comes up before the API server is reachable, or before its RBAC is applied. The cluster recovers seconds later; the provider stays dead until someone restarts the pod, with nothing in the error saying so.

The fix

Clients are built on first use and cached. A failure is reported without retrying for 30 s, then retried — so the provider heals itself, while a loop of failing calls can't hammer the API server (each attempt can block for a full dial timeout).

Also fixed: the client check ran before argument validation, so with no cluster a call missing manifest_name reported failed to create kubernetes config: no configuration has been provided instead of the missing argument. Validation needs no cluster, so it now runs first in the five handlers with required arguments.

Risk / compatibility

  • NewKubescapeTool can no longer fail — a broken config surfaces at the first tool call instead of at registration. That's the intent, but it moves the signal; kubescape_check_health is still the way to ask if the install is reachable.
  • No output shape changes, apart from missing-argument calls now reporting the missing argument.
  • No new dependencies, confined to pkg/kubescape, and no other provider used this pattern.

The 30 s interval is the one number worth arguing about — it trades recovery latency against load on an already-unhealthy API server. I rejected retrying on every call because during an outage every call would pay a full dial timeout.

Testing

go test ./pkg/kubescape/... (59), -race, go vet, golangci-lint, and go test -tags=test ./pkg/... ./internal/... (738) all pass. No cluster needed; test/e2e needs kind and was not run.

Client construction and the clock are injected so the two behaviours that matter are testable without a cluster or a sleep: a build that fails once then succeeds proves recovery, and a second call inside the interval proves no re-attempt. The ordering test asserts the builder is never called — I confirmed it fails by reverting the ordering in one handler.

Relationship to the other PRs

Independent of #75/#76/#77; all four branch from main. Touches the same file as #76/#77, so whichever lands later needs a trivial rebase.

Ticket

None.

AI-skills: superpowers:brainstorming,superpowers:test-driven-development,armosec-shared-rules:agent-dispatch-policy | cmds: /compact

The provider built its Kubernetes clients once, in RegisterTools. If that
failed, initError was set and all ten handlers short-circuited on it for
the lifetime of the process:

    tool := NewKubescapeTool(kubeconfig)   // sets initError, never retried
    ...
    if k.initError != nil { return toolErr.ToMCPResult(), nil }

Nothing cleared or retried it. In-cluster this is reachable on ordinary
start ordering -- the pod comes up before the API server is reachable, or
before its RBAC has been applied -- and the result is that every Kubescape
tool stays dead until someone restarts the pod. The cluster recovers; the
provider does not.

Clients are now built on first use and cached. A failed attempt is reported
without retrying for 30s and retried after that, so the provider heals
itself once the cluster is reachable, while a loop of failing tool calls
cannot hammer the API server -- each attempt can block for a full dial
timeout.

NewKubescapeTool no longer fails, so a broken config now surfaces on the
first tool call instead of at registration. That is the point of the change,
but it does move the signal: kubescape_check_health remains the way to ask
whether the installation is reachable.

Also fixes the ordering it exposed: the client check ran before argument
validation, so with no cluster a call missing a required argument reported

    **Error**: failed to create kubernetes config: ... no configuration has
    been provided

instead of "manifest_name parameter is required". Argument validation needs
no cluster, so it now runs first in the five handlers that have required
arguments.

Construction and the clock are injected so the tests cover what matters
without a cluster and without sleeping: a build that fails once and then
succeeds proves recovery, and a second call inside the interval proves the
failure is not retried. Both fail against the previous code.

Docs-exempt: bug fix; no existing doc describes provider client lifecycle

Signed-off-by: Ben Hirschberg <ben@armosec.io>
Signed-off-by: Ben <ben@armosec.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant