Skip to content

Elasticsearch restore never finalizes when ILM deletes restored backing indices #39

Description

@viliakov

After #36, Elasticsearch restore completion requires every expected snapshot index to remain present with all primary shards active. This makes check-and-finalize --wait stick in IN_PROGRESS when ILM deletes restored data-stream backing indices before the check completes.

Reproduced repeatedly on stac-25604 with snapshot sts-backup-20260902-0300-pu2shn0iqlyuykt1ooqxuq:

  • snapshot state: SUCCESS (267/267 shards)
  • _cat/recovery?active_only=true: empty
  • all 145 recorded snapshot recoveries: done
  • cluster and data stream: green; no restore tasks or pending tasks
  • expected STS indices from snapshot: 109
  • expected indices still present: 92
  • missing indices: 17 .ds-sts_k8s_logs-* backing indices from August 26–27
  • sts_k8s_logs-policy deletes indices after 7 days

All 30 non-data-stream expected indices were present and healthy. The old log backing indices were eligible for ILM deletion, so the current exact-name predicate can never become true. Workloads remain scaled to zero and the restore lock remains held until the operator uses check-and-finalize --finalize-only. This happens more often than not during these restores.

Acceptance criteria:

  • completion remains protected against the pre-recovery false-success race fixed by STAC-25639: fix false-success on elasticsearch restore #36
  • ILM deletion of restored data-stream backing indices cannot keep a completed restore in progress
  • missing ordinary STS indices still prevent successful finalization
  • regression coverage includes a snapshot index removed by ILM during restore monitoring
  • the CLI reports enough progress detail to distinguish active recovery from indices removed by lifecycle management

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions