Skip to content

Add observability for memory + agent-loop lifecycle signals#18

Merged
jkyberneees merged 2 commits into
mainfrom
claude/agent-signal-observability-UizE4
Jun 6, 2026
Merged

Add observability for memory + agent-loop lifecycle signals#18
jkyberneees merged 2 commits into
mainfrom
claude/agent-signal-observability-UizE4

Conversation

@jkyberneees
Copy link
Copy Markdown
Contributor

Summary

Memory and several agent-loop signals were completely silent — no logs, no notifications, no programmatic surface — while the skills subsystem already had a full event pipeline (event → notifier → terminal/web/Telegram/programmatic). This PR brings memory and the loop up to parity using the same proven notifier pattern.

Memory lifecycle events (internal/memory)

  • New notifier.go: MemoryEvent + MemoryNotifier interface + Noop/Multi implementations, mirroring skills/notifier.go.
  • MemoryManager.SetNotifier propagates to the EpisodeStore so facts and episodes share one sink.
  • Events fire at every lifecycle point:
Event Fired when
fact_added a new durable fact is appended (skips silent dedups)
fact_merged merge-on-write folds a fact into a near-duplicate (with similarity)
fact_replaced / fact_removed a fact is replaced / removed
fact_consolidated LLM consolidation merges entries (before → after)
episode_stored a session episode is persisted (carries Untrusted)
episode_deduped a new episode replaces a near-duplicate
episode_evicted episodes pruned by TTL / count cap (batched)
episode_promoted a tainted episode is user-approved
episode_pending_review an untrusted episode is stored but excluded from recall
  • Drive-by fix: AddFact's merge-on-write branch now marks the system prompt dirty, so a merged fact is visible on the next turn (was previously invisible until an unrelated mutation).

Agent-loop signals (internal/loop)

  • New signal.go: SignalEvent + SignalHandler + Engine.SetSignalHandler.
  • Emits context_trimmed (proactive budget trim and post-error survival trim) and tool_recovery (repeated tool failure → corrective hint) — both previously silent.

Surfaces (all four)

  • Terminal: render.Renderer memory/episode/signal methods, gated behind a new memoryVerbose flag (enabled by verbose interaction mode).
  • Programmatic: Config.MemoryEventHandler + Config.AgentSignalHandler with adapters that fan out alongside the renderer.
  • Web UI: serve.go streams memory_event/agent_signal over the WebSocket; app.js dispatches them to toasts (high-frequency events kept silent).
  • Telegram: verbose-gated chat notifications for the meaningful events.

Tests & docs

  • internal/memory/notifier_test.go (facts/episodes, dedup-is-silent, manager→episode propagation), internal/loop/signal_test.go, and render tests for the new methods.
  • go vet ./... clean, gofmt clean, full go test ./... passes.
  • Docs updated: new observability section in docs/MEMORY.md (full event table) and AGENTS.md source layout.

Note

The terminal surface is opt-in (verbose mode) to avoid flooding default output — matching the skills precedent. Web/Telegram/programmatic handlers receive events regardless. Flipping terminal logs on by default is a one-line gating change if preferred.

https://claude.ai/code/session_01L66aFQQ4SyniU7m5Ah3iTX


Generated by Claude Code

claude added 2 commits June 6, 2026 14:06
Memory and several agent-loop signals were completely silent — no logs,
no notifications, no programmatic surface — while the skills subsystem
already had a full event pipeline. This brings memory and the loop up to
parity using the same proven notifier pattern.

Memory (internal/memory):
- New notifier.go: MemoryEvent + MemoryNotifier interface + Noop/Multi
  implementations, mirroring skills/notifier.go.
- MemoryManager.SetNotifier propagates to the EpisodeStore so facts AND
  episodes share one sink.
- Fire events at every lifecycle point: fact_added (skips silent dedups),
  fact_merged (merge-on-write, with similarity), fact_replaced,
  fact_removed, fact_consolidated (before→after), episode_stored
  (carries Untrusted), episode_deduped, episode_evicted (TTL/cap, batch),
  episode_promoted, episode_pending_review.
- Drive-by fix: AddFact's merge-on-write branch now marks the system
  prompt dirty so a merged fact is visible on the next turn.

Agent loop (internal/loop):
- New signal.go: SignalEvent + SignalHandler + Engine.SetSignalHandler.
- Emit context_trimmed (proactive budget trim + post-error survival trim)
  and tool_recovery (repeated tool failure → corrective hint).

Surfaces (all four):
- Terminal: render.Renderer memory/episode/signal methods, gated behind a
  new memoryVerbose flag (enabled by verbose interaction mode).
- Programmatic: Config.MemoryEventHandler + Config.AgentSignalHandler with
  adapters that fan out alongside the renderer.
- Web UI: serve.go streams memory_event/agent_signal over the WebSocket;
  app.js dispatches them to toasts (noisy events kept silent).
- Telegram: verbose-gated chat notifications for meaningful events.

Tests: notifier_test.go (facts/episodes, dedup-is-silent, propagation),
signal_test.go, and render tests for the new methods. Docs updated
(MEMORY.md observability section, AGENTS.md source layout).
Verification-protocol pass on the memory observability change surfaced a
structural finding plus two coverage gaps; this commit repairs them.

Finding (axis 2.4/2.5 — callback under lock): memory lifecycle events
were delivered while internal locks were held — EpisodeStore.notify under
e.mu (writeLocked/Prune/Promote) and the fact events under the shared
per-dir facts lock. A notifier runs arbitrary caller code (a WebSocket
send under `odek serve`, or a handler that re-enters the store), so firing
under the lock serialized writes behind the sink and risked a reentrancy
deadlock.

Fix (behavior-preserving): collect events while locked and fan them out
after unlocking.
- EpisodeStore.writeLocked now returns []MemoryEvent; WriteWithProvenance,
  Prune, and Promote fire via the new notifyAll once e.mu is released.
- MemoryManager fact methods collect into a pending slice fired by
  fireAfterUnlock after the facts-dir lock is released.

Coverage repairs (additive):
- Test the merge-on-write fact_merged path (previously untested), with a
  regression guard that the merged fact reaches the system prompt
  (markPromptDirty fix on that path).
- README: note the new memory observability surface (axis 2.9 doc gap).

Verified: full `go test ./...` green; `go test -race` clean on memory and
loop; go vet + gofmt clean.
Copy link
Copy Markdown
Contributor Author

📜 Verification Certificate — AI Verification Protocol v5.2.7

Ran the AI Verification Protocol against this PR's diff (acting as the B/C/D/E pipeline) and auto-repaired the safe/additive findings. Repairs landed in 5993a8c.

Classification: GeneratedCode — AI-authored diff. ⚠️ Single-actor pass: the same actor authored both the code and this review, so this is not an independent multi-family pipeline (§0.1/§3.5). ρ is structurally non-zero; the human auditor remains the verdict authority.

Axes Summary

Axis Status Finding & action
2.1 Semantic Correctness ✅ (repaired) fact_merged merge-on-write path was untested and I'd added a markPromptDirty fix there → added a test with a system-prompt regression guard.
2.2 Behavioral Contract Events match the documented MemoryEvent / SignalEvent contracts.
2.3 Security Surface Content passes ScanContent on write; web toasts use target / session_id (not raw fact text), avoiding an injection sink.
2.4 Structural Integrity ✅ (repaired) Callback-under-lock anti-pattern — see primary finding.
2.5 Behavioral Exploration ✅ (repaired) Reentrancy/contention risk closed; -race clean.
2.6 Dependency Integrity No new dependencies.
2.7 Generator Provenance ⚠️ Single-actor pipeline (informational, §2.7).
2.8 Adversarial Surface No untrusted string reaches an exec/eval/HTML sink.
2.9 Documentation Coverage ✅ (repaired) Added README observability note (MEMORY.md / AGENTS.md already covered it).

Primary Finding (🔴 → fixed)

Memory lifecycle events were delivered while internal locks were heldEpisodeStore.notify under e.mu (writeLocked / Prune / Promote), and fact events under the shared per-dir facts lock. A notifier runs arbitrary caller code (a WebSocket send under odek serve, or a handler that re-enters the store), so firing under the lock serialized writes behind the sink and risked a reentrancy deadlock.

Repair (behavior-preserving): collect events while locked, fan out after unlocking — writeLocked now returns []MemoryEvent; WriteWithProvenance / Prune / Promote fire via notifyAll post-unlock; manager fact methods use fireAfterUnlock.

Repairs Applied (all safe/additive per §7.4)

# Axis Repair Auto-applied
1 Structural / Behavioral Fire lifecycle events outside locks
2 Semantic fact_merged test + system-prompt regression guard
3 Documentation README observability note

Repair Gate (§7.5)

  • ✅ Tests pass — full go test ./... green
  • go test -race clean (memory, loop)
  • go vet + gofmt clean
  • ✅ Repairs reversible (own commit 5993a8c)

Verdict

HumanReviewRecommended — functionally green and hardened, but provenance is single-actor (no independent C/D family), so a human should give it the final review.

Rationale: binding gate is §2.7 generator provenance (single-actor pipeline); no axis at 🔴 after repairs.


Generated by Claude Code

@jkyberneees jkyberneees merged commit d1ba73e into main Jun 6, 2026
6 checks passed
@jkyberneees jkyberneees deleted the claude/agent-signal-observability-UizE4 branch June 6, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants