[bug] Unable to use program

[pickuperast]

"unexpected status 503 Service Unavailable: All managed Codex accounts are temporarily unavailable for this runtime request., url: http://127.0.0.1:60333/responses"

doctor passed 17/17
have 2 available accounts

[coderabbitai]

Coding Plan

Summary

• add instance method clearAccountTransientState() to AccountManager that wipes per-account cooldowns and rate-limit state
• relax the recoverStaleRuntimeState guard at lib/runtime-rotation-proxy.ts:984-989 to allow recovery when accounts have transient skip reasons like "rate-limited" or "cooling-down*"
• call the new clearing method after loadFromDisk() in the recovery path in lib/runtime/rotation-proxy-state.ts so reloaded accounts start fresh
• the combination breaks the deadlock where transient state both prevents account selection and blocks the recovery that would clear it

Design Choices

Design Choice 1: should clearing transient state be opt-in or always happen during recovery?

Options Considered:

1. add a flag to loadFromDisk({ clearTransientState: true }) to conditionally clear
2. always clear transient state in loadFromDisk
3. add a separate instance method called explicitly in recovery path

Chosen Option: 3

Rationale: a separate instance method keeps the recovery intent explicit and avoids changing loadFromDisk behavior for other callers.

Design Choice 2: which skip reasons should still block recovery?

Options Considered:

1. remove all skip reason checks from the guard
2. only keep "policy-blocked" as a blocking reason (policy won't change with reload)
3. keep "rate-limited", "cooling-down*", and "policy-blocked" as blocking

Chosen Option: 2

Rationale: "rate-limited" and "cooling-down*" are transient states that recovery is designed to escape. "policy-blocked" is external and won't change with reload.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: extend accountmanager to clear per-account transient state

add an instance method that resets cooldown timestamps and rate-limit entries on all managed accounts, enabling clean recovery from stuck states.

Task 1: add clearAccountTransientState instance method

clears cooldown and rate-limit state on all accounts held by the manager instance.

• add clearAccountTransientState() method to AccountManager class in lib/accounts.ts after line 669
• iterate over all accounts and set coolingDownUntil = 0 and cooldownReason = undefined
• call clearAccountRateLimits(account) helper (create if needed) to wipe rateLimitResetTimes map
• call saveToDiskDebounced() to persist the cleared state so subsequent reloads don't restore stale cooldowns
• no-op if no accounts loaded (defensive check)

Task 2: add clearAccountRateLimits helper

extracts rate-limit clearing logic for reuse.

• add helper function near clearExpiredRateLimits in lib/accounts.ts
• clears the rateLimitResetTimes (or equivalent rateLimits) map on a single ManagedAccount
• reuses existing pattern from clearExpiredRateLimits but removes all entries instead of just expired ones

🤖 Prompt for AI agents

this phase adds per-account transient state clearing to `AccountManager` in
`lib/accounts.ts`. the goal is to expose a clean recovery hook that wipes
cooldown and rate-limit state on all managed accounts.

- add a `clearAccountRateLimits(account: ManagedAccount)` helper function near
`clearExpiredRateLimits` in `lib/accounts.ts`
- it should clear the entire `rateLimitResetTimes` (or equivalent `rateLimits`)
map on the given account
- mirror the pattern from `clearExpiredRateLimits` but remove ALL entries, not
just expired ones
- add `clearAccountTransientState()` as an instance method on `AccountManager`
in `lib/accounts.ts` after line 669
  - add a defensive no-op guard if no accounts are loaded
  - iterate over all managed accounts
  - set `coolingDownUntil = 0` and `cooldownReason = undefined` on each account
  - call `clearAccountRateLimits(account)` for each account
- call `saveToDiskDebounced()` after iterating so the cleared state is persisted
and won't be restored on next reload
- flag: watch for concurrency risks if accounts array is mutated during
iteration

Phase 2: fix the stale recovery path

relax the guard condition and ensure recovery produces a clean slate.

Task 1: relax the recovery guard condition

allow recovery to fire when accounts have transient skip reasons.

• modify condition at lib/runtime-rotation-proxy.ts:984-989
• remove reason === "rate-limited" from the .some() check
• remove reason.startsWith("cooling-down") from the check
• keep reason === "policy-blocked" as blocking (policy is external, won't change with reload)
• this allows recovery to attempt when accounts are stuck in transient cooldown or rate-limit states

Task 2: call clearAccountTransientState after reload

ensures the reloaded manager starts with clean transient state.

• in lib/runtime/rotation-proxy-state.ts around line 102, after AccountManager.loadFromDisk() completes
• call reloadedManager.clearAccountTransientState() before assigning to state.activeAccountManager
• this happens after the existing AccountManager.resetVolatileRuntimeState() call which handles global singletons
• together these ensure circuit breakers, trackers, cooldowns, and rate-limits are all cleared on recovery

Task 3: add regression tests for recovery path

ensures the fix works and prevents future regressions.

• add test in test/runtime-rotation-proxy.test.ts or appropriate test file
• test case 1: verify clearAccountTransientState() clears cooldown timestamps on accounts
• test case 2: verify clearAccountTransientState() clears rate-limit maps on accounts
• test case 3: verify recovery fires when all accounts have "cooling-down*" skip reasons
• test case 4: verify recovery does NOT fire when accounts have "policy-blocked" skip reason
• test case 5: verify that after recovery, accounts are selectable (no longer skipped)
• flag: check windows compat for timestamp comparisons in cooldown logic

🤖 Prompt for AI agents

this phase fixes the stale recovery path in `lib/runtime-rotation-proxy.ts` and
`lib/runtime/rotation-proxy-state.ts` so transient account states no longer
deadlock recovery, and adds regression coverage.

**task 1 — relax guard at `lib/runtime-rotation-proxy.ts:984-989`:**
- locate the `.some()` check in condition 6 of the recovery guard
- remove the `reason === "rate-limited"` predicate from the check
- remove the `reason.startsWith("cooling-down")` predicate from the check
- retain `reason === "policy-blocked"` as the sole blocking reason — policy is
external and won't change with a reload

**task 2 — call `clearAccountTransientState()` in
`lib/runtime/rotation-proxy-state.ts` around line 102:**
- after `AccountManager.loadFromDisk()` completes, call
`reloadedManager.clearAccountTransientState()`
- this must happen before assigning `reloadedManager` to
`state.activeAccountManager`
- this call comes after the existing
`AccountManager.resetVolatileRuntimeState()` call so global singletons (circuit
breakers, trackers) are handled first, then per-account cooldowns and
rate-limits are cleared
- flag: concurrency risk — ensure no concurrent request can pick up the manager
between `loadFromDisk` and `clearAccountTransientState` completing

**task 3 — regression tests in `test/runtime-rotation-proxy.test.ts`:**
- test case 1: `clearAccountTransientState()` sets `coolingDownUntil = 0` and
`cooldownReason = undefined` on all accounts
- test case 2: `clearAccountTransientState()` empties the `rateLimitResetTimes`
map on all accounts
- test case 3: recovery fires when all accounts carry `"cooling-down*"` skip
reasons
- test case 4: recovery does NOT fire when accounts carry `"policy-blocked"`
skip reason
- test case 5: accounts are selectable (no longer skipped) after recovery
completes
- flag: check windows compat for timestamp comparisons in cooldown logic —
`Date.now()` vs `performance.now()` behavior may differ
- flag: missing regression tests for concurrent reload scenarios would be a gap
here

Research

the rotation proxy in lib/runtime-rotation-proxy.ts maintains per-request account selection with skip reasons accumulated in accountSkipReasons. the stale recovery mechanism in lib/runtime/rotation-proxy-state.ts reloads accounts from disk when selection fails, but condition 6 at lib/runtime-rotation-proxy.ts:984-989 blocks recovery when any account has "rate-limited" or "cooling-down*" skip reasons. AccountManager.resetVolatileRuntimeState() at lib/accounts.ts:666-669 only clears global singletons (trackers, circuit breakers) but not per-account cooldown timestamps or rate-limit maps stored on ManagedAccount objects. this creates a deadlock: accounts can't be used due to transient state, and recovery can't fire because of that same state.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

this phase adds per-account transient state clearing to `AccountManager` in
`lib/accounts.ts`. the goal is to expose a clean recovery hook that wipes
cooldown and rate-limit state on all managed accounts.

- add a `clearAccountRateLimits(account: ManagedAccount)` helper function near
`clearExpiredRateLimits` in `lib/accounts.ts`
- it should clear the entire `rateLimitResetTimes` (or equivalent `rateLimits`)
map on the given account
- mirror the pattern from `clearExpiredRateLimits` but remove ALL entries, not
just expired ones
- add `clearAccountTransientState()` as an instance method on `AccountManager`
in `lib/accounts.ts` after line 669
  - add a defensive no-op guard if no accounts are loaded
  - iterate over all managed accounts
  - set `coolingDownUntil = 0` and `cooldownReason = undefined` on each account
  - call `clearAccountRateLimits(account)` for each account
- call `saveToDiskDebounced()` after iterating so the cleared state is persisted
and won't be restored on next reload
- flag: watch for concurrency risks if accounts array is mutated during
iteration
===============================================================================

Task: 2

this phase fixes the stale recovery path in `lib/runtime-rotation-proxy.ts` and
`lib/runtime/rotation-proxy-state.ts` so transient account states no longer
deadlock recovery, and adds regression coverage.

**task 1 — relax guard at `lib/runtime-rotation-proxy.ts:984-989`:**
- locate the `.some()` check in condition 6 of the recovery guard
- remove the `reason === "rate-limited"` predicate from the check
- remove the `reason.startsWith("cooling-down")` predicate from the check
- retain `reason === "policy-blocked"` as the sole blocking reason — policy is
external and won't change with a reload

**task 2 — call `clearAccountTransientState()` in
`lib/runtime/rotation-proxy-state.ts` around line 102:**
- after `AccountManager.loadFromDisk()` completes, call
`reloadedManager.clearAccountTransientState()`
- this must happen before assigning `reloadedManager` to
`state.activeAccountManager`
- this call comes after the existing
`AccountManager.resetVolatileRuntimeState()` call so global singletons (circuit
breakers, trackers) are handled first, then per-account cooldowns and
rate-limits are cleared
- flag: concurrency risk — ensure no concurrent request can pick up the manager
between `loadFromDisk` and `clearAccountTransientState` completing

**task 3 — regression tests in `test/runtime-rotation-proxy.test.ts`:**
- test case 1: `clearAccountTransientState()` sets `coolingDownUntil = 0` and
`cooldownReason = undefined` on all accounts
- test case 2: `clearAccountTransientState()` empties the `rateLimitResetTimes`
map on all accounts
- test case 3: recovery fires when all accounts carry `"cooling-down*"` skip
reasons
- test case 4: recovery does NOT fire when accounts carry `"policy-blocked"`
skip reason
- test case 5: accounts are selectable (no longer skipped) after recovery
completes
- flag: check windows compat for timestamp comparisons in cooldown logic —
`Date.now()` vs `performance.now()` behavior may differ
- flag: missing regression tests for concurrent reload scenarios would be a gap
here

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• [bug] Accounts look healthy but codex throws 503 #479

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!