Skip to content

fix: make E2E test timeouts configurable and add sensitive directory test#206

Merged
jeremyeder merged 2 commits into
mainfrom
fix/test-reliability
Dec 10, 2025
Merged

fix: make E2E test timeouts configurable and add sensitive directory test#206
jeremyeder merged 2 commits into
mainfrom
fix/test-reliability

Conversation

@jeremyeder

Copy link
Copy Markdown
Contributor

Summary

  • Make E2E test timeouts configurable to prevent flaky failures on slower CI runners
  • Add E2E test for sensitive directory blocking (critical security feature)

Changes

  • Add DEFAULT_TIMEOUT environment variable (default: 90s)
  • Replace all hardcoded timeout=60 with DEFAULT_TIMEOUT
  • Add new TestCriticalSecurityFeatures class
  • Add test_assess_blocks_sensitive_directories() E2E test

Benefits

  • Prevents flaky test failures on slower CI runners
  • Allows developers to adjust timeout based on environment
  • Adds end-to-end test coverage for critical security feature
  • Validates sensitive directory prompts work correctly

Configuration

Set AGENTREADY_TEST_TIMEOUT environment variable to override default:

export AGENTREADY_TEST_TIMEOUT=120  # For slower systems
pytest tests/e2e/

Test Plan

  • All linters pass (black, isort, ruff)
  • Pre-commit hooks pass
  • Conventional commit format verified
  • New E2E test for sensitive directories passes (requires /etc directory)

Fixes #192

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 4.5 noreply@anthropic.com

jeremyeder and others added 2 commits December 10, 2025 17:53
Add max_retries parameter to prevent infinite retry loops when API rate limits are hit.

Changes:
- Add max_retries parameter (default: 3) to LLMEnricher.enrich_skill()
- Add retry counter tracking with _retry_count internal parameter
- Implement jitter to prevent thundering herd problem
- Add graceful fallback to heuristic skill on max retries exceeded
- Add --llm-max-retries CLI option (range: 0-10, default: 3)
- Thread parameter through learning_service.py workflow

Security improvements:
- Bounded retries prevent runaway API costs
- Helpful error messages with API quota link
- Graceful degradation when retries exhausted

Fixes #104

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…test

Improve test reliability and security coverage.

Changes:
- Add DEFAULT_TIMEOUT environment variable (default: 90s)
- Replace all hardcoded timeout=60 with DEFAULT_TIMEOUT
- Add new E2E test for sensitive directory blocking

Benefits:
- Prevents flaky test failures on slower CI runners
- Allows developers to adjust timeout based on environment
- Adds end-to-end test coverage for critical security feature
- Validates sensitive directory prompts work correctly

Configuration:
- Set AGENTREADY_TEST_TIMEOUT environment variable to override default
- Example: export AGENTREADY_TEST_TIMEOUT=120 (for slower systems)

Fixes #192

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@jeremyeder jeremyeder merged commit 27e87e5 into main Dec 10, 2025
@jeremyeder jeremyeder deleted the fix/test-reliability branch December 10, 2025 22:55
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 2.19.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test Reliability: Configurable timeouts and sensitive directory E2E test

1 participant