Skip to content

test: make random component execution safer#10169

Merged
paulbalandan merged 6 commits into
codeigniter4:developfrom
memleakd:refactor/random-tests-commands
May 11, 2026
Merged

test: make random component execution safer#10169
paulbalandan merged 6 commits into
codeigniter4:developfrom
memleakd:refactor/random-tests-commands

Conversation

@memleakd
Copy link
Copy Markdown
Contributor

@memleakd memleakd commented May 6, 2026

Description

Related to #9968.

This one took a few rounds: every time the Commands suite looked settled, another bit of shared test state quietly raised its hand. The result is a more random-order-safe Commands component, plus a few small shared-state hardening fixes that the parallel random component runs exposed along the way.

This enables Commands in the random test config and updates the affected tests so they own their setup/cleanup instead of depending on earlier tests running first.

Main changes:

  • reset scaffold test CLI state and make generated command/scaffold cleanup reliable between tests
  • isolate migration command tests from generated App migrations and leftover tables
  • make database command tests seed and select their own expected state
  • close/reset shared database connections before deleting SQLite test databases
  • restore locale/config state in translation command tests
  • reset mocked cache services after cache command tests
  • make the Unsuffixable support command provide its own template data
  • scope CreateNewChangelogTest cleanup in AutoReview so it no longer runs repository-wide git restore / git clean while random component jobs are running in parallel

This PR also touches a few areas outside tests/system/Commands because the failures were not purely local to one test class:

  • system/Commands/Database/ShowTableInfo.php now keeps db:table --show display IDs sequential even when table arrays have sparse keys
  • system/HotReloader/DirectoryHasher.php now hashes files in a stable order, which avoids filesystem traversal order differences
  • system/Cache/FactoriesCache/FileVarExportHandler.php now guards cache writes against concurrent cleanup/recreation races
  • tests/system/HotReloader/DirectoryHasherTest.php now uses isolated writable fixtures instead of shared app/system directories
  • system/Log/Handlers/FileHandler.php now treats the final chmod after creating a log file as best-effort, so concurrent log cleanup cannot turn logging into a test/app error

Verified locally:

  • repeated random-order Commands runs, including failed CI seeds
  • HotReloader, Cookie random-order runs
  • composer phpstan:check
  • vendor/bin/rector process --dry-run
  • composer cs

Checklist:

  • Securely signed commits
  • Component(s) with PHPDoc blocks, only if necessary or adds value (without duplication)
  • Unit testing, with >80% coverage
  • User guide updated
  • Conforms to style guide

@memleakd memleakd force-pushed the refactor/random-tests-commands branch 14 times, most recently from 0f09214 to 39a32be Compare May 8, 2026 08:36
@github-actions github-actions Bot added the github_actions Pull requests that update Github_actions code label May 8, 2026
@memleakd memleakd force-pushed the refactor/random-tests-commands branch from 39a32be to c325e39 Compare May 8, 2026 08:38
@github-actions github-actions Bot removed the github_actions Pull requests that update Github_actions code label May 8, 2026
@memleakd memleakd force-pushed the refactor/random-tests-commands branch 2 times, most recently from d2c7fda to 50c93a1 Compare May 8, 2026 09:56
@memleakd memleakd force-pushed the refactor/random-tests-commands branch from 50c93a1 to 811476f Compare May 8, 2026 10:07
@memleakd memleakd changed the title test: make Commands tests random-order safe test: make random component execution safer May 8, 2026
Comment thread system/Cache/FactoriesCache/FileVarExportHandler.php
Comment thread system/Cache/FactoriesCache/FileVarExportHandler.php
Comment thread system/Commands/Database/ShowTableInfo.php Outdated
Comment thread tests/system/Commands/Generators/ScaffoldGeneratorTest.php
memleakd added 2 commits May 9, 2026 08:46
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
Copy link
Copy Markdown
Member

@paulbalandan paulbalandan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking at the random order testing! Few comments:

Comment thread system/Database/BaseConnection.php Outdated
Comment thread tests/system/Commands/Database/MigrateStatusTest.php
Comment thread tests/system/Commands/MigrationIntegrationTest.php Outdated
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
Copy link
Copy Markdown
Member

@michalsn michalsn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please replace the silent returns with log_message() calls. We can't just return and say nothing bad happened here - we have to at least log an error so this leaves some trace for developers.

Comment thread system/Cache/FactoriesCache/FileVarExportHandler.php
Comment thread system/Cache/FactoriesCache/FileVarExportHandler.php
Comment thread system/Cache/FactoriesCache/FileVarExportHandler.php
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
Comment thread system/Log/Handlers/FileHandler.php
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
Copy link
Copy Markdown
Member

@michalsn michalsn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@paulbalandan paulbalandan merged commit 83d509c into codeigniter4:develop May 11, 2026
55 checks passed
@memleakd memleakd deleted the refactor/random-tests-commands branch May 11, 2026 17:35
@memleakd
Copy link
Copy Markdown
Contributor Author

Thank you @neznaika0 @paulbalandan @michalsn

paulbalandan added a commit that referenced this pull request May 22, 2026
* docs: add changelog and upgrade for v4.7.3 (#10068)

* chore: migrate SCSS from deprecated `@import` usage (#10066)

* docs: clarify `Model::find()` note for null argument (#10072)

* chore: upload as artifacts the debug files of failing random execution tests (#10074)

* test: indicate components that already pass random execution tests (#10073)

* chore: fix wrong trigger name for manually runnable workflow (#10077)

* chore: upgrade to `gvenzl/oracle-free` (#10075)

* docs: fix formatting in Time library guide (#10078)

* docs: update 014.php (#10083)

* chore: resolve PHPStan nullCoalesce and isset errors on Config properties (#10081)

* chore: resolve PHPStan nullCoalesce and isset errors on Config properties

* fix tests

* fix: make Autoloader composer path injectable to fix parallel test race condition (#10082)

* chore(deps-dev): update rector/rector requirement

Updates the requirements on [rector/rector](https://github.com/rectorphp/rector) to permit the latest version.

Updates `rector/rector` to 2.4.0
- [Release notes](https://github.com/rectorphp/rector/releases)
- [Commits](rectorphp/rector@2.3.9...2.4.0)

---
updated-dependencies:
- dependency-name: rector/rector
  dependency-version: 2.4.0
  dependency-type: direct:development
  dependency-group: composer-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore(deps-dev): update rector/rector requirement

Updates the requirements on [rector/rector](https://github.com/rectorphp/rector) to permit the latest version.

Updates `rector/rector` to 2.4.1
- [Release notes](https://github.com/rectorphp/rector/releases)
- [Commits](rectorphp/rector@2.4.0...2.4.1)

---
updated-dependencies:
- dependency-name: rector/rector
  dependency-version: 2.4.1
  dependency-type: direct:development
  dependency-group: composer-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: remove useless @var

* refactor: add full testing to `logs:clear` command (#10090)

* fix: store SPL closures in register() so unregister() can remove them (#10097)

* refactor: add full testing for `debugbar:clear` command (#10093)

* refactor: pass `--do-not-cache-result` to prevent shared cache corruption (#10098)

Co-authored-by: John Paul E Balandan <paulbalandan@gmail.com>

* refactor: add full testing for `cache:clear` command (#10094)

* chore: re-comment transiently failing component tests (#10095)

* test: group commands tests similar to `system/Commands/` (#10096)

* chore(deps): bump actions/github-script in / (#10100)

Bumps [actions/github-script](https://github.com/actions/github-script) in `/` from 8.0.0 to 9.0.0.


Updates `actions/github-script` from 8.0.0 to 9.0.0
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](actions/github-script@ed59741...3a2844b)

---
updated-dependencies:
- dependency-name: actions/github-script
  dependency-version: 9.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github_actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: ensure output buffer is closed after use of `command()` (#10099)

* chore: fix labeler workflow (#10104)

* chore: fix labeler workflow

* revert now to use pull_request_target

* chore: refactor phpunit config file (#10102)

* chore: fixes for php-cs-fixer and psalm (#10105)

* fix: preserve null values in Validation::getValidated() (#10101)

* test: refactor tests on `BaseCommand` and `Commands` (#10103)

* fix: Rename phpunit.xml.dist (#10111)

* fix: refactor inconsistent behavior on `CLI::write()` and `CLI::error()` (#10106)

* test: fix command tests that may hang on linux due to sudo (#10107)

* refactor: rename `-h` option of `routes` command as `--handler` (#10113)

* fix: ensure calling `env` command with options only would not throw (#10114)

* docs: Improve guide (#10109)

* docs: Update "Managing your Applications"

* docs: Update "Composer Installation"

* docs: Update "Worker Mode"

* docs: Update "Testing"

* fix: Move next line

* refactor: start only required services (#10115)

* chore(deps): bump actions/upload-artifact in / (#10116)

Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) in `/` from 7.0.0 to 7.0.1.


Updates `actions/upload-artifact` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@bbbca2d...043fb46)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github_actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump actions/cache in / (#10117)

Bumps [actions/cache](https://github.com/actions/cache) in `/` from 5.0.4 to 5.0.5.


Updates `actions/cache` from 5.0.4 to 5.0.5
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](actions/cache@6682284...27d5ce7)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: 5.0.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github_actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: fix label-pr verification step (#10118)

* chore: fix label-pr verification step

* revert to pull request target

* chore(deps-dev): update rector/rector requirement

Updates the requirements on [rector/rector](https://github.com/rectorphp/rector) to permit the latest version.

Updates `rector/rector` to 2.4.2
- [Release notes](https://github.com/rectorphp/rector/releases)
- [Commits](rectorphp/rector@2.4.1...2.4.2)

---
updated-dependencies:
- dependency-name: rector/rector
  dependency-version: 2.4.2
  dependency-type: direct:development
  dependency-group: composer-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: fix transient random test failures (#10122)

* chore(deps): bump actions/setup-node in / (#10123)

Bumps [actions/setup-node](https://github.com/actions/setup-node) in `/` from 6.3.0 to 6.4.0.


Updates `actions/setup-node` from 6.3.0 to 6.4.0
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](actions/setup-node@53b8394...48b55a0)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-version: 6.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github_actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: suppress stty stderr leak in `CLI::generateDimensions()` when stdin is not a TTY (#10124)

* refactor: further rename `--handler` to `--sort-by-handler` for `routes` (#10125)

* test: optimize AutoReview tests (#10127)

* refactor: UX: `ClearLogs::execute()` error message is misleading after interactive `'n'` (#10126)

* docs: document Axios header configuration for AJAX (#10069)

Added Axios information regarding the X-Requested-With header.

* docs: refactor AJAX request and clarify framework examples (#10129)

* docs: fix indentation on `4.7.2` and `4.7.3` changelogs (#10131)

* docs: add version switcher to docs page (#10135)

* fix: reset Kint CSP state in worker mode (#10139)

Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>

* refactor: simplify `FileLocator::listFiles()` (#10142)

* fix: make Time::createFromTimestamp locale-independent (#10151)

* chore(deps): bump actions/labeler in / (#10161)

Bumps [actions/labeler](https://github.com/actions/labeler) in `/` from 6.0.1 to 6.1.0.


Updates `actions/labeler` from 6.0.1 to 6.1.0
- [Release notes](https://github.com/actions/labeler/releases)
- [Commits](actions/labeler@634933e...f27b608)

---
updated-dependencies:
- dependency-name: actions/labeler
  dependency-version: 6.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github_actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: SQLSRV driver's `decrement()` method (#10155)

Co-authored-by: Michal Sniatala <michal@sniatala.pl>
Co-authored-by: John Paul E. Balandan, CPA <paulbalandan@gmail.com>

* fix: suppress tput stderr leak when TERM is not present (#10167)

* fix: support third-party loggers in toolbar logs collector (#10173)

* fix: PostgreSQL Builder's `increment()` and `decrement()` methods not working for numeric columns (#10172)

* test: fix random-order failures in Config, Honeypot, and Test (#10168)

* chore: use single class per file when possible on tests/ directory

* chore: add return array iterable doc and regenerate baseline

* refactor: reduce PHPStan child return type baseline (#10165)

Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>

* refactor: remove PHPStan callable signature baseline (#10166)

Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>

* fix: preserve cached table list shape (#10179)

* fix: preserve cached table list shape

* docs: add changelog entry for cached table list fix

Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>

---------

Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>

* fix: harden regex matching on `key:generate` command (#10183)

* chore: apply `withRootFiles()` on rector config (#10188)

* chore: apply withRootFiles() on rector config

* chore: run cs fix

* test: make random component execution safer (#10169)

* test: make random component execution safer

* test: remove unnecessary normalization

Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>

* fix: stabilize cached table names for random tests

Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>

* test: address review feedbacks

Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>

* fix: log factories cache write failures

Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>

* fix: document best-effort log chmod

Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>

---------

Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>

* chore: add Structarmed to QA (#10180)

* chore: Add Structarmed to QA

* chore: use latest structarmed 0.3.1

* chore: bump structarmed to 0.3.2

* bump structarmed to 0.3.3 to properly fix cache on CI

* chore: bump to structarmed 0.3.4 to fix very long list progressbar

* chore: bump to structarmed 0.3.5

* chore: bump structarmed to 0.4.0

* chore: remove ignore platform php 8.5 on test-structarmed workflow

* chore: use php 8.5 in php-versions

* use tools: composer under Setup PHP

* docs: fix Bitnami link (#10190)

* fix: restore deep dot-notation traversal in `Language::getLine()` (#10189)

* fix: make frankenphp-worker.php template idempotent on watcher restart (#10191)

* chore: bump structarmed to 0.4.5

* chore: skip system/ThirdParty

* fix: `Entity::normalizeValue()` must handle `UnitEnum` before `toArray()` (#10137)

* chore: remove checkout step for base branch (#10194)

* chore(deps-dev): update rector/rector requirement

Updates the requirements on [rector/rector](https://github.com/rectorphp/rector) to permit the latest version.

Updates `rector/rector` to 2.4.3
- [Release notes](https://github.com/rectorphp/rector/releases)
- [Commits](rectorphp/rector@2.4.2...2.4.3)

---
updated-dependencies:
- dependency-name: rector/rector
  dependency-version: 2.4.3
  dependency-type: direct:development
  dependency-group: composer-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore(deps): bump shivammathur/setup-php in / (#10201)

Bumps [shivammathur/setup-php](https://github.com/shivammathur/setup-php) in `/` from 2.37.0 to 2.37.1.


Updates `shivammathur/setup-php` from 2.37.0 to 2.37.1
- [Release notes](https://github.com/shivammathur/setup-php/releases)
- [Commits](shivammathur/setup-php@accd612...7c071df)

---
updated-dependencies:
- dependency-name: shivammathur/setup-php
  dependency-version: 2.37.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github_actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Merge pull request #10198 from samsonasik/migrate-deptrac-to-structarmed

chore: migrate from `deptrac` to `structarmed`

* chore(deps-dev): update boundwize/structarmed requirement (#10202)

Updates the requirements on [boundwize/structarmed](https://github.com/boundwize/structarmed) to permit the latest version.

Updates `boundwize/structarmed` to 0.5.5
- [Release notes](https://github.com/boundwize/structarmed/releases)
- [Commits](boundwize/structarmed@0.5.4...0.5.5)

---
updated-dependencies:
- dependency-name: boundwize/structarmed
  dependency-version: 0.5.5
  dependency-type: direct:development
  dependency-group: composer-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix(config): recognize disabled zlib compression values (#10193)

* fix: escape `--host` option in `serve` command (#10203)

* chore: bump phpstan to ^2.1.55 and fix callable docblock notice (#10219)

* chore(deps-dev): update boundwize/structarmed requirement (#10218)

Updates the requirements on [boundwize/structarmed](https://github.com/boundwize/structarmed) to permit the latest version.

Updates `boundwize/structarmed` to 0.6.8
- [Release notes](https://github.com/boundwize/structarmed/releases)
- [Commits](boundwize/structarmed@0.5.5...0.6.8)

---
updated-dependencies:
- dependency-name: boundwize/structarmed
  dependency-version: 0.6.8
  dependency-type: direct:development
  dependency-group: composer-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Merge commit from fork

* fix: validate client extension in ext_in upload rule

* add changelog and upgrade notes

* chore(deps-dev): bump the composer-dependencies group with 2 updates (#10224)

Updates the requirements on [boundwize/structarmed](https://github.com/boundwize/structarmed) and [rector/rector](https://github.com/rectorphp/rector) to permit the latest version.

Updates `boundwize/structarmed` to 0.6.15
- [Release notes](https://github.com/boundwize/structarmed/releases)
- [Commits](boundwize/structarmed@0.6.8...0.6.15)

Updates `rector/rector` to 2.4.4
- [Release notes](https://github.com/rectorphp/rector/releases)
- [Commits](rectorphp/rector@2.4.3...2.4.4)

---
updated-dependencies:
- dependency-name: boundwize/structarmed
  dependency-version: 0.6.15
  dependency-type: direct:development
  dependency-group: composer-dependencies
- dependency-name: rector/rector
  dependency-version: 2.4.4
  dependency-type: direct:development
  dependency-group: composer-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Prep for 4.7.3 release (#10227)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
Co-authored-by: Michal Sniatala <michal@sniatala.pl>
Co-authored-by: Toto <totoprayogo1916@gmail.com>
Co-authored-by: Robson Jonathas <68930311+robsonjonathas@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Abdul Malik Ikhsan <samsonasik@gmail.com>
Co-authored-by: neznaika0 <ozornick.ks@gmail.com>
Co-authored-by: Asad <asadkhan4230@gmail.com>
Co-authored-by: memleakd <121398829+memleakd@users.noreply.github.com>
Co-authored-by: Vansh Patel <developer.patelvansh@gmail.com>
Co-authored-by: maniaba <61078470+maniaba@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants