Release 2.4

.github/workflows/main.yml → .github/workflows/check.yml

@@ -14,3 +14,4 @@ jobs:
         tests: "./test/**/*_test.js"
         token: ${{ secrets.GITHUB_TOKEN }}
         has-tests-label: true
+        comment-on-empty: true

.gitignore

@@ -19,3 +19,4 @@ testpullfilecache*
 package-lock.json
 yarn.lock
 /.vs
+typings/types.d.ts

CHANGELOG.md

@@ -1,3 +1,27 @@
+## 2.4.0
+
+* Improved setup wizard with `npx codecept init`:
+  * **enabled [retryFailedStep](/plugins/#retryfailedstep) plugin for new setups**.
+  * enabled [@codeceptjs/configure](/configuration/#common-configuration-patterns) to toggle headless/window mode via env variable
+  * creates a new test on init
+  * removed question on "steps file", create it by default.
+* Added [pauseOnFail plugin](/plugins/#pauseonfail). *Sponsored by Paul Vincent Beigang and his book "[Practical End 2 End Testing with CodeceptJS](https://leanpub.com/codeceptjs/)"*.
+* Added [`run-rerun` command](/commands/#run-rerun) to run tests multiple times to detect and fix flaky tests. By @Ilrilan and @Vorobeyko.
+* Added [`Scenario.todo()` to declare tests as pending](/basics#todotest). See #2100 by @Vorobeyko
+* Added support for absolute path for `output` dir. See #2049 by @elukoyanov
+* Fixed error in `npx codecept init` caused by calling `console.print`. See #2071 by @Atinux.
+* [Filesystem] Methods added by @aefluke:
+  * `seeFileNameMatching`
+  * `grabFileNames`
+* [Puppeteer] Fixed grabbing attributes with hyphen by @Holorium
+* [TestCafe] Fixed `grabAttributeFrom` method by @elukoyanov
+* [MockRequest] Added support for [Polly config options](https://netflix.github.io/pollyjs/#/configuration?id=configuration) by @ecrmnn
+* [TestCafe] Fixes exiting with zero code on failure. Fixed #2090 with #2106 by @koushikmohan1996
+* [WebDriver][Puppeteer] Added basicAuth support via config. Example: `basicAuth: {username: 'username', password: 'password'}`. See #1962 by @PeterNgTr
+* [WebDriver][Appium] Added `scrollIntoView` by @pablopaul
+* Fixed #2118: No error stack trace for syntax error by @senthillkumar
+* Added `parse()` method to data table inside Cucumber tests. Use it to obtain rows and hashes for test data. See #2082 by @Sraime
+
 ## 2.3.6
 
 * Create better Typescript definition file through JSDoc. By @lemnis
@@ -8,7 +32,7 @@ exports.config = {
   tests: '{./workers/base_test.workers.js,./workers/test_grep.workers.js}',
 }
 ```
-* Added new command `npx codeceptjs info` which print information about your environment and CodeceptJS configs. By @jamesgeorge007 
+* Added new command `npx codeceptjs info` which print information about your environment and CodeceptJS configs. By @jamesgeorge007
 * Fixed some typos in documantation. By @pablopaul @atomicpages @EricTendian
 * Added PULL_REQUEST template.
 * [Puppeteer][WebDriver] Added `waitForClickable` for waiting clickable element on page.

docs/advanced.md

@@ -228,3 +228,45 @@ Please note that some config changes can't be applied on the fly. For instance,
 
 Configuration changes will be reverted after a test or a suite.
 
+
+### Rerunning Flaky Tests Multiple Times <Badge text="Since 2.4" type="warning"/>
+
+End to end tests can be flaky for various reasons. Even when we can't do anything to solve this problem it we can do next two things:
+
+* Detect flaky tests in our suite
+* Fix flaky tests by rerunning them.
+
+Both tasks can be achieved with [`run-rerun` command](/commands/#run-rerun) which runs tests multiple times until all tests are passed.
+
+You should set min and max runs boundaries so when few tests fail in a row you can rerun them until they are succeeded.
+
+```js
+// inside to codecept.conf.js
+exports.config = { // ...
+  rerun: {
+    // run 4 times until 1st success
+    minSuccess: 1,
+    maxReruns: 4,
+  }
+}
+```
+
+If you want to check all your tests for stability you can set high boundaries for minimal success:
+
+```js
+// inside to codecept.conf.js
+exports.config = { // ...
+  rerun: {
+    // run all tests must pass exactly 5 times
+    minSuccess: 5,
+    maxReruns: 5,
+  }
+}
+```
+
+Now execute tests with `run-rerun` command:
+
+```
+npx codeceptjs run-rerun
+```
+

docs/basics.md

@@ -450,7 +450,7 @@ The interactive shell can be started outside of test context by running
 npx codeceptjs shell
 ```
 
-### Pause on Failure
+### Pause on Failure <Badge text="Since 2.4" type="warning"/>
 
 To start interactive pause automatically for a failing test you can run tests with [pauseOnFail Plugin](/plugins/#pauseonfail).
 When a test fails, the pause mode will be activated, so you can inspect current browser session before it is closed.
@@ -465,14 +465,24 @@ This is an **essential feature to debug flaky tests**, as you can analyze them i
 By default CodeceptJS saves a screenshot of a failed test.
 This can be configured in [screenshotOnFail Plugin](/plugins/#screenshotonfail)
 
+> **screenshotOnFail plugin is enabled by default** for new setups
+
 ### Step By Step Report
 
 To see how the test was executed, use [stepByStepReport Plugin](/plugins/#stepbystepreport). It saves a screenshot of each passed step and shows them in a nice slideshow.
 
 ## Retries
 
+### Auto Retry
+
+You can auto-retry a failed step by enabling [retryFailedStep Plugin](/plugins/#retryfailedstep).
+
+> **autoRetry plugin is enabled by default** for new setups since CodeceptJS 2.4
+
 ### Retry Step
 
+Unless you use retryFailedStep plugin you can manually control retries in your project.
+
 If you have a step which often fails, you can retry execution for this single step.
 Use the `retry()` function before an action to ask CodeceptJS to retry it on failure:
 
@@ -505,9 +515,6 @@ I.retry({
 
 Pass a function to the `when` option to retry only when an error matches the expected one.
 
-### Auto Retry
-
-You can auto-retry a failed step by enabling [retryFailedStep Plugin](/plugins/#retryfailedstep).
 
 ### Retry Scenario
 
@@ -601,8 +608,11 @@ within('.js-signup-form', () => {
 I.see('There were problems creating your account.');
 ```
 
+> ⚠ `within` can cause problems when used incorrectly. If you see a weired behavior of a test try to refactor it to not use `within`. It is recommended to keep within for simplest cases when possible.
+
 `within` can also work with IFrames. A special `frame` locator is required to locate the iframe and get into its context.
 
+
 See example:
 
 ```js
@@ -611,6 +621,8 @@ within({frame: "#editor"}, () => {
 });
 ```
 
+> ℹ IFrames can also be accessed via `I.switchTo` command of a corresponding helper.
+
 Nested IFrames can be set by passing an array *(WebDriver, Nightmare & Puppeteer only)*:
 
 ```js
@@ -759,12 +771,14 @@ Like in Mocha you can use `x` and `only` to skip tests or to run a single test.
 * `xScenario` - skips current test
 * `Scenario.only` - executes only the current test
 
-## Todo test
+## Todo Test <Badge text="Since 2.4" type="warning"/>
+
+You can use `Scenario.todo` when you are planning on writing tests.
 
-You can use `Scenario.todo` when you are planning on writing tests. This test will be skipped like with usual `skip`.
-But this method to adds skip-message: `Test not implemented!` to test instance.
+This test will be skipped like with regular `Scenario.skip` but with additional message "Test not implemented!":
+
+Use it with a test body as a test plan:
 
-Example #1 - with callback:
 ```js
 Scenario.todo('Test',  I => {
 /**
@@ -774,19 +788,11 @@ Scenario.todo('Test',  I => {
  * Result:
  * 3. Field contains text
  */
-})
-```
-
-Example #2 - without callback:
-```js
-Scenario.todo('Test')
+});
 ```
 
-And you can access the `message` of `test.opts.skipInfo` in the events
+Or even without a test body:
 
-Example #3 - access into event hooks
 ```js
-  event.dispatcher.on(event.test.before, (test) => {
-    test.opts.skipInfo //  contains {message and description}
-  });
-```
+Scenario.todo('Test');
+```

docs/bdd.md

@@ -11,7 +11,7 @@ Behavior Driven Development (BDD) is a popular software development methodology.
 
 BDD was introduced by [Dan North](https://dannorth.net/introducing-bdd/). He described it as:
 
->  outside-in, pull-based, multiple-stakeholder, multiple-scale, high-automation, agile methodology. It describes a cycle of interactions with well-defined outputs, resulting in the delivery of working, tested software that matters.
+> outside-in, pull-based, multiple-stakeholder, multiple-scale, high-automation, agile methodology. It describes a cycle of interactions with well-defined outputs, resulting in the delivery of working, tested software that matters.
 
 BDD has its own evolution from the days it was born, started by replacing "test" to "should" in unit tests, and moving towards powerful tools like Cucumber and Behat, which made user stories (human readable text) to be executed as an acceptance test.
 
@@ -253,10 +253,11 @@ Given('I have products in my cart', (table) => { // eslint-disable-line
 });
 ```
 
-You can also use the parse() function to obtain an object that allow you to get a simple version of the table parsed by column or row, with header (or not) : 
-- raw(): returns the table as a 2-D array
-- rows(): returns the table as a 2-D array, without the first row
-- hashes(): returns an array of objects where each row is converted to an object (column header is the key)
+You can also use the `parse()` method to obtain an object that allow you to get a simple version of the table parsed by column or row, with header (or not):
+
+- `raw()` - returns the table as a 2-D array
+- `rows()` - returns the table as a 2-D array, without the first row
+- `hashes()` - returns an array of objects where each row is converted to an object (column header is the key)
 
 If we use hashes() with the previous exemple :
 

docs/commands.md

@@ -88,26 +88,35 @@ Run tests in parallel threads.
 npx codeceptjs run-workers 3
 ```
 
-## Run Rerun
+## Run Rerun <Badge text="Since 2.4" type="warning"/>
 
-Run tests with rerun of all suite some cycles. Use config.rerun params:
+Run tests multiple times to detect and fix flaky tests.
 
 ```
+npx codeceptjs run-rerun
+```
+
+For this command configuration is required:
+
+```js
 {
-...
-rerun: {
-  //how many times all tests in suite must pass 
+  // inside codecept.conf.js
+  rerun: {
+    // how many times all tests should pass
     minSuccess: 2,
-  //how many times we can try to rerun all test suite for reaching minSuccess count of passed test suite  
+
+    // how many times to try to rerun all tests
     maxReruns: 4,
   }
 }
 ```
 
-For example: 
- - minSuccess 1, maxReruns 5 - CodeceptJS will run all test suite no more than 5 times, until first successful run
- - minSuccess 3, maxReruns 5 - CodeceptJS will run all test suite no more than 5 times, until reaching 3 successfull runs
- - minSuccess 10, maxReruns 10 - CodeceptJS will run all test suite 10 times, and if any one test in any run will fail - all suite is failed.
+Use Cases:
+
+* `minSuccess: 1, maxReruns: 5` - run all tests no more than 5 times, until first successful run.
+* `minSuccess: 3, maxReruns: 5` - run all tests no more than 5 times, until reaching 3 successfull runs.
+* `minSuccess: 10, maxReruns: 10` - run all tests exactly 10 times, to check their stability.
+
 
 ## Dry Run
 

docs/helpers/ApiDataFactory.md

@@ -1,6 +1,6 @@
 ---
-permalink: helpers/ApiDataFactory
-editLink: https://github.com/Codeception/CodeceptJS/blob/master/lib/helper/ApiDataFactory.js
+permalink: /helpers/ApiDataFactory
+editLink: false
 sidebar: auto
 title: ApiDataFactory
 ---