Skip to content

Make Gherkin test runner behave more like Ruby's Cucumber#2

Closed
fourpastmidnight wants to merge 2 commits into
fix-#1124-pester-does-not-print-DocStrings-or-DataTablesfrom
skip-rest-of-scenario-steps-after-first-non-passing-step
Closed

Make Gherkin test runner behave more like Ruby's Cucumber#2
fourpastmidnight wants to merge 2 commits into
fix-#1124-pester-does-not-print-DocStrings-or-DataTablesfrom
skip-rest-of-scenario-steps-after-first-non-passing-step

Conversation

@fourpastmidnight
Copy link
Copy Markdown
Owner

@fourpastmidnight fourpastmidnight commented Nov 19, 2018

This represents an early first pass at updating the Gherkin implementation within Pester. This branch sits atop of PR pester#1142/Issue pester#1124 in the upstream Pester repository.

I would've just created a PR on the main repository, but I didn't want to waste CI build resources on this at the moment--as there are two tests that need to be fixed up due to the changes in this branch, and this is too early for merging consideration into the upstream project anyway.

  1. I wanted more Cucumber-like output from Gherkin-style tests. So this branch contains a very early first crack at customizing the output for the Gherkin test output while leaving the Pester output alone. It definitely needs work pursuant to conversations that have taken place on Feature Request: Support an implicit @ignore tag pester/Pester#1115 at the upstream repository.
  2. To make the above possible, Gherkin-style tests now skip all steps after the first failing step in a scenario. After all, why bother running the rest of the scenario if a step has failed? The scenario has already failed. This leads nicely to the next thing.
  3. Allow Gherkin-Style steps to explicitly set themselves as Pending through a new function: Set-PendingStep. Cucumber does this when it gives you a "suggested" step implementation for an undefined step.
    • As with failed steps, once a Pending step is encountered, the rest of the scenario steps are skipped and the scenario is counted as Pending.
  4. Allow Gherkin-style tests to recognize and report Undefined steps.
    • In the current implementation, Pester counts these as inconclusive. I still count them as "inconclusive" but they are displayed as Undefined.
    • Also currently, Pester would execute other defined steps which come after an undefined step. Since this makes no sense, any following defined steps are skipped and the scenario is counted as Undefined
      • NOTE: Actually, there is a TODO on this right now--technically, the rest of the steps are currently not skipped, but that will change in a later commit.

I'm unhappy with a few things: namely in Output.ps1, I basically copied the existing Write-PesterResult function to a Write-GherkinResult function and modified it for Gherkin. But pursuant to our conversation on pester#1115, this is where that "extensibility" model that has been discussed would come in handy, if possible. And this is what I plan to investigate next after cleaning up a few things with the current branch state.

@fourpastmidnight fourpastmidnight self-assigned this Nov 19, 2018
fourpastmidnight
fourpastmidnight commented Nov 19, 2018
View reviewed changes
Comment thread Functions/Assertions/Set-StepPending.ps1
return $errorRecord
}

function Set-StepPending {
Copy link
Copy Markdown
Owner Author

@fourpastmidnight fourpastmidnight Nov 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider removing this function and adding a -Pending parameter to GherkinStep in Gherkin.ps1. See #1.

@fourpastmidnight fourpastmidnight force-pushed the skip-rest-of-scenario-steps-after-first-non-passing-step branch from dc0bcb2 to eb95df0 Compare November 20, 2018 21:35
@fourpastmidnight fourpastmidnight force-pushed the fix-#1124-pester-does-not-print-DocStrings-or-DataTables branch from 48fc7c3 to fe174f6 Compare December 6, 2018 18:59
@fourpastmidnight fourpastmidnight force-pushed the skip-rest-of-scenario-steps-after-first-non-passing-step branch from eb95df0 to 4a36fca Compare December 6, 2018 19:19
@fourpastmidnight fourpastmidnight force-pushed the fix-#1124-pester-does-not-print-DocStrings-or-DataTables branch from fe174f6 to ac0ba5e Compare December 16, 2018 21:33
@fourpastmidnight fourpastmidnight force-pushed the skip-rest-of-scenario-steps-after-first-non-passing-step branch from 4a36fca to 2b0c48c Compare December 16, 2018 21:33
@fourpastmidnight fourpastmidnight force-pushed the fix-#1124-pester-does-not-print-DocStrings-or-DataTables branch from ac0ba5e to 66d6c6e Compare December 19, 2018 19:24
@fourpastmidnight fourpastmidnight force-pushed the skip-rest-of-scenario-steps-after-first-non-passing-step branch from 2b0c48c to 257fd32 Compare December 19, 2018 19:24
@fourpastmidnight fourpastmidnight force-pushed the fix-#1124-pester-does-not-print-DocStrings-or-DataTables branch from 66d6c6e to 4851e3c Compare January 3, 2019 16:11
@fourpastmidnight fourpastmidnight force-pushed the skip-rest-of-scenario-steps-after-first-non-passing-step branch from 257fd32 to e6410a8 Compare January 3, 2019 16:12
@fourpastmidnight fourpastmidnight force-pushed the fix-#1124-pester-does-not-print-DocStrings-or-DataTables branch from 4851e3c to 0a84607 Compare January 3, 2019 20:52
@fourpastmidnight fourpastmidnight force-pushed the skip-rest-of-scenario-steps-after-first-non-passing-step branch from e6410a8 to c4e7dab Compare January 3, 2019 20:53
@fourpastmidnight fourpastmidnight force-pushed the fix-#1124-pester-does-not-print-DocStrings-or-DataTables branch from 0a84607 to e6f8553 Compare January 4, 2019 16:56
@fourpastmidnight fourpastmidnight force-pushed the skip-rest-of-scenario-steps-after-first-non-passing-step branch from c4e7dab to 97300c4 Compare January 4, 2019 16:57
@fourpastmidnight fourpastmidnight force-pushed the fix-#1124-pester-does-not-print-DocStrings-or-DataTables branch from e6f8553 to ca0015d Compare January 4, 2019 22:34
@fourpastmidnight fourpastmidnight force-pushed the skip-rest-of-scenario-steps-after-first-non-passing-step branch from 97300c4 to 52caeee Compare January 4, 2019 22:34
@fourpastmidnight
Add a Set-StepPending function to Assertions
5af19ce 
In Cucumber, when you write a Feature file and then run cucmuber on it
without implementing any steps whatsoever, the result is that the
scenario (and steps) are considered 'undefined'. For Pester, we
currently treat these as Inconclusive (which is fine--we can synonomize
Inconclusive with Undefined). However, there's no explicit mechanism to
mark a step as pending.

This commit adds a Set-StepPending "assertion" which is similar in
nature to the Set-TestInconclusive "assertion". This will mark that step
as 'Pending'.

This commit only adds the Set-StepPending function. The next series of
commits will update Pester's Gherkin implementation to use it.
@fourpastmidnight fourpastmidnight force-pushed the skip-rest-of-scenario-steps-after-first-non-passing-step branch from 52caeee to 0a1a877 Compare January 7, 2019 15:26
@fourpastmidnight fourpastmidnight force-pushed the fix-#1124-pester-does-not-print-DocStrings-or-DataTables branch from ca0015d to ba42aec Compare January 7, 2019 15:26
@fourpastmidnight
@wip - add undefined/pending scenarios/steps
26fb0a8 
This commit contains work-in-progress towards the following:

* Adding the ability for Gherkin-style Pester tests to recognize and
report undefined steps and scenarios
    * Undefined steps are steps in a feature file for which no
    PowerShell script block has been written.
        * TODO: Still need to add output that shows a "suggested"
	implementation
	* TODO: If any preceding step definitions are undefined, then
	any defined steps in the scenario should be skipped.
    * If at least one scenario step is undefined, then the scenario as a
    whole is undefined.
* Added the ability for Gherkin-style Pester tests to recognize pending
steps and scenarios
    * Pending steps are steps whose implementation explicitly sets the
    step to pending (via the new Set-StepPending function), which is
    similar to cucumber's pending function.
    * Once a pending step has been encountered, all other steps in the
    scenario are skipped -- afterall, even if other steps are
    implemented, how can the test possibly pass if there's a pending
    step?
    * If at least one scenario step is pending, then the scenario is
    also Pending.
* If a scenario step fails, all other steps are skipped. Afterall,
what's the point in running the rest of the scenario if a previous step
has failed? The scenario has still failed.
    * As alluded, if at least one step has failed, then the scenario has
    failed.
* Customized the output of Gherkin style tests to match that of
cucumber.
    * Mostly a stylistic change, which is optional
    * HOWEVER, the 'Inconclusive' state is used for 'Undefined'. Gherkin
    doesn't actually make use of an Inconclusive state, and since Pester
    expects certain states, all of which are used in Gherkin _except_
    Inconclusive, it made sense to usurp this and use it for Undefined.
    	* NOTE: I'm actually not happy about having to do this. The
	mental mapping that needs to take place when working on the
	Pester code, while simple now, will become much more complex as
	time goes on. Something more extensible needs to be implemented.
    * For the stylistic changes, I'm setting up the codebase to allow
    "output theming". But, this is in its very infancy and is not well
    thought out yet.
@fourpastmidnight fourpastmidnight force-pushed the skip-rest-of-scenario-steps-after-first-non-passing-step branch from 0a1a877 to 26fb0a8 Compare January 7, 2019 19:41
@fourpastmidnight
Copy link
Copy Markdown
Owner Author

fourpastmidnight commented Dec 13, 2019

Closing this PR. This has been accomplished with a PR against the main project (pester#1276).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@fourpastmidnight fourpastmidnight

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@fourpastmidnight