Create scenario for every example#1168
Merged
Merged
Conversation
* Every single example is a scenario, even inside an example set * We create a unique scenario name by using an index
Member
|
@fourpastmidnight / @Jaykul could you give this a look and tell me if I should merge please :) |
Member
|
@cgnuechtel Also thanks for contributing 👍 |
Contributor
|
I think I would hold off on this one. The reason being is, I'm currently working on modifying how Gherkin-style tests in Pester are displayed to the console, especially with respect to step definitions containing DocStrings and Tables. My current solution is "working"--it prints DocStrings and Tables associated with Step Definitions (see #1124 and #1142). However, the implementation in #1142 currently messes up the VS Code problem matcher. Furthermore, #1142 does not yet address the Examples table for Scenario Outlines/Templates, but this support is planned. |
gnuechtel
added a commit
to gnuechtel/Pester
that referenced
this pull request
Jan 4, 2019
Contributor
fourpastmidnight
added a commit
to fourpastmidnight/Pester
that referenced
this pull request
Apr 5, 2019
** WIP COMMIT ** This commit builds on the work of @cgnuechtel via pester#1167 and PR pester#1168 and brings the implementation more in line with how Ruby Cucumber processes scenario outlines. This current commit stil prints all the scenario outline scenarios "long form". This from is as if you had specified --expand with Ruby Cucumber, or the -Expand parameter recently added to Pester Gherkin in a previous commit. In a future commit, the "compact form" will be implemented, which will streamline the output, as well as have the knock on effect of reducing the test run time because not as much printing to the console will occur, which is a slow operation.
fourpastmidnight
added a commit
to fourpastmidnight/Pester
that referenced
this pull request
Apr 5, 2019
This commit builds on the work of @cgnuechtel via pester#1167 and PR pester#1168 and brings the implementation more in line with how Ruby Cucumber processes scenario outlines, especially with respect to how the scenario outline example table scenarios are named and subsequently displayed in the console. This current commit stil prints all the scenario outline scenarios "long form". This form is as if you had specified --expand with Ruby Cucumber, or the -Expand parameter recently added to Pester Gherkin in a previous commit. In a future commit, the "compact form" will be implemented, which will streamline the output, as well as have the knock on effect of reducing the test run time because not as much printing to the console will occur, which is a slow operation. NOTE: I found a bug earlier today in the NUnit XML generation. While I fixed some unit tests with respect to XPaths for how secnario outline example scenarios are nested in the NUnit XML, I did not fix the underlying issue with NUnit XML generation yet as reported on pester#1285.
fourpastmidnight
added a commit
to fourpastmidnight/Pester
that referenced
this pull request
Apr 5, 2019
This commit builds on the work of @cgnuechtel via pester#1167 and PR pester#1168 and brings the implementation more in line with how Ruby Cucumber processes scenario outlines, especially with respect to how the scenario outline example table scenarios are named and subsequently displayed in the console. This current commit stil prints all the scenario outline scenarios "long form". This form is as if you had specified --expand with Ruby Cucumber, or the -Expand parameter recently added to Pester Gherkin in a previous commit. In a future commit, the "compact form" will be implemented, which will streamline the output, as well as have the knock on effect of reducing the test run time because not as much printing to the console will occur, which is a slow operation. NOTE: I found a bug earlier today in the NUnit XML generation. While I fixed some unit tests with respect to XPaths for how secnario outline example scenarios are nested in the NUnit XML, I did not fix the underlying issue with NUnit XML generation yet as reported on pester#1285.
fourpastmidnight
added a commit
to fourpastmidnight/Pester
that referenced
this pull request
Apr 6, 2019
This commit builds on the work of @cgnuechtel via pester#1167 and PR pester#1168 and brings the implementation more in line with how Ruby Cucumber processes scenario outlines, especially with respect to how the scenario outline example table scenarios are named and subsequently displayed in the console. This current commit stil prints all the scenario outline scenarios "long form". This form is as if you had specified --expand with Ruby Cucumber, or the -Expand parameter recently added to Pester Gherkin in a previous commit. In a future commit, the "compact form" will be implemented, which will streamline the output, as well as have the knock on effect of reducing the test run time because not as much printing to the console will occur, which is a slow operation. NOTE: I found a bug earlier today in the NUnit XML generation. While I fixed some unit tests with respect to XPaths for how secnario outline example scenarios are nested in the NUnit XML, I did not fix the underlying issue with NUnit XML generation yet as reported on pester#1285.
fourpastmidnight
added a commit
to fourpastmidnight/Pester
that referenced
this pull request
Apr 8, 2019
This commit builds on the work of @cgnuechtel via pester#1167 and PR pester#1168 and brings the implementation more in line with how Ruby Cucumber processes scenario outlines, especially with respect to how the scenario outline example table scenarios are named and subsequently displayed in the console. This current commit stil prints all the scenario outline scenarios "long form". This form is as if you had specified --expand with Ruby Cucumber, or the -Expand parameter recently added to Pester Gherkin in a previous commit. In a future commit, the "compact form" will be implemented, which will streamline the output, as well as have the knock on effect of reducing the test run time because not as much printing to the console will occur, which is a slow operation. NOTE: I found a bug earlier today in the NUnit XML generation. While I fixed some unit tests with respect to XPaths for how secnario outline example scenarios are nested in the NUnit XML, I did not fix the underlying issue with NUnit XML generation yet as reported on pester#1285.
fourpastmidnight
added a commit
to fourpastmidnight/Pester
that referenced
this pull request
Apr 16, 2019
This commit builds on the work of @cgnuechtel via pester#1167 and PR pester#1168 and brings the implementation more in line with how Ruby Cucumber processes scenario outlines, especially with respect to how the scenario outline example table scenarios are named and subsequently displayed in the console. This current commit stil prints all the scenario outline scenarios "long form". This form is as if you had specified --expand with Ruby Cucumber, or the -Expand parameter recently added to Pester Gherkin in a previous commit. In a future commit, the "compact form" will be implemented, which will streamline the output, as well as have the knock on effect of reducing the test run time because not as much printing to the console will occur, which is a slow operation. NOTE: I found a bug earlier today in the NUnit XML generation. While I fixed some unit tests with respect to XPaths for how secnario outline example scenarios are nested in the NUnit XML, I did not fix the underlying issue with NUnit XML generation yet as reported on pester#1285.
fourpastmidnight
added a commit
to fourpastmidnight/Pester
that referenced
this pull request
Apr 17, 2019
This commit builds on the work of @cgnuechtel via pester#1167 and PR pester#1168 and brings the implementation more in line with how Ruby Cucumber processes scenario outlines, especially with respect to how the scenario outline example table scenarios are named and subsequently displayed in the console. This current commit stil prints all the scenario outline scenarios "long form". This form is as if you had specified --expand with Ruby Cucumber, or the -Expand parameter recently added to Pester Gherkin in a previous commit. In a future commit, the "compact form" will be implemented, which will streamline the output, as well as have the knock on effect of reducing the test run time because not as much printing to the console will occur, which is a slow operation. NOTE: I found a bug earlier today in the NUnit XML generation. While I fixed some unit tests with respect to XPaths for how secnario outline example scenarios are nested in the NUnit XML, I did not fix the underlying issue with NUnit XML generation yet as reported on pester#1285.
fourpastmidnight
added a commit
to fourpastmidnight/Pester
that referenced
this pull request
Aug 5, 2019
This commit builds on the work of @cgnuechtel via pester#1167 and PR pester#1168 and brings the implementation more in line with how Ruby Cucumber processes scenario outlines, especially with respect to how the scenario outline example table scenarios are named and subsequently displayed in the console. This current commit stil prints all the scenario outline scenarios "long form". This form is as if you had specified --expand with Ruby Cucumber, or the -Expand parameter recently added to Pester Gherkin in a previous commit. In a future commit, the "compact form" will be implemented, which will streamline the output, as well as have the knock on effect of reducing the test run time because not as much printing to the console will occur, which is a slow operation. NOTE: I found a bug earlier today in the NUnit XML generation. While I fixed some unit tests with respect to XPaths for how secnario outline example scenarios are nested in the NUnit XML, I did not fix the underlying issue with NUnit XML generation yet as reported on pester#1285.
fourpastmidnight
added a commit
to fourpastmidnight/Pester
that referenced
this pull request
Nov 9, 2019
This commit builds on the work of @cgnuechtel via pester#1167 and PR pester#1168 and brings the implementation more in line with how Ruby Cucumber processes scenario outlines, especially with respect to how the scenario outline example table scenarios are named and subsequently displayed in the console. This current commit stil prints all the scenario outline scenarios "long form". This form is as if you had specified --expand with Ruby Cucumber, or the -Expand parameter recently added to Pester Gherkin in a previous commit. In a future commit, the "compact form" will be implemented, which will streamline the output, as well as have the knock on effect of reducing the test run time because not as much printing to the console will occur, which is a slow operation. NOTE: I found a bug earlier today in the NUnit XML generation. While I fixed some unit tests with respect to XPaths for how secnario outline example scenarios are nested in the NUnit XML, I did not fix the underlying issue with NUnit XML generation yet as reported on pester#1285.
fourpastmidnight
added a commit
to fourpastmidnight/Pester
that referenced
this pull request
Nov 19, 2019
This commit builds on the work of @cgnuechtel via pester#1167 and PR pester#1168 and brings the implementation more in line with how Ruby Cucumber processes scenario outlines, especially with respect to how the scenario outline example table scenarios are named and subsequently displayed in the console. This current commit stil prints all the scenario outline scenarios "long form". This form is as if you had specified --expand with Ruby Cucumber, or the -Expand parameter recently added to Pester Gherkin in a previous commit. In a future commit, the "compact form" will be implemented, which will streamline the output, as well as have the knock on effect of reducing the test run time because not as much printing to the console will occur, which is a slow operation. NOTE: I found a bug earlier today in the NUnit XML generation. While I fixed some unit tests with respect to XPaths for how secnario outline example scenarios are nested in the NUnit XML, I did not fix the underlying issue with NUnit XML generation yet as reported on pester#1285.
fourpastmidnight
added a commit
to fourpastmidnight/Pester
that referenced
this pull request
Nov 24, 2019
This commit builds on the work of @cgnuechtel via pester#1167 and PR pester#1168 and brings the implementation more in line with how Ruby Cucumber processes scenario outlines, especially with respect to how the scenario outline example table scenarios are named and subsequently displayed in the console. This current commit stil prints all the scenario outline scenarios "long form". This form is as if you had specified --expand with Ruby Cucumber, or the -Expand parameter recently added to Pester Gherkin in a previous commit. In a future commit, the "compact form" will be implemented, which will streamline the output, as well as have the knock on effect of reducing the test run time because not as much printing to the console will occur, which is a slow operation. NOTE: I found a bug earlier today in the NUnit XML generation. While I fixed some unit tests with respect to XPaths for how secnario outline example scenarios are nested in the NUnit XML, I did not fix the underlying issue with NUnit XML generation yet as reported on pester#1285.
fourpastmidnight
added a commit
to fourpastmidnight/Pester
that referenced
this pull request
Nov 28, 2019
This commit builds on the work of @cgnuechtel via pester#1167 and PR pester#1168 and brings the implementation more in line with how Ruby Cucumber processes scenario outlines, especially with respect to how the scenario outline example table scenarios are named and subsequently displayed in the console. This current commit stil prints all the scenario outline scenarios "long form". This form is as if you had specified --expand with Ruby Cucumber, or the -Expand parameter recently added to Pester Gherkin in a previous commit. In a future commit, the "compact form" will be implemented, which will streamline the output, as well as have the knock on effect of reducing the test run time because not as much printing to the console will occur, which is a slow operation. NOTE: I found a bug earlier today in the NUnit XML generation. While I fixed some unit tests with respect to XPaths for how secnario outline example scenarios are nested in the NUnit XML, I did not fix the underlying issue with NUnit XML generation yet as reported on pester#1285.
fourpastmidnight
added a commit
to fourpastmidnight/Pester
that referenced
this pull request
Dec 13, 2019
This commit builds on the work of @cgnuechtel via pester#1167 and PR pester#1168 and brings the implementation more in line with how Ruby Cucumber processes scenario outlines, especially with respect to how the scenario outline example table scenarios are named and subsequently displayed in the console. This current commit stil prints all the scenario outline scenarios "long form". This form is as if you had specified --expand with Ruby Cucumber, or the -Expand parameter recently added to Pester Gherkin in a previous commit. In a future commit, the "compact form" will be implemented, which will streamline the output, as well as have the knock on effect of reducing the test run time because not as much printing to the console will occur, which is a slow operation. NOTE: I found a bug earlier today in the NUnit XML generation. While I fixed some unit tests with respect to XPaths for how secnario outline example scenarios are nested in the NUnit XML, I did not fix the underlying issue with NUnit XML generation yet as reported on pester#1285.
fourpastmidnight
added a commit
to fourpastmidnight/Pester
that referenced
this pull request
Apr 1, 2020
This commit builds on the work of @cgnuechtel via pester#1167 and PR pester#1168 and brings the implementation more in line with how Ruby Cucumber processes scenario outlines, especially with respect to how the scenario outline example table scenarios are named and subsequently displayed in the console. This current commit stil prints all the scenario outline scenarios "long form". This form is as if you had specified --expand with Ruby Cucumber, or the -Expand parameter recently added to Pester Gherkin in a previous commit. In a future commit, the "compact form" will be implemented, which will streamline the output, as well as have the knock on effect of reducing the test run time because not as much printing to the console will occur, which is a slow operation. NOTE: I found a bug earlier today in the NUnit XML generation. While I fixed some unit tests with respect to XPaths for how secnario outline example scenarios are nested in the NUnit XML, I did not fix the underlying issue with NUnit XML generation yet as reported on pester#1285.
fourpastmidnight
added a commit
to fourpastmidnight/Pester
that referenced
this pull request
Apr 3, 2020
This commit builds on the work of @cgnuechtel via pester#1167 and PR pester#1168 and brings the implementation more in line with how Ruby Cucumber processes scenario outlines, especially with respect to how the scenario outline example table scenarios are named and subsequently displayed in the console. This current commit stil prints all the scenario outline scenarios "long form". This form is as if you had specified --expand with Ruby Cucumber, or the -Expand parameter recently added to Pester Gherkin in a previous commit. In a future commit, the "compact form" will be implemented, which will streamline the output, as well as have the knock on effect of reducing the test run time because not as much printing to the console will occur, which is a slow operation. NOTE: I found a bug earlier today in the NUnit XML generation. While I fixed some unit tests with respect to XPaths for how secnario outline example scenarios are nested in the NUnit XML, I did not fix the underlying issue with NUnit XML generation yet as reported on pester#1285.
fourpastmidnight
added a commit
to fourpastmidnight/Pester
that referenced
this pull request
Apr 4, 2020
This commit builds on the work of @cgnuechtel via pester#1167 and PR pester#1168 and brings the implementation more in line with how Ruby Cucumber processes scenario outlines, especially with respect to how the scenario outline example table scenarios are named and subsequently displayed in the console. This current commit stil prints all the scenario outline scenarios "long form". This form is as if you had specified --expand with Ruby Cucumber, or the -Expand parameter recently added to Pester Gherkin in a previous commit. In a future commit, the "compact form" will be implemented, which will streamline the output, as well as have the knock on effect of reducing the test run time because not as much printing to the console will occur, which is a slow operation. NOTE: I found a bug earlier today in the NUnit XML generation. While I fixed some unit tests with respect to XPaths for how secnario outline example scenarios are nested in the NUnit XML, I did not fix the underlying issue with NUnit XML generation yet as reported on pester#1285.
nohwnd
pushed a commit
that referenced
this pull request
May 26, 2021
* Add some other helpful VS Code settings
This commit adds some other helpful VS Code settings when working with
PowerShell Code:
* Ensure that *.ps1xml files are treated as XML files in the editor
* Ensure *.feature files have:
* comment quick suggestions turned off
* an editor ruler is displayed at column 110
* platyPs notes that 110 is a good value because GitHub tends
to cut off text at around column 110, causing horizontal
scrolling to occur
* tab size is 2 spaces
* Show a column ruler at columen 110 for PowerShell files
* Ensure PowerShell files are encoded with UTF8 with BOM
* This provides the most compatible encoding for both Windows and
Linux platforms
I've successfully used these settings on many of my own projects and
have found them very helpful.
This commit also adds four launch configurations for debugging some
Pester tests.
Lastly, it adds a tasks.json file which allows you to easily run the
Pester test suite in both the Desktop and Core versions of Powershell,
allowing you to run the test suite for only the RSpec runner, the
Gherkin runner, or the entire suite. There's even a task for running
PowerShell in v2 compatibility mode. But note that not all unit tests
pass in "compatibility" mode. It's still best to always test on a
machine running PowerShell v2.
* Rename Gherkin.ps1 to Gherkin.Core.ps1
This commit renames the main Gherkin file to Gherkin.Core.ps1. This is
an effort to better organize the Gehrkin files, especially since there
will be many more changes coming to improve the Gherkin test runner.
* Add new Gherkin.Output.ps1 file
This commit adds Gherkin-specific output functions. Similarly, all
usages of Write-Describe, Write-Context and Write-PesterResult have been
replaced with Write-Feature, Write-Scenario and Write-GherkinResult.
For now, these new Write-* functions are pretty much plain copies of th
existing Write-* functions. However, these will be updated as changes
are made to improve the Gherkin console output. The upcoming changes
will streamline the output which will have the overall result of making
the tests "run faster" (you'd be surprised how slow it is to print to
the console. These changes made in my private copy of Pester have seen
as much as a 50% improvement in perceived test run execution speed.)
These early commits are all about laying the groundwork for future
commits.
* Clean up Import-GherkinFeature
Cleaned up Import-GherkinFeature a bit in preparation for bigger
changes. Included in this changes is the Pester state object is a
mandatory parameter for Import-GherkinFeature. (I don't like magic
parameters that are sometimes passed, sometimes not, but still in scope.
If you have the parameter use it. Otherwise, make it a Script level
variable. I prefer the parameter.)
Making that parameter mandatory made some tests failed. I needed a way
to create and inject a Pester state object for the tests. So it seemed
like creating a New-GherkinPesterState function served that purpose,
whil also cleaning up Invoke-Gherkin a bit. Besides, there are changes
coming to that hunk of code anyway, so this will keep things nice and
neat and tidy.
* Put report strings and theme in Gherkin.psd1
This commit takes the ReportTheme defined in Output.ps1 and places it as
a property in the object being created in Gherkin.psd1. Similarly, the
"top-level" properties in the Data object in Gherkin.psd1 were placed as
properties on a new object assigned to the top-level property
ReportStrings.
In this way, all information for formatting of the test console output
is in the Gherkin.psd1 file.
These changes are being made to support upcoming output changes. The
output changes will result in streamlined reporting to the console,
which will lead to perceived faster test runs (up to 50% in some cases)
and will be more inline with how Ruby Cucumber displays its output.
* Replace all "-strings w/ '-strings in $SafeCommands
For every lookup involving $SafeCommands, replace the use of
double-quoted strings with single-quoted strings.
* Process exclude tags for features earlier
This commit filters out feature files which are excluded via exclude
tags _before_ processing any of the scenarios within the feature, only
to "throw away" all the processing work later.
* Don't Convert-Tags on Backgrounds
Don't bother propagating tags to background steps. First off, the
Gherkin.Ast.Background object doesn't have a property Tags, nor does it
inherit from any object or implement any interface which has a property
Tags. And secondly, background steps only ever execute in the context of
a scenario. So tags on a background would never be evaluated because all
tags for the scenario would already have been evaluated.
* Adds NoMultiline parameter to Invoke-GherkinStep
Pivoting from working on Import-GherkinFeature, now I'm divivg deep and
working inside out. First up is qualifying some of the parameters to
Invoke-GherkinStep and adding the new NoMultiline switch parameter.
This parameter is similar to the --no-multiline option for Ruby Cucumber
and prevents output of step definition data tables and DocStrings. In an
upcoming commit, support will be added for displaying data tabales and
DocStrings in the console during the test run.
* Invoke-GherkinStep improvements
This commit revamps Invoke-GherkinStep and adds several new
capabilities. It also simplified Invoke-GherkinScenario quite a bit by
taking advantage of a base-type inherited by both Gherkin Backgrounds
and Gherkin Scenarios and Scenario Outlines in Gherkin.dll v5.x,
allowing me to eliminate duplicate code, pushing that logic down into
Invoke-GherkinStep where it belongs.
The function was reorganized. The stop watch that times how long it
takes to execute the step definition, was moved very close to where the
step definition script block is actually executed. This more accurately
reports the time it takes to execue the step definition script block
itself, as opposed to the overhead execution time imposed by the runner
(which, I hope is recorded separately somewhere).
The logic was simplified for determining whether or not a step
definition should even be executed. All scenarios have a synthetic
'Result' property added to them when the feature files are imported and
parsed and the scenarios are processed. This made the simplification of
Invoke-GherkinScenario possible. This is much more efficient than
looping over all the steps for each scenario step to determine whether
or not the step should be executed; this should improve test run
execution times a bit. Once a step has a non-passing result, the code
checks whether the result is 'Inconclusive' (a/k/a in Gherkin,
undefined). If so, it attempts to determine whether or not the step that
would be executed exists or not. If not, the step is also marked as
'Inconclusive'. Otherwise, it is skipped. This is in keeping with how
Ruby Cucumber processes scenario steps. If a previous scenario step is
failed, then all future steps are skipped, regardless of whether or not
they may be undefined.
If no step definition is found for a scenario step, a new error record
is created using the new New-UndefinedStepErrorRecord function. If the
current scenario result is not 'Passed' or 'Inconclusive', then a new
error record is created using the New-StepSkippedErrorRecord function or
New-UndefinedStepErrorRecord function, as appropriate. If the step has
failed, the New-StepFailedErrorRecord function is used to capture
relevant information about the failure (i.e. stack trace, source
location, etc.). This new function neatly encapsulates a lot of the
code that was previously just hanging out at the bottom of
Invoke-GherkinStep. Lastly, if the scenario is currently passing and
there is a step definition, but the new Set-StepPending function is
called, then that function calls New-StepPendingErrorRecord function to
return an error record for an undefined step.
There used to be code that would update the test result stack trace in
the event of a non-passing step. That's all been moved into the new
ConvertTo-GherkinStepResult function. There was sufficiently different
behavior from the ConvertTo-PesterResult function that it just makes
sense to not try to reuse it and instead create one specific for
Gherkin. Likewise, there was sufficient differences in displaying the
stack trace between Pester RSpec and Gherkin such that
ConvertTo-FailureLines wasn't really cutting it, and so a new
Format-FailedStepResult function was created. Currently, there is a lot
of commented out code, copied from ConvertTo-FailureLines, that resides
at the bottom of the function. This may be needed to show more detailed
stack traces--but if it is determined not to be needed, then that
commented out code will be removed prior to the PR implementing these
changes being merged into master.
Lastly, there was a check: `if ($Pester -and $Visible)` that is actually
erroneous. first, `$Pester` should always exist. Secondly, if `$Visible`
is not specified, it would mess up the step counts in the final test
report. I'm not sure that was the intent of the `-Visible` parameter. In
any event, the `-Visible` parameter was always used with
`Write-PesterResult` (now called Write-GherkinStepResult), so the
parameter was simply removed altogether. Again, there will be sufficient
differences in how step results are displayed in the console as compared
to Pester RSpec-style tests, such that Write-PesterResult was replaced
with Write-GherkinStepResult for Gherkin-style tests. This new function
also takes a new parameter, `-NoMultiline` (as does Invoke-Gherkin,
Invoke-GherkinFeature, Invoke-GherkinScenario, and Invoke-GherkinStep,
to pass it down when given from the command line), which controls
whether or not step definition DataTables and DocStrings are displayed
in the console. (NOTE: This parameter is not used yet--that will be in
an upcoming commit when printing DocStrings and DataTables is added.)
* Exchange "-strings for '-strings
It's better to use single-quoted strings whenever you can (at leat,
according to Winows PowerShell in Action, 3rd Ed.). It was suggested
this improves performance because PS knows it doesn't have to be on the
lookout for expression substitutions in the string, which makes string
processing faster.
* Show step DocStrings and DataTables in console
This commit changes how steps are displayed in the console. First off,
the code was regorganized to make smaller, more descriptive functions.
This commit also adds support for printing DocStrings and DataTables. In
addition, the steps no longer display with [+] or [-] to denote their
status. Instead, color is used. This is in keeping with how Cucumber
displays its output. This change was made in support of further changes
to how console output will be displayed for features, scenarios, and
scenario outlines in upcoming commits.
Lastly, a change was made in how failing, undefined, and pending step
error messages and stack traces are displayed in the console. One aim
was to keep the format consistent so as to aid in viewing the errors in
the console. But another consideration was that of the Visual Studio
Code problem matchers. In the past, it has been difficult to make
changes to the Gherkin runner output without breaking the problem
matchers. While this change, too, breaks existing problem matchers, in
the end, it actually provides a consistent way of defining the problem
matchers, and so this is an improvement. Here are the problem matchers
I'm using in my VS Code tasks.json file for Pester Gherkin:
```json
{
// Your Task definition here...
"problemMatcher": [
{
"owner": "powershell",
"fileLocation": ["absolute"],
"severity": "error",
"pattern": [
{
"regexp": "^\\s*Error:\\s+(.*)$",
"message": 1
},
{
"regexp": "^\\s+at\\s+[^,]+,\\s*(.*?):line\\s+(\\d+)$",
"file": 1,
"line": 2
}
]
},
{
"owner": "powershell",
"fileLocation": ["absolute"],
"severity": "warning",
"pattern": [
{
"regexp": "^\\s*TODO: \\(Pester::Pending\\)$",
"message": 0
},
{
"regexp": "^\\s*at\\s+[^,]+,\\s*(.*?):line\\s+(\\d+)$",
"file": 1,
"line": 2
}
]
},
{
"owner": "powershell",
"fileLocation": ["absolute"],
"severity": "warning",
"pattern": [
{
"regexp": "^\\s*>>> No matching step definition found.$",
"message": 0
},
{
"regexp": "^\\s*at\\s+(.*?):line\\s+(\\d+)$",
"file": 1,
"line": 2
}
]
}
]
}
```
* Complete support for NoMultiline and Expand
This commit adds the NoMultiline and Expand parameters to Invoke-Gherkin
and ensures that they are passed down to where they're needed, paving
the way for the final set of changes that will produce MUCH nice output
with respect to Feature backgrounds and Scenario Outlines on par with
Ruby Cucumber.
Also, the `autoformatOnSave` setting auto-formatted some things in this
file. :|
* Improve Gherkin PesterState, more cleanup
This commit contains some more "cleanup" items before moving on to the
last part that constitutes the main goal of this PR: making the
display of Pester Gherkin test runs in the console on par with Ruby
Cucumber test runs. This has a few benefits: the output in the console
will look very similar to your feature files. This means that the
background steps won't be printed over and over and over and over again
for each scenario in a feature. They'll be displayed just once. You
can't imagine how much time this can potentially shave off of a test
run.
The Gherkin PesterState object was updated to include not only the names
of passed/failed scenarios and passed/failed count, but also the
pending/undefined secnarios and their counts, too. The code used to
retrieve the scenarios for each status was simplified. This was made
possible by using a TestGroup hint of 'Feature' rather than simply
'Script' and by ensuring the TestGroup's Describe and Context properties
had values (the feature name and scenario name, respectively).
The Invoke-GherkinFeature and Invoke-GherkinScenario functions had some
more reorganization performed. For features, all the code that "sets up"
the feature before actually enumerating over the scenarios to execute
them was moved closer to the scenario enumeration. In fact, that "setup"
code was moved to after all the tag filtering code. In addition, the
"feature teardown code" was moved to a finally block. Previously, the
line `Invoke-GherkinHook AfterEachFeature ...` was executed after we had
changed locations back to the location where the test run was started
from. This was sa bug, as the AfterEachFeature hook might have dependend
on the location set in Invoke-Gherkin. (This reorganization is important
because in a future commit, most of that tag filtering code will move to
further improve test run execution times.)
Likewise, a similar reorganization was made in Invoke-GherkinScenario.
"The World", as it's called in Ruby Cucumber, is setup at the very top
of the Try block where the secnario steps are enumerated and executed.
Likewise, "the World" is torn down in an added finally block. And the
ordering of teardown was changed to mirror the ordering of setup--like
bookends. Again, a future commit will change how scenarios and scenario
outlines are processed, and these changes help towards organizing the
code for these upcoming changes.
Finally, with the addition of PendingScenarios and UndefinedScenarios
properties on the Pester state object for Gherkin tests, one of the
exitsing unit tests failed (prior to this commit, scenarios with
undefined steps were considered "failing"). So that test was corrected
and two new tests were created to cover these additional PesterState
properites. In addition, 3 more tests were added for testing that the
proper step results are returned. For example, if you have a failing
step early in your scenario, and a later step is undefined, the
scenario's result is failed, but the undefined step would still have a
result of undefined. This is inline with how Ruby Cucumber represents
the results of scenario steps under similar circumstances.
All-in-all, these changes once again bring us a bit closer to being on
par with Ruby Cucumber and how it processes gherkin features and step
definitions.
* Fixes PSv2 PSObject Unwrapping, PSv6 Group-Object Sorting
So, this commit fixes two issues that ended up being interrelated to one
another--somewhat by accident.
A few commits back, I lamented that PSv2 was losing the PSObject wrapper
over the Scenario and Background objects in Import-GherkinFeature when
passing them to Invoke-GherkinScenario (actually, the wrapper was lost
when Invoke-GherkinScenario passed them along to Invoke-GherkinStep). At
the time, the workaround was to check if the ETS property Result was
still attached to the object, and if not, then reattach it. This had the
knock-on effect that I couldn't make use of the Result ETS property on
the Scenario object in the Pester TestResults object to easily determine
the passed, failed, pending and undefined scenarios. And I think this
has been a long-standing issue in the code since before I touched it,
leading to somewhat complicated code to determine the statu of the
secnario a'la ETS ScriptProperties added to the PesterState object.
I found a way to simplify that code a great deal. The only problem is,
as of PSv6.1, Group-Object now sorts the groups on their property key(s)
to achieve n*log(n) performance rather than n*n performance. Quite a
significant improvement. Though, I wasn't sure if the ordering of teh
secnario results was important at all--who knows how the Results object
is being used once it's left Pester.
If only I could use that Result ETS property on the scenarios. Because
then I wouldn't have to do any grouping on the individual step results,
but could query the secnarios directly. It would be faster, and simpler,
and maintain the order of the scenarios. So I decided to find out why
the PSObject wrapper was being lost along the way for PSv2.
It turns out, unlike in PSv3 and later, when you specify a parameter
type directoly, such as `[Gherkin.Ast.Scenario]`, PSv2 coerces the type
back to the type specified (so long as there's a PSTypeName for it or PS
can convert it), thereby losing the PSObject ATS/ETS wrapper. I left the
types of the paramteres commented out in the code, more as a reminder.
Plus, in Pester v5, PSv2 support is being dropped and this whole issue
goes away, allowing us to use more "staically-typed" parameters for
type-safety.
Anyway, fixing that issue allowed the ETS Result property to flow all
the way through the test run so that it was available on the Scenario
objects which where added as a collection ETS property on the Feature
object, which is included in the final test results object. This allowed
me to push down all of those properties directly onto the
Pester.Gherkin.Result object returned when using the -PassThru
parameter, and use much simpler code to determine the result of any one
scenario.
Nice.
At this point, I'm ready for the last set of changes to better display
features, scenarios and backgrounds during the test run, bringing Pester
Gherkin nearly on par with Ruby Cucumber.
* Process scenario outlines similarly to Ruby Cucumber
This commit builds on the work of @cgnuechtel via #1167 and PR #1168 and
brings the implementation more in line with how Ruby Cucumber processes
scenario outlines, especially with respect to how the scenario outline
example table scenarios are named and subsequently displayed in the
console.
This current commit stil prints all the scenario outline scenarios "long
form". This form is as if you had specified --expand with Ruby Cucumber,
or the -Expand parameter recently added to Pester Gherkin in a previous
commit.
In a future commit, the "compact form" will be implemented, which will
streamline the output, as well as have the knock on effect of reducing
the test run time because not as much printing to the console will
occur, which is a slow operation.
NOTE: I found a bug earlier today in the NUnit XML generation. While I
fixed some unit tests with respect to XPaths for how secnario outline
example scenarios are nested in the NUnit XML, I did not fix the
underlying issue with NUnit XML generation yet as reported on #1285.
* Extract method to get DataTable max column widths
* Adds support for unexpanded scenarios
This commit adds support for unexpanded scenarios. All this means in the
context of this commit is that the Scenario Outline example scenario
names are formatted such that, when they are printed in tabular form,
all the columns will line up nice and neat. However, this commit does
not add the support for printing out the compact secnario outline form.
That's coming up in a few commits. However, before adding support for
"pretty printing" of scenario outlines, pretty-printing of features and
scenarios themselves must be updated now that everything is in place.
* Improves Gherkin console test run output
This commit improves Gherkin test run console output. The "theme" has
been updated to match the output used by the Ruby Cucumber pretty-print
formatter.
Among the improvements are:
* The default "margin" is 2 spaces. Each "level", e.g. scenario
outlines, scenarios, examples, etc., are indented by a multiple of this
margin. NOTE: Description text is not reformatted in any way--i.e. all
spacing and indentation in the feature file is preserved.
* The background is printed only once per feature, instead of once per
scenario. This leads to nicer looking output in the console. It's also
less repetitive in the console. And, due to these steps only being
displayed once, results in faster output (which leads to a perceived
increase in test run speed).
* Both exapanded and compact scenario outline example scenarios are
supported.
* Adds Write-GherkinReport
This commit adds a Gherkin-specific console test report summary which
matches the output of Ruby Cucumber detailing the number of Scenarios
for each scenario result, as well as the number of steps by result, and
the time it took for the test run.
At some point earlier in this PR, I had "pushed down" the creation of
the PassedScenarios, FailedScenarios, etc. properties from the
PesterState object to the test results object, forgetting that these
properties were used on the PesterState object for this console test
summary. But all was not in vain, because in moving this code around, I
found a very succinct way of obtaining this information. So even though
I have to move it back, the result is much cleaner.
* Return a feature w/ its background and scenarios
This commit is mainly a cleanup commit. Instead of using the ,-operator,
use the @() syntax to ensure that the scenarios returned from
Import-GehrkinFeature are stored as an array. Also, instead of returning
a 3-tuple of Feature, Background, and Scenarios from
Import-GherkinFeature, the feature is more properly a parent of a
background and one or more scenario definitions (each of which are
either a scenario outline or a regular scenario). So just attach those
items to the Feature object and, if there are any scenarios, then return
the feature.
The rest were odds/ends, removing old comments, fixing spelling, etc.
* Rework stack trace printing
Back when I first reworked console printing, I only emitted abberviated
stack traces, leaving a significant portion of code commented out. This
commit reinstates much of that code, but in a reorganized fashion.
This reorganization makes sense for the Gherkin runner. And this
reorganization allowed me to reduce the number of lines of code used for
step result printing by ~80%.
This change allows for the abbreviated stack traces that were being
printed before, as well as full stack traces (via the global
PesterDebugPreference_ShowFullErrors variable), useful for debugging the
Pester test framework itself.
* Update StackTrace printing to work with PSv2
This commit refactors the previous stack trace rework code so that
printing stack traces is compatible with PowerShell v2.
In Gherkin.Core.ps1, there was code in Import-GherkinFeature which never
worked in PowerShell v2, but which did not throw a terminating error.
Instead, this was compensated for later. (I believe it was compensated
for when converting the result into a pester test result, where the
stack trace was massaged.)
Instead of using "compensating code" or having more than one way to do
the same thing (one for PSv2, one for everything else), I opted to
change the way information is stored and propagated in order to have the
information required to create nice, consistent stack traces.
* Fix test report output for single scenario/step results
If a test run consisted of a single scenario, part of the regex used to match
the test output in order to colorize it failed to match because it was
looking for the word 'scenarios'. When only a single scenario is run,
the text 'scenarios' is replaced with the singular 'scenario'. So the
regex was updated so that the entire report line would be matched: e.g.
Before this fix:
1 undefined)
After this fix:
1 scenario (1 undefined)
A similar change was made for the code which reports the step results.
* Fixes Invoke-GherkinHook break jump label target
There has been a long-standing bug that, fortunately, the code where
it's located is not executed often and so hasn't been found until
recently by @Jaykul. Branch target labels should not have the ':'
prepended to their name.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implementation for #1167