WIP: Fix #1124 pester does not print doc strings or data tables

[fourpastmidnight]

1. General summary of the pull request

Fixes #1124.

For Gherkin Step Definitions which have DocString or DataTable data, this data is now printed to the console by default. A new switch parameter has been added to Invoke-Gherkin called HideStepData which controls whether or not DocString/DataTable data is printed to the console: when enabled, this data is not printed. The default is for the data to be printed.

[fourpastmidnight]

Looking for feedback on this PR. @Jaykul maybe you could take a look when you have some time.

[fourpastmidnight]

One possible improvement that could be made is to extract this functionality out into a private Function. The same could be said for where this function processes Scenario Outlines to create new Scenarios for each table row of data.

[fourpastmidnight]

I meant to completely remove this--not just comment it out. I guess I should first solicit comments on the removal of this unused function--because perhaps it's there for a reason? Depending on the responses I get, I'll either drop this commit or rebase the actual removal of the function.

[Jaykul]

This is meant for tooling --hypothetically-- your editor can call this when it sees you writing in a .feature file to show you intellisense, to auto-complete "when ..." sentences, or highlight missing steps, etc. It's basically the first step toward integrating with the PSES...

[fourpastmidnight]

OH!! Definitely useful in the future. Ok, I’ll drop that commit. Thanks!!

[fourpastmidnight]

Wow, I recently used Find-GherkinStep to find an existing step and execute it from within another step to help keep my test code dry. So this is definitely a useful cmdlet.

[fourpastmidnight]

@Jaykul: Just wanted to check, based on my previous comment on #1124, do you think it's a good idea to definitely add the option to disable the displaying of DocStrings/DataTables in the console to the OutputType enum? I'd like to wrap this up.

@nohwnd Do you have any preference as to how this should be handled, especially given that RSpec style tests don't make use of DocStrings nor DataTables (to my knowledge—I did see something called implicit docstrings, but that also implied the use of an extension to RSpec, so I wouldn't think this applies to Pester)?

[nohwnd]

Replied to that here

[fourpastmidnight]

Well this is a bummer. I think printing the DocStrings and Tables to the console messes up the problem matcher. At least there's an option to turn off displaying these in this feature.

This seems to be a common issue however: changes made to the output mess up the problem matcher.

Perhaps whenever a step fails, immediately after the step text, there's some "sentinal" text, like --- Error Details --- is printed to the console. In this way, the problem matcher can match this line, then the first two lines following would be the actual data that is used to populate the Problems pane, followed by any additional information.

[fourpastmidnight]

OK, I think the problem matcher being broken is not a huge deal. Let me explain why.

First: There's a way to turn off "pretty printing" of Tables and DocStrings. So, if you really want the problem matcher to work, just don't print Tables and DocStrings. Ideal? No.

(NOTE: it might be nice to output the Tables and DocStrings to the NUnit XML report, even if pretty-printing of Tables and DocStrings to the console is disabled. Hmm, and this may require another option switch to Invoke-Gherkin. This would aid tools that take the NUnit XML report and create nice documentation from the specs, such as Pickles and/or DonutReport (sample DonutReport demo.).)

Second: I think I could get more bang for my buck if I were to look into implementing a test adapter VS Code extension that can be used by the VS Code Test Explorer extension. Again, this could be an "opt-in" type thing in lieu of using the problem matcher. And again we could leverage the NUnit XML Report output. I believe that even though this option requires additional work, longer-term it could inoculate us from breaking the all-too-fragile problem matcher everytime console output changes. I also believe this would provide a much better user experience within VS Code for surfacing test failures and navigating to them from the Test Explorer hub. (As an aside, just now I thought how it's funny that Pester, a test framework which does not require VS Code to run, has code specifically to support VS Code. It would be nice to remove this accidental coupling/knowledge about the outside world from Pester.)

Any thoughts, comments, concerns?

[fourpastmidnight]

However, I did just manage to get the VS Code problem matcher to work, and I kind of like the results. I'll push up the necessary changes later, so please hold off on merging this until then. It did require a change to the problem matcher regular expressions, however.

It appears that the problem matcher is acquired one of two ways:

1. Implementing the regular expressions yourself in the VS Code tasks.kson file
2. Copying an example tasks.json file from the VS Code PowerShell extension

Perhaps we should publish the example tasks.json file with the required problem matcher specification, so that Pester is the canonical source? (Of course, pursuant to my comment above, this is purely for convenience, as Pester does not require the use of VS Code.)

[nohwnd]

@fourpastmidnight are there more changes needed? I renamed it to WIP: so I stop re-reading it when I go through open PRs. When it's complete please remove the WIP and ideally tag me in comment. Thx

[fourpastmidnight]

@nohwnd: Yes, more changes are needed and forthcoming. I should’ve renamed this awhile back. Sorry about that. And, thanks for renaming.

…

Sent from my Windows 10 phone From: Jakub Jareš Sent: Saturday, January 5, 2019 8:17 To: pester/Pester Cc: Craig E. Shea; Mention Subject: Re: [pester/Pester] WIP: Fix #1124 pester does not print doc stringsor data tables (#1142) @fourpastmidnight are there more changes needed? I renamed it to WIP: so I stop re-reading it when I go through open PRs. When it's complete please remove the WIP and ideally tag me in comment. Thx — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[fourpastmidnight]

OK, so now that a separate scenario is created for each row of an example table for a scenario outline, the implementation of this needs to change a great deal.

BUT, that's OK. Because this has caused me to realize that we've mixed two unrelated concerns in Pester's Gherkin test implementation. The mixing of these two concerns was going to cause greater and greater friction at some point anyway. Adding in #1168 just caused this friction to happen sooner. But again, this is a good thing. It provides the opportunity to clean up the code and separate these concerns.

The concerns in question are:

1. Generating the objects necessary to execute the tests, e.g. features, scenarios, steps, etc.
2. Formatting various strings to display the test run execution to the console (and, also in other outputs, such as the NUnit XML reports)

[fourpastmidnight]

Hmm, reading The Cucumber Book on around pg. 70 where they delve more into Scenario Outlines and DataTables, cucumber has a flag: --expand which, instead of "pretty printing" the DataTables, causes the individual scenarios created from each row in the data table to be printed.

So, it would seem after the implementation of #1168, though it's not printed correctly with that implementation, the effect is that cucumber --expand is being executed for Gherkin-style tests. So, let's use that to our advantage to control how scenario outlines will be displayed to the console with this PR.

In fact, based on how cucumber --expand produces its output for a scenario outline, we should aim to produce similar output.

[fourpastmidnight]

And here's a blog article that shows the output from running cucumber --expand.

[fourpastmidnight]

OK, looking at the implementation of Gherkin.dll, Gherkin does not make a distinction between Scenario Outlines and Scenarios. It parses a "ScenarioLine" and description and then some steps and then "Examples", which consist of an "ExamplesLine" (e.g. the keyword and name for the example table(s)), a description (optional) and one or more "ExamplesTable"s.

This is interesting considering the output shown in the previously linked-to blog article. It's almost as if a Scenario Outline functions much like an RSpec Context would within a Describe (except that, there's no shared state being created--the "context" is merely acting as a container for all the scenarios created from the example table(s)). This is exactly how I was planning on modeling this in Pester for Gherkin style tests. It's good to know I was on the right track.

[fourpastmidnight]

Note to self: Rename -HideStepData to -NoMultiline which closely matches the cucumber switch --no-multiline which also suppresses outputting to the console window DataTables and DocStrings of Step Definitions.

[fourpastmidnight]

I've come to the realization that the current implementation of Gherkin tests in Pester isn't conducive to running them as cucumber would run them. In fact, the current implementation leaves a lot to be desired in terms of performance and reporting. Now, I'm not "bashing" all the work that went into this--I think what has been done is absolutely phenomenal!, and my hat's off to all who have contributed to make possible what we have with Gherkin-style tests in Pester!

But it seems to me that this organically grown implementation of Gherkin tests living within the Pester framework needs a different approach. How will this fit in with the RSpec style of tests Pester supports? I'm unsure. But I am working on overhauling the Gherkin implementation, with the focus on these key areas:

1. Improved Performance
2. Improved reporting -- both to the console and as reporting artifacts
3. Fix current scoping issues when using Mocks (hopefully 😉)

So, if you think I've been quiet lately, it's because I just finished reading Windows PowerShell in Action, 3rd Ed. about one week ago and have been taking my first stab at a new approach to Gherkin style tests in Pester.

When I have something to show, I will submit a new PR and officially abandon this one. The new PR will be far from "Mergeable", more a WIP, but with the intent of garnering feedback.

[nohwnd]

I think it would be better to wait for beta of v5 and do it there. We can then work together to do this, if you are able to describe to me what the changes should be, and I to you how the new internals work :D

[fourpastmidnight]

I was thinking this would be a v5 change. 😉 Maybe I could create a v5 “issue” item with the details where further discussions can be had.

…

Sent from my Windows 10 phone From: Jakub Jareš Sent: Tuesday, February 12, 2019 22:30 To: pester/Pester Cc: Craig E. Shea; Mention Subject: Re: [pester/Pester] WIP: Fix #1124 pester does not print doc stringsor data tables (#1142) I think it would be better to wait for beta of v5 and do it there. We can then work together to do this, if you are able to describe to me what the changes should be, and I to you how the new internals work :D — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[nohwnd]

@fourpastmidnight sure :)

[fourpastmidnight]

I've recently made significant progress on this for the 4.x branch of Pester. I'm still wrapping up some loose ends. I will warn however, that I have a lot of stuff coming up that will require some time/attention in the next three weeks, so it may be just a bit longer than I would like. But, wow. Significant progress here!

[nohwnd]

@fourpastmidnight that's great :) I still did not have time to look into your code :/ Hopefully when Pester v5 beta is out.

[fourpastmidnight]

This is being superseded by #1276.