Export test results to HTML

[gnuechtel]

• Support additional output format HTML for Invoke-Pester and Invoke-Gherkin
• Count total, passed, inconclusive and failed results by three groups
  • Files/Groups/Specs for Pester Specs
  • Features/Scenarios/Steps for Gherkin
• Include summary of information like PowerShell version, date/time, culture info, operation system or duration
• Internally use XSLT script NUnitXml-html.xslt to transform NUnit XML to HTML
• Export Convert-Report cmdlet to allow conversion of already created XML files
• HTML output is also XHTML to allow easier testing
• Add extensive tests for Gherkin HTML reports
• Add smaller set of tests for PSpec HTML reports which should be sufficient
• Changes should be backward compatible since string parameters for output file and format were replaced by string[]

Share XML test code

• Store it in Functions/TestUtilities/Xml.ps1
• Use it in Gherkin.Tests.ps1 and TestResults.Tests.ps1
• Remove it before release in cleanUpBeforeBuild.ps1

[gnuechtel]

This pull request adds a new output format HTML.

There are two unclear things with the code:

• the inclusion of shared XML test code
• the location of the used XSLT script

Any advice is welcome!

[gnuechtel]

Example report (clipped)

[gnuechtel]

Another example (Gherkin)

[nohwnd]

@cgnuechtel I like the functionality, but it has very thin connection to Pester itself, because it depends on the NUnit schema. So I am thinking if this should be included in Pester or not, and I am so far leaning more towards not merging this. That said I like the code, I like how it looks, I just don't know if I want to be stuck supporting this for the next 10 years.

Have you considered publishing this as a separate module?

[gnuechtel]

@cgnuechtel I like the functionality, but it has very thin connection to Pester itself, because it depends on the NUnit schema. So I am thinking if this should be included in Pester or not, and I am so far leaning more towards not merging this. That said I like the code, I like how it looks, I just don't know if I want to be stuck supporting this for the next 10 years.

I am with you at this point that the dependency to NUnit XML should be removed.
For example, if Pester switches to NUnit 3 format, the HTML output would be broken.
(Which would not be very problematic since there are HTML output tests.)
Furthermore I have to admit, that Export-PesterResults is too long now and has too much functions which should be separated.

Though the HTML is pretty simple. And XSLT is not so bad as some people may think.
But from the maintenance point of view it could be a little bit difficult.

I suggest one of the following options to remove the dependency of NUnit XML format:

1. Create an intermediate XML format for Pester and adjust the XSLT
2. Directly write HTML in PowerShell code (in TestResults.ps1 or a similar file)

Have you considered publishing this as a separate module?

Not yet. I think a HTML report should be part of the core.
For example Cucumber has HTML output.

But in general the export should be more modular and separated by different formats.

Would that changes convince you to include HTML output into Pester core?

[fourpastmidnight]

Just to chime in here, I think the NUnit XML output is just fine. I think the objection here is adding support, into Pester itself, for transforming that report to HTML. If my understanding of the objection is correct, I agree. There are already several formatters available which can consume the NUnit XML report and output nice HTML reports, as I mentioned on #1142, such as Pickles and DonutReport (Demo).

[fourpastmidnight]

BTW, I also liked the work you did in this PR; I just happen to agree with @nohwnd in this case. I also think the work done here on this PR would be a valuable contribution as a module that others could use apart from, but in concert with Pester--as another option to the aforementioned formatters above.

[nohwnd]

@cgnuechtel I think you understood me the other way around. What @fourpastmidnight says is what I meant. Your code can convert any NUnit XML output, not just the output Pester produces (and that's a good thing). So your code can be shipped in it's own module without adding XSLT "dependency" to Pester. I hope it's not too disappointing :/

[nohwnd]

@cgnuechtel I am going to close this as: won't merge. I value your work on this, but I don't think it belongs into the framework. It adds yet another thing that I (or potential other maintainers) need to know and the functionality can easily be provided separately.

[gnuechtel]

It is okay to close this for now.

I am considering to provide an extra module.
But I am not sure, if this makes any sense.
Because the other report tools are amazing and go far beyond this simple output.

The goal of this pull request was to provide a simple output like Cucumber does.
(Not to provide a sophisticated reporting tool.)

Cucumber also provides a JSON output (which DonutReport seems to require).

Still wondering, if the output should be more generic and not tied to just one output format.

NUnit XML is a widely accepted format by many CI servers.
Also more output formats will increase complexity.
But, I still think that a limited amount of output formats would be fine, if maintenance is easier than the code of this pull request.

If it makes sense, I will provide some code later with a more generic approach (without XSLT and separated code by output format).

But before, I will try out the reporting tools.

[fourpastmidnight]

Well, Pester has an argument that allows you to specify the output type. It just so happens that only NunitXML is supported at this time. But there’s no reason another format could not be added. However, before jumping in with that, there would need to be discussion about how additional formats should be added, because I’ve seen in the last few months a few questions with respect to PRs and whether or not it breaks the report output. So the question that needs to be answered is, what’s the correct approach for adding output formats such that: 1. When contributions are made, one can unequivocally know whether they’ve broken an output formatter. (Obviously, tests for the output—but they must be comprehensive enough to be helpful) 2. The formatter is easy to maintain and self-contained. I think this is the bigger point of discussion. a. Should Pester “natively” support output formats? b. Or, should there be some “extensibility” mechanism that’s used? c. This question probably should be discussed with ongoing discussions surrounding the “modularization” of Pester. But I too, want to thank you for your work on this item, because it does look nice, even if it’s not as sophisticated as other formatters for NunitXML. Depending on the person, what you’ve done may be “good enough” and may still be worthy of a nice module in it’s own right.

…

Sent from my Windows 10 phone From: Christian Gnüchtel Sent: Saturday, January 5, 2019 9:14 To: pester/Pester Cc: Craig E. Shea; Mention Subject: Re: [pester/Pester] Export test results to HTML (#1186) It is okay to close this for now. I am considering to provide an extra module. But I am not sure, if this makes any sense. Because the other report tools are amazing and go far beyond this simple output. The goal of this pull request was to provide a simple output like Cucumber does. (Not to provide a sophisticated reporting tool.) Cucumber also provides a JSON output (which DonutReport seems to require). Still wondering, if the output should be more generic and not tied to just one output format. NUnit XML is a widely accepted format by many CI servers. Also more output formats will increase complexity. But, I still think that a limited amount of output formats would be fine, if maintenance is easier than the code of this pull request. If it makes sense, I will provide some code later with a more generic approach (without XSLT and separated code by output format). But before, I will try out the reporting tools. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[nohwnd]

@fourpastmidnight To discuss this further there should be another thread, but a short answer to your questions, is that in my opinion the approach that we are taking now is the way to go:

• Have a single output format that widely used and supported by CI servers -> NUnit, and at best support two versions of it.
• Transform the output internally from the same -PassThru object that we "publish", to make sure that the information is accessible.
• Let people implement their transformers for the -PassThru object and try not to introduce breaking changes in it.

So all in all, I will try to keep/reduce the scope of what Pester can natively do to the crucial parts + useful stuff that 90% of the users will definitely need, and the rest of the stuff should come as a separate module. We have a powerful package delivery system provided by PSGallery, so let's use it.

[fourpastmidnight]

I think this is a very pragmatic approach at this time.

…

Sent from my Windows 10 phone From: Jakub Jareš Sent: Saturday, January 5, 2019 11:45 To: pester/Pester Cc: Craig E. Shea; Mention Subject: Re: [pester/Pester] Export test results to HTML (#1186) @fourpastmidnight To discuss this further there should be another thread, but a short answer to your questions, is that in my opinion the approach that we are taking now is the way to go: • Have a single output format that widely used and supported by CI servers -> NUnit, and at best support two versions of it. • Transform the output internally from the same -PassThru object that we "publish", to make sure that the information is accessible. • Let people implement their transformers for the -PassThru object and try not to introduce breaking changes in it. So all in all, I will try to keep/reduce the scope of what Pester can natively do to the crucial parts + useful stuff that 90% of the users will definitely need, and the rest of the stuff should come as a separate module. We have a powerful package delivery system provided by PSGallery, so let's use it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.