v5: Discussion

[nohwnd]

🙋‍♀️

I am raising a few questions in v5 readme and this is the place to start discussion so you don't have to go through all the respective issues and ask there.

[rdbartram]

I like the idea of Add-Dependency to simplify importing scripts, but what about modules? I had some test classes I created for pester testing vmware PowerCLi but I had to mark them as .ps1 and use add-dependency in the beforeall in order that they worked. Previously, i had them as using module statements at the top of the .test.ps1 but that doesn't work anymore.

Could we do something with Add-Dependency so it could import modules/classes?

Where is the code for Add-Dependency? If I use get-command, I see its in the pester.runtime. Where is that?

Finally, I'm abit confused if we should be using Add-Dependency or not because here #1283, you say using beforeall and not add-dependency, but in the readme you say using Add-Dependency. Thanks for the clarification

[nohwnd]

@rdbartram when I wrote Add-Dependency I made it do the same thing as ., but then I realized that I am not using it as such. Instead I am using the Add-Dependency in the same manner as BeforeAll, so the goal of the issue is to use the same syntax so there is one less cmdlet to think about.

I don't know if I am able to do anything about using module, so that is still something I need to research.

All in all the readme is going a bit outdated, as I go and form how it will all work. :)

[rdbartram]

ok. so i'll stick to dot sourcing in the beforeall as in v4. Is a cool idea though, to simplify the dependencies. Could even make it something like frontmatter at the start of the tests.ps1

[Stephanevg]

After writing this, I am not 100% sur if this should be in this thread.
Although the -Passthru object format is not mentionned in the readme document, I think it would make sense to discuss it a bit, as the change is pretty heavy compared to the v4 one. If you prefere having me open a issue of it own to track this just let me know.

Hi,
I looked at the output of the -Passthru object a bit today, and there is really a very big difference with the v4 one.

Describe "simple test" {
    it 'Should be false' {
        $true | Should be $false
    }

    it 'Should be true' {
        $true | should be $true
    }
}

Currently, executing the following test in v4 will result in this output:

pester v4.X output

TagFilter         : 
ExcludeTagFilter  : 
TestNameFilter    : 
ScriptBlockFilter : 
TotalCount        : 2
PassedCount       : 1
FailedCount       : 1
SkippedCount      : 0
PendingCount      : 0
InconclusiveCount : 0
Time              : 00:00:00.2968309
TestResult        : {@{ErrorRecord=Expected $false, but got $true.; ParameterizedSuiteName=; Describe=simple test; Parameters=System.Collections.Specialized.OrderedDictionary; Passed=False; Show=All; 
                    FailureMessage=Expected $false, but got $true.; Time=00:00:00.0120719; Name=Should be false; Result=Failed; Context=; StackTrace=at <ScriptBlock>, 
                    C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1: line 3
                    3:         $true | Should be $false}, @{ErrorRecord=; ParameterizedSuiteName=; Describe=simple test; Parameters=System.Collections.Specialized.OrderedDictionary; Passed=True; Show=All; 
                    FailureMessage=; Time=00:00:00.0058684; Name=Should be true; Result=Passed; Context=; StackTrace=}}

pester v5 output

Content              : C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1
Type                 : File
PSTypename           : ExecutedBlockContainer
OneTimeTestSetup     : 
Focus                : False
DiscoveryDuration    : 00:00:00.2508997
ExecutedAt           : 9/28/2019 9:19:08 AM
EachTestTeardown     : 
Id                   : 
ShouldRun            : True
PluginData           : {}
ScriptBlock          : 
AggregatedPassed     : False
Tests                : {}
FrameworkData        : {PreviouslyGeneratedTests, PreviouslyGeneratedBlocks}
Passed               : True
IsRoot               : True
OwnDuration          : 00:00:00
BlockContainer       : @{Content=C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1; Type=File}
FrameworkDuration    : 00:00:00.5209437
Root                 : @{OneTimeTestSetup=; First=True; Focus=False; DiscoveryDuration=00:00:00.2508997; ExecutedAt=9/28/2019 9:19:08 AM; EachTestTeardown=; Id=; ShouldRun=True; 
                       PluginData=System.Collections.Hashtable; ScriptBlock=; AggregatedPassed=False; Tests=System.Collections.Generic.List`1[System.Object]; Name=Root; 
                       FrameworkData=System.Collections.Hashtable; Passed=True; IsRoot=True; OwnDuration=00:00:00; BlockContainer=; FrameworkDuration=00:00:00; Root=; OneTimeBlockTeardown=; 
                       Blocks=System.Collections.Generic.List`1[System.Object]; Exclude=False; OneTimeTestTeardown=; Executed=True; OneTimeBlockSetup=; EachBlockSetup=; Path=; EachTestSetup=; 
                       EachBlockTeardown=; Last=True; StandardOutput=; Tag=; Parent=; Duration=00:00:00; ErrorRecord=System.Collections.Generic.List`1[System.Object]}
OneTimeBlockTeardown : 
Blocks               : {@{OneTimeTestSetup=; First=True; Focus=False; DiscoveryDuration=00:00:00.0230244; ExecutedAt=9/28/2019 9:19:09 AM; EachTestTeardown=; Id=0; ShouldRun=True; 
                       PluginData=System.Collections.Hashtable; ScriptBlock=
                           it 'Should be false' {
                               $true | Should be $false
                           }
                       
                           it 'Should be true' {
                               $true | should be $true
                           }
                       ; AggregatedPassed=True; Tests=System.Collections.Generic.List`1[System.Object]; Name=simple test; FrameworkData=System.Collections.Hashtable; Passed=True; IsRoot=False; 
                       OwnDuration=00:00:00.0137001; BlockContainer=; FrameworkDuration=00:00:00.3347260; Root=; OneTimeBlockTeardown=; Blocks=System.Collections.Generic.List`1[System.Object]; Exclude=False; 
                       OneTimeTestTeardown=; Executed=True; OneTimeBlockSetup=; EachBlockSetup=; Path=System.String[]; EachTestSetup=; EachBlockTeardown=; Last=True; StandardOutput=; Tag=System.String[]; 
                       Parent=; Duration=00:00:00.0137001; ErrorRecord=System.Collections.Generic.List`1[System.Management.Automation.ErrorRecord]}}
Exclude              : False
OneTimeTestTeardown  : 
Executed             : True
OneTimeBlockSetup    : 
EachBlockSetup       : 
EachTestSetup        : 
EachBlockTeardown    : 
Duration             : 00:00:00.0137001
ErrorRecord          : {}

I have written quite some automation around the object that -Passthru outputs in v3 / v4 and I was wondering if the object currently returned in v5 would be definitive one?

as much as I love all that extra information we get on the details of every run, I miss some of the information I have been used to through the different versions of pester.
Although that going up a major version we allow our selfs to introduce some breaking changes, I am a bit afraid of how much of my automation and CI checks I will need to change.

Also, it seems like that the information contained in the v5 object contains the details of the internal PesterRun.

Discussion

The part that would be really missing I think is the header of the v4 object.

TagFilter         : 
ExcludeTagFilter  : 
TestNameFilter    : 
ScriptBlockFilter : 
TotalCount        : 2
PassedCount       : 1
FailedCount       : 1
SkippedCount      : 0
PendingCount      : 0
InconclusiveCount : 0
Time              : 00:00:00.2968309
TestResult: 

I know that my self (and quite a lot of people I work with) use something like this to see if all tests passed.

$PesterRun = Invoke-Pester .\SimpleTest.ps1 -PassThru

If($PesterRun.FailedCount -ge 1){
  # Query failed pestertests from $PesterRun.TestsResults
}

This might be a naïve thought, but would it make sense to simply add a property to the existing v4 object, and put the new object of v5 in that one?
Something like RunDetails perhaps ?

Implementing the -Passthru as described above, could result in a -Passthruobject that would look like this

TagFilter         : 
ExcludeTagFilter  : 
TestNameFilter    : 
ScriptBlockFilter : 
TotalCount        : 2
PassedCount       : 1
FailedCount       : 1
SkippedCount      : 0
PendingCount      : 0
InconclusiveCount : 0
Time              : 00:00:00.2642866
TestResult        : {@{ErrorRecord=Expected $false, but got $true.; ParameterizedSuiteName=; Describe=simple test; Parameters=System.Collections.Specialized.OrderedDictionary; Passed=False; Show=All; 
                    FailureMessage=Expected $false, but got $true.; Time=00:00:00.0105793; Name=Should be false; Result=Failed; Context=; StackTrace=at <ScriptBlock>, 
                    C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1: line 3
                    3:         $true | Should be $false}, @{ErrorRecord=; ParameterizedSuiteName=; Describe=simple test; Parameters=System.Collections.Specialized.OrderedDictionary; Passed=True; Show=All; 
                    FailureMessage=; Time=00:00:00.0055245; Name=Should be true; Result=Passed; Context=; StackTrace=}}
RunDetails        : 
                    
                    Content              : C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1
                    Type                 : File
                    PSTypename           : ExecutedBlockContainer
                    OneTimeTestSetup     : 
                    Focus                : False
                    DiscoveryDuration    : 00:00:00.2698095
                    ExecutedAt           : 9/28/2019 9:25:02 AM
                    EachTestTeardown     : 
                    Id                   : 
                    ShouldRun            : True
                    PluginData           : {}
                    ScriptBlock          : 
                    AggregatedPassed     : False
                    Tests                : {}
                    FrameworkData        : {PreviouslyGeneratedTests, PreviouslyGeneratedBlocks}
                    Passed               : True
                    IsRoot               : True
                    OwnDuration          : 00:00:00
                    BlockContainer       : @{Content=C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1; Type=File}
                    FrameworkDuration    : 00:00:00.4905946
                    Root                 : @{OneTimeTestSetup=; First=True; Focus=False; DiscoveryDuration=00:00:00.2698095; ExecutedAt=9/28/2019 9:25:02 AM; EachTestTeardown=; Id=; ShouldRun=True; 
                                           PluginData=System.Collections.Hashtable; ScriptBlock=; AggregatedPassed=False; Tests=System.Collections.Generic.List1[System.Object]; Name=Root; 
                                           FrameworkData=System.Collections.Hashtable; Passed=True; IsRoot=True; OwnDuration=00:00:00; BlockContainer=; FrameworkDuration=00:00:00; Root=; OneTimeBlockTeardown=; 
                                           Blocks=System.Collections.Generic.List1[System.Object]; Exclude=False; OneTimeTestTeardown=; Executed=True; OneTimeBlockSetup=; EachBlockSetup=; Path=; 
                    EachTestSetup=; 
                                           EachBlockTeardown=; Last=True; StandardOutput=; Tag=; Parent=; Duration=00:00:00; ErrorRecord=System.Collections.Generic.List1[System.Object]}
                    OneTimeBlockTeardown : 
                    Blocks               : {@{OneTimeTestSetup=; First=True; Focus=False; DiscoveryDuration=00:00:00.0299721; ExecutedAt=9/28/2019 9:25:03 AM; EachTestTeardown=; Id=0; ShouldRun=True; 
                                           PluginData=System.Collections.Hashtable; ScriptBlock=
                                               it 'Should be false' {
                                                   True | Should be False
                                               }
                                           
                                               it 'Should be true' {
                                                   True | should be True
                                               }
                                           ; AggregatedPassed=True; Tests=System.Collections.Generic.List1[System.Object]; Name=simple test; FrameworkData=System.Collections.Hashtable; Passed=True; 
                    IsRoot=False; 
                                           OwnDuration=00:00:00.0153029; BlockContainer=; FrameworkDuration=00:00:00.3050340; Root=; OneTimeBlockTeardown=; 
                    Blocks=System.Collections.Generic.List1[System.Object]; Exclude=False; 
                                           OneTimeTestTeardown=; Executed=True; OneTimeBlockSetup=; EachBlockSetup=; Path=System.String[]; EachTestSetup=; EachBlockTeardown=; Last=True; StandardOutput=; 
                    Tag=System.String[]; 
                                           Parent=; Duration=00:00:00.0153029; ErrorRecord=System.Collections.Generic.List1[System.Management.Automation.ErrorRecord]}}
                    Exclude              : False
                    OneTimeTestTeardown  : 
                    Executed             : True
                    OneTimeBlockSetup    : 
                    EachBlockSetup       : 
                    EachTestSetup        : 
                    EachBlockTeardown    : 
                    Duration             : 00:00:00.0153029
                    ErrorRecord          : {}

Like that we won't break code that is strongly based the structure of the v4 object, and still offer new functionality to the Pester end users.

[nohwnd]

There is a function that summarizes the new object to the old one. It does not do everything exactly yet but it should in the future. It’s probably not published right now but it is used internally to get the summary at the end of the run. That is imho the way to go. Just put the adapter function in the pipeline after Invoke-Pester and before your existing build infrastructure and you are good to go.

[BoSorensen]

Could you also mention the breaking change to Should -Throw, it was introduced in #1003 and means that the substring match has been replaced with the -like operator: link to the v4 docs https://pester.dev/docs/usage/assertions for reference.

I guess it means that it is not possible to use Should -Throw that works both for v4 and v5, if you want to do substring matching.

Perhaps #675 could be reconsidered ?

[nohwnd]

@BoSorensen Okay gotcha. I think that #675 or something similar would be useful for migration where you can run your test suite in a mixed mode. It would probably be difficult to merge the test results, but I have a Legacy result converter, so maybe not. 🙂

[nohwnd]

@Stephanevg Coverter to legacy object will be merged shortly. #1472

[sdecker]

Are all the things listed in "Breaking changes" expected to be that way in the final release? It feels like many are "known issues". But some are actual breaking changes. If so could you please separate this section into two?

For example, not an exhaustive list
True breaking changes: these seems like they are expected not to work going forward

• Legacy syntax Should Be (without -) is removed, see Migrating from Pester v3 to v4
• Assert-VerifiableMocks was removed, Assert-MockCalled was renamed (but aliases exist), see Should -Invoke
• Assert-VerifiableMocks was removed, Assert-MockCalled was renamed (but aliases exist), see Should -Invoke

Really should be known issues: And these seem like known issues expected to be resolved or improved before final release

• Set-ItResult is published but does not work
• Code coverage report is not available.
• Mocks and Discovery are slow (whole execution is about 40% slower than on v4)

As it stands it's hard to make a business case of the value of moving to Pester 5 without knowing which are permanent.

[nohwnd]

@sdecker good point! I expect some of the known issues to be solved before 5.0 and some before 5.1, will, take that into consideration as well :)

[nohwnd]

@sdecker done 🙂

[dbafromthecold]

@nohwnd this may be me missing something, but has the -Script parameter for Invoke-Pester been removed?

Just tried updating the module to RC 5 and am getting...

A parameter cannot be found that matches parameter name 'Script'

When running something like...

Invoke-Pester -Script @{Path = "C:\pestertest\pestertest1.ps1";
Parameters = @{Param1 = $Value1;}}
-TestName 'MyPesterTest'
-PassThru -Show None

I checked the breaking features of the docs and didn't see it mentioned there. The function itself has -Script in the examples but it's not declared.

[nohwnd]

@dbafromthecold you are right the -Script does not exist anymore. It did too much stuff. It is called -Path now and accepts paths. The parametrized script running is not implemented and is documented here. #1485

I added -Script to breaking changes list.