Add -Parallel switch

[nohwnd]

Add the ability to run tests in parallel by providing the number of concurrent runners, and making sure the output is written in blocks and not one over other.

[Jaykul]

At what scope are we talking about parallelizing? I mean, what "things" would become parallel?

Fixtures? Describe? It?

[nohwnd]

It is whole files (for now?). They usually have common setup on the top of the file and one or more Describes.

[nohwnd]

Moving this up to 5.x, it should be technically possible but needs more research to make the api simple and useful at the same time.

[DhruvM7]

Would love to see this feature, our build times are getting longer as we keeping adding tests for new powershell code.

[JustinGrote]

Would love to see this feature, our build times are getting longer as we keeping adding tests for new powershell code.

As a workaround, if your tests are not dependent on each other you can just use Foreach -Parallel to run each test file or even each test separately and then stitch the returned result objects back together, that's what we do.

[kilasuit]

^^ was what I had been doing with PoshRSJobs ages back

[kilasuit]

If Pester took a dependency on the ThreadJob module then you could do this quite quickly and easily.

I think as mentioned it would need to be File level based testing & not Describe Context It block levels to make the most sense

[JustinGrote]

@kilasuit wouldn't really need Threadjob, Runspaces and Runspace pools are "relatively" simple to work with compared to a lot of what is done in Pester (which Threadjob is just a wrapper around).

My opinion is there are so many potential issues around state and race conditions, it's pretty easy to "bring your own threading" to Pester now so I wouldn't say it's a "need to have" but definitely a "nice to have"

[majkinetor]

@nohwnd
It is whole files (for now?).

@JustinGrote
As a workaround, if your tests are not dependent on each other you can just use Foreach -Parallel to run each test file or even each test separately and then stitch the returned result objects back together, that's what we do.

This is a workaround I considered on my project, but some test files are way more heavy then the others. So it really needs to send each specific test to available thread for full parallelization effect. This is currently not trivially achievable as you can't execute test by name (something along that line is mentioned on #608). If test results actually provide info about LineNumber (the same as VsCode peste stub) it would be easy, although IMO the correct way is to make single test execution by name possible then orchestrate individual tests over available threads.

My opinion is there are so many potential issues around state and race conditions, it's pretty easy to "bring your own threading" to Pester now so I wouldn't say it's a "need to have" but definitely a "nice to have"

How your own threading solution resolves state and race conditions ? This is pretty expected option from any test runner, so I would categorize it as a must rather then nice to have.

[nohwnd]

@majkinetor You can run tests by name, but actually doing Discovery and from there taking the tests to run would be much easier way to do this. A cmdlet that can only do discovery is not published but is implemented and mostly works. That said, why I consider only file-level parallelization is because tests have their own Setups and teardowns and you would need to run those for every test based on where you are running. You would also have to make sure that your teardowns can run more than one time, because we would never know how many times they would run.

Another reason is that you need to start one executor for each, and load modules in it, so this is far from lightweight.

All of this would burden the user with a lot (we see the same problems and complaints in VSTest / MSTest), and for the majority of code bases the added performance is not worth the complication that per-test paralellization would introduce.

[majkinetor]

That said, why I consider only file-level parallelization is because tests have their own Setups and teardowns and you would need to run those for every test based on where you are running.

Its IMO not generally wise to deduce how people will use the library. In my case, my setup routines are very fast, taking less then a sec. In fact, I often do 1..1000 | % { invoke-pester -TagFilter debug } when chasing flaky test.

But I get your point and it could be easily solvable using options. Parallelization could be run using following option:

• Number of tests to run from single file at once: 1..1000.

If set to 1, it runs 1 test from test file per thread (my scenario)
if very large, like 1000, it runs all the test in a file (your scenario)
And people can fine tune it, depending on the speed of the setup/teardown, lets say somebody will for example use 10. If I set 10, then test file with 55 tests will run in 6 threads in parallel.

This allows maximal flexibility without complicating implementation almost at all.