RFC: Module development and release management

[renehernandez]

1. General summary of the issue

The purpose of this is to begin a conversation about applying community practices for the Pester project with a focus on module development and release management

Static Analysis

• Introduce usage of PSScriptAnalyzer to check the code before running the tests
• The Set-StrictMode and Help validation checks could be moved as custom rules for PSScriptAnalyzer

Folder structure reorganization

• Separate functionality using a Private/Public folders approach within a Pester folder.
• Split components in Private into multiple files
• Move code out of Pester.psm1 into new functions files that will invoked within Pester.psm1

Tests

• Analyze whether we should split tests into unit tests, integration, etc, including the tests for the output and release process

Build Management

• Switch to use a task-based automation framework (could be either InvokeBuild or PSake)
• Manage build dependencies using PSDepend
• Unify build jobs on Azure Pipeline, is it possible?
• Remove manual steps for releasing new version, avoiding future issues like (Broken build on master for v2 #1303)

Help and Documentation

• Automate generation/update of markdown files for public Help information using PlatyPS
• Document release process

Any additions for these sections are welcomed and could be discussed (also removal of any o fthe listed ones), or any new section that I am missing

If there are already issues that are tracking some of these sections, let's link it here to have better accountability of the work

[nohwnd]

We can start doing this when Pester 5 is out and v4 is not supported anymore, otherwise it will be hell for me to merge patches. So let's not take action on any of this right now :))

I would of course like to have a nicer code layout, and it would be much easier to do in Pester v5, where most of the pieces are disconnected, but ultimately what is shipped should be a single file because that loads way way faster. So thinking about the benefits vs the additional work I can see that re-organizing the project would bring a lot of additional work to me as the maintainer, very little benefit to the end-user, and some benefits to current and potential contributors:

Static Analysis

I generally agree and the code should be already in reasonably good health, there was some work done on complying with PSScritAnalyzer rules in the past. On the other hand there are a lot of edge cases and weird practices that need to be there, because Pester does a lot of stuff other modules don't need to do.

Folder structure reorganization

Sure why not if the organization of the code is well described and generally agreed upon as the standard. I assume this is coming from the powershell style guide?

This should be only for development time, on publish there should be just one psm1 file with everything. Right now there are also sub modules, it would be useful to flatten them, or remove them, which moves further from nicely organized and separated code, but it is necessary due to lack of tooling (like npm) that would allow us to manage dependencies easily and tree shake them.

Tests

I don't think there are many integration tests (as in integration with real-world system), and most of the tests are going agains the public stuff (or they should), so the test files will still be rather large if we split them on function.ps1 and function.tests.ps1 basis (e.g. tests for Mock).

Build management

• the build itself is not very difficult, I am not sure if adding another level of DSL above it is necessary, I personally never learned how to use PSAke because scripts were always more readable for me
• is psdepend a matured project now? Last time I checked it was more of a proof of concept that people would love to have working, but would need a lot of work to get to a working solution, and also changes in tooling and how people approach modules
• Unify builds to azure - it is possible but not desirable, the builds now run on free tiers of each build cloud (apart from azure), they can run in parallel, and each platform runs on cloud that specializes in it (e.g. travis is great for linux and osx but probably not as great for windows)
• Manual issues caused way less problems than the fully automated solutions. I can't tell you how much work it was to deploy through Team City when it decided that it will not do the work that day, :) Automating it from start to end should not be difficult, but I still need to be able to release only after confirmation (which would be easy in VSTS)

Help and Documentation

• generating the help via platyps is in the plan, and the issue for it is open for a long time
• the release process is quite simple, documenting it would be easy, on the other hand I am the only person holding the signature key and I know how to release :)

All in all this is a huge amount of work, and I only have a limited amount of time that I rather spend on the functionality, but once v5.3 is out, and there are less than 20 open issues, then it will be time to start re-organizing the solution.

[renehernandez]

I totally agree on not doing the work twice. I just want us to have a single location where we can keep track of the overall changes required to get this done.

And this won't be something that you would have to do alone :)

[fourpastmidnight]

I am in favor of this. I have developed a very well organized module project structure that has been serving me well now for a while. I've been rather busy lately and have not had time to work on my PR here--so I would love to say I'd post something to my GitHub account, but I don't want to make promises I can't keep! 😞

Suffice it to say, the structure I've been using facilitates development in VS Code, complete with build tasks (tasks.json) and debugging (launch.json).

If there's interest in seeing my project structure, please say so, and I will attempt to post the template structure in my GitHub account.

[renehernandez]

@fourpastmidnight Please don't hesitate to chime with the your examples when you have the time. We are still very much away of finalizing any decision regarding how we plan to implement this and the more info we can gather the better :)

[fflaten]

Status:

• CI is unified in Azure Pipelines
• Static analysis enabled and scanning PRs using GitHub Actions
• Help for public functions are validated using tests. Only mandatory for public functions and operators, so easier to scan when module is built.
• Help is still in code, but markdown docs for pester.dev are generated using platyPS (indirectly). Not sure if markdown-first is planned
• Code split into multiple files and merged into Pester.psm1 during build. Not split 1:1 per function atm.

[nohwnd]

I remember trying to split to functions, which was not very difficult. But making tests work afterwards was the hard part. Many tests test a part of the whole thing, and splitting into separate files makes us identify those "subsystems" and maintain them even though in the final build we build everything into one file.