Run tests against Windows

[danielbachhuber]

To help prevent regressions like wp-cli/wp-cli#5521, it would be nice to run our test suites against Windows.

Done is:

• Our automated tests are running against Windows in all repos.

[mlipe-nxs]

As part of the nightly testing of the mentioned issue I've run the test suite on Windows which fail with the following error.

1) Extractor_Test::test_extract_tarball
Failed asserting that 1 is identical to 0.

wp-cli\tests\test-extractor.php:119

Tracing the error I find this.

tar: Option --force-local is not supported

Removing the ( Utils\is_windows() ? ' --force-local' : '' ) part of tar command within \Extractor_Test::test_extract_tarball makes all tests pass.

OK, but incomplete, skipped, or risky tests!
Tests: 270, Assertions: 722, Risky: 1.

I tested on Windows 10 against:

1. PHP 7.4
2. PHP 8.0
3. PHP 8.1

[danielbachhuber]

Removing the ( Utils\is_windows() ? ' --force-local' : '' ) part of tar command within \Extractor_Test::test_extract_tarball makes all tests pass.

Looks like that was added ~5 years ago: wp-cli/wp-cli@271cd75#diff-ce006bf8351a3174bac26fef9078e893523d313ace23732b67ad97942e7b2ec4R115-R116

It was included in the same PR as the original issue: wp-cli/wp-cli#4572

My guess is that @gitlost fixed some "earlier versions of PHP on Windows" issue that no longer exists. I don't know what the particular issue was or when it was resolved, though.

Pardon my extremely naive question but... which versions of Windows should we support?

[lipemat]

Great question...
I believe the following are probably a reasonable place to start.

1. Windows 10
2. Windows 11
3. Windows Server 2019
4. Windows Server 2022

[BrianHenryIE]

No harm running tests against MacOS either, if it's easy to add to the matrix. I've had to fix differences in tar and sed. ...it's not unlikely that those same issues will reoccur under Windows. Unfortunately I don't have a regular Windows environment to test in (currently searching to see is Azure a tool I could use).