Do not by default delay the execution of test cases

[brasmusson]

Summary

Do not by default delay the execution of test cases until all have been processed through the filter chain.

Details

• Do not by default delay the execution of test cases until all have been processed through the filter chain.
• Change the name of the TestRunStarted event to TestCount event.
• Add the option --count-first which ensures that the TestCount event is issued before any test case is executed.

Depends on cucumber/cucumber-ruby-core#147

Motivation and Context

See #1082 (comment).
Fixes #1167.

How Has This Been Tested?

The automatic test suite has been updated to verify this behaviour.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

• I've added tests for my code
• My change requires a change to the documentation.
• I have updated the documentation accordingly.

[mattwynne]

I think I'd like this to be called test_case_count

WDYT?

[mattwynne]

Thanks for pushing away at this @brasmusson, I'll be glad to have the code back to normally not processing all the test cases before execution starts.

Two big thoughts here:

1. Rather than having a configuration switch, could the BroadcastTestCountEvent filter interrogate the config to see if there are any listeners subscribed to the test_count event, and act as a no-op if there aren't any?

2. Wouldn't it be more useful to have an event that surfaces a complete list of all the test cases, rather than just a count?

What do you think?

[brasmusson]

There are a couple of choices here:

• Optional events?
  Should a "test count" event be optional, and only issued when batching up the test cases before execution? So far we do not have any optional events. On the other hand issuing the "test count" event after the test case has executed, as currently in the PR, does not seem very useful.
• The possibility to query the event bus for handlers, or the existence of handlers, for events?
  So far the event sources are ignorant of the existence of, or the identity of any handlers for the events they issue. Since handlers can register themselves after events have started to be issued, if an event should only be issued if there are any handlers registered, then we have to consider when the latest time to register that handler and still trigger the issuing of the optional event. Should that time be at the time of the afterConfiguration-hook, or immediately before the issuing of the optional event?

In this RP I choose not to introduce any new concepts (optional events of querying the event bus for handlers), therefore a configuration switch is used, and the result is a change in the execution order and a reordering of events.

[brasmusson]

@mattwynne I've update the PR so that the test_case_count event is only fired if a listener has registered itself or the event.

[mattwynne]

Nice one @brasmusson!

I was thinking it would be more generally useful if it send a list of all the test case file:line locations. You could count that, but you could also use it to look up (and render details about) the forthcoming test cases' scenarios from Gherkin documents.

Do you think it makes sense to replace this event with that one now, or as part of another iteration?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[stale]

This issue has been automatically closed because of inactivity. You can support the Cucumber core team on opencollective.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.