From 8cc8643203f7ffd477696174e316e364834af9ce Mon Sep 17 00:00:00 2001 From: Phil Davis Date: Tue, 19 Feb 2019 18:31:42 +0545 Subject: [PATCH 1/3] Add section about tagging scenarios to acceptance-tests.adoc --- .../pages/core/acceptance-tests.adoc | 156 ++++++++++++++++++ 1 file changed, 156 insertions(+) diff --git a/modules/developer_manual/pages/core/acceptance-tests.adoc b/modules/developer_manual/pages/core/acceptance-tests.adoc index 15c5705130..84f332839f 100644 --- a/modules/developer_manual/pages/core/acceptance-tests.adoc +++ b/modules/developer_manual/pages/core/acceptance-tests.adoc @@ -482,6 +482,162 @@ then put them into a `Background:` section. For example: This reduces some duplication in feature files. +[[controlling-test-scenarios]] +== Controlling Running Test Scenarios In Different Environments + +A feature or test scenario might only be relevant to run on a system-under-test +that has a particular environment. For example, a particular app enabled. + +To allow the test runner script to run the features and scenarios relevant to +the system-under-test the feature file or individual scenarios are tagged. +The test runner script can then filter by tags to select the relevant features and/or scenarios. + +=== Tagging A Whole Feature Or Single Scenario + +If a tag is mentioned at the top of a feature file, on the line before the `Feature` keyword, +then that tag applies to all scenarios in the feature file. + +[source,gherkin] +---- +@api @provisioning_api-app-required +Feature: add groups + As an admin + I want to be able to add groups + So that I can more easily manage access to resources by groups rather than individual users +---- + +If a tag is mentioned on the line before a particular `Scenario` or `Scenario Outline`, +then that tag applies just to that `Scenario` or `Scenario Outline`. + +[source,gherkin] +---- + @smokeTest + Scenario Outline: Creating a comment on a file belonging to myself +---- + +Multiple tags can be put one one line or on multiple lines. Use multiple lines for readability, +particularly if there are many tags or long tag names. + +[source,gherkin] +---- + @smokeTest @skipOnOcV10.0 + @comments-app-required + @issue-12345 + Scenario: Delete a comment on a shared file +---- + +=== Tagging Features By API, CLI and webUI + +Tag every feature with its major acceptance test type `api`, `cli` or `webUI`: + +[source,gherkin] +---- +@api +Feature: add groups + As an admin + I want to be able to add groups + So that I can more easily manage access to resources by groups rather than individual users +---- + +[source,gherkin] +---- +@cli +Feature: add group + As an admin + I want to be able to add groups + So that I can more easily manage access to resources by groups rather than individual users +---- + +[source,gherkin] +---- +@webUI +Feature: login users + As a user + I want to be able to log into my account + So that I have access to my files +---- + +This allows the the tests of a particular major type to be easily run or skipped. + +=== Tagging Scenarios That Require An App + +When a feature or scenario requires a core app to be enabled then tag it like: + +[source,gherkin] +---- + @comments-app-required + @federation-app-required + @files_trashbin-app-required + @files_versions-app-required + @notifications-app-required + @provisioning-app-required + @systemtags-app-required +---- + +The above apps might be disabled on a system-under-test. Tagging the feature or scenario allows +all tests for the app to be easily run or skipped. + +For tests in an app repository, do not tag them with the app name (e.g. `files_texteditor-app-required`). +It is already a given that the app in the repository is required for running the tests! + +=== Tagging Scenarios That Need to Be Skipped + +==== Skip UI Tests On A Particular Browser + +Some browsers have difficulty with some automated test actions. To skip scenarios for a browser tag them with the relevant tags: + +[source,gherkin] +---- + @skipOnCHROME + @skipOnFIREFOX + @skipOnINTERNETEXPLORER + @skipOnMICROSOFTEDGE +---- + +=== Skip Tests On A Particular Version Of ownCloud + +The acceptance test suite is sometimes run against a system-under-test that has an older version of ownCloud. +When writing new test scenarios for a new or changed feature, tag them to be skipped on the previous recent release of ownCloud. +Use tag formats like the following to skip on a particular major, minor or patch version. + +[source,gherkin] +---- + @skipOnOcV10 + @skipOnOcV10.0 + @skipOnOcV10.0.10 + @skipOnOcV10.1 +---- + +=== Skip Tests In Other Environments + +`@skipOnLDAP` - skip the scenario if the test is running with the LDAP backend. For example, some user provisioning features may not be relevant when LDAP is the backend for authentication. + +`@skipOnStorage:ceph` - skip the scenario if the test is running with `ceph` backend storage. + +`@skipOnStorage:scality` - skip the scenario if the test is running with `scality` backend storage. + +`@skipOnEncryption` - skip the scenario if the test is running with encryption enabled. + +`@skipOnEncryptionType:masterkey` - skip the scenario if the test is running with `masterkey` encryption enabled. + +`@skipOnEncryptionType:user-keys` - skip the scenario if the test is running with `user-keys` encryption enabled. + +== Tags For Tests To Run In Special Environments + +`@smokeTest` - this scenario has been selected as part of a base set of smoke tests. + +`@TestAlsoOnExternalUserBackend` - this scenario is selected as part of a base set of tests to run when a special user backend is in place (e.g. LDAP). + +`@local_storage` - this scenario requires and tests the local storage feature. + +== Special Tags for UI Tests + +`@insulated` - this makes the browser driver restart the browser session between each scenario. It helps isolate browser state. +When the browser session is recording there is a separate video for each scenario. Use this tag on all UI scenarios. + +`@disablePreviews` - generating previews/thumbnails takes time. Use this tag on UI test scenarios that do not need to test thumbnail behavior. + + [[how-to-add-a-new-feature]] == How to Add New Test Steps From cfc6ea7b7e77a3a745b1ca6a09364636b22097de Mon Sep 17 00:00:00 2001 From: Phil Davis Date: Wed, 20 Feb 2019 11:46:53 +0545 Subject: [PATCH 2/3] Use a reference to the Behat tags doc --- .../pages/core/acceptance-tests.adoc | 34 +------------------ 1 file changed, 1 insertion(+), 33 deletions(-) diff --git a/modules/developer_manual/pages/core/acceptance-tests.adoc b/modules/developer_manual/pages/core/acceptance-tests.adoc index 84f332839f..a67eeb98fa 100644 --- a/modules/developer_manual/pages/core/acceptance-tests.adoc +++ b/modules/developer_manual/pages/core/acceptance-tests.adoc @@ -492,39 +492,7 @@ To allow the test runner script to run the features and scenarios relevant to the system-under-test the feature file or individual scenarios are tagged. The test runner script can then filter by tags to select the relevant features and/or scenarios. -=== Tagging A Whole Feature Or Single Scenario - -If a tag is mentioned at the top of a feature file, on the line before the `Feature` keyword, -then that tag applies to all scenarios in the feature file. - -[source,gherkin] ----- -@api @provisioning_api-app-required -Feature: add groups - As an admin - I want to be able to add groups - So that I can more easily manage access to resources by groups rather than individual users ----- - -If a tag is mentioned on the line before a particular `Scenario` or `Scenario Outline`, -then that tag applies just to that `Scenario` or `Scenario Outline`. - -[source,gherkin] ----- - @smokeTest - Scenario Outline: Creating a comment on a file belonging to myself ----- - -Multiple tags can be put one one line or on multiple lines. Use multiple lines for readability, -particularly if there are many tags or long tag names. - -[source,gherkin] ----- - @smokeTest @skipOnOcV10.0 - @comments-app-required - @issue-12345 - Scenario: Delete a comment on a shared file ----- +For general information on tagging features and scenarios see http://behat.org/en/latest/user_guide/organizing.html#tags[the Behat tags documentation]. === Tagging Features By API, CLI and webUI From 69386e9627256426e8568e5b4a099eb9c14a625b Mon Sep 17 00:00:00 2001 From: Matthew Setter Date: Wed, 20 Feb 2019 10:55:35 +0100 Subject: [PATCH 3/3] Minor restructure of the acceptance tests changes This change helps make better use of native AsciiDoc features, such as tables, reflows sentences one per-line, uses headers to provide extra clarity, removes extra formatting in some code blocks. --- .../pages/core/acceptance-tests.adoc | 123 +++++++++++------- 1 file changed, 74 insertions(+), 49 deletions(-) diff --git a/modules/developer_manual/pages/core/acceptance-tests.adoc b/modules/developer_manual/pages/core/acceptance-tests.adoc index a67eeb98fa..b0d6ad5804 100644 --- a/modules/developer_manual/pages/core/acceptance-tests.adoc +++ b/modules/developer_manual/pages/core/acceptance-tests.adoc @@ -1,4 +1,5 @@ = Acceptance Tests +:behat-tags-docs-url: http://behat.org/en/latest/user_guide/organizing.html#tags [[the-test-directory-structure]] == The Test Directory Structure @@ -482,21 +483,22 @@ then put them into a `Background:` section. For example: This reduces some duplication in feature files. -[[controlling-test-scenarios]] == Controlling Running Test Scenarios In Different Environments -A feature or test scenario might only be relevant to run on a system-under-test -that has a particular environment. For example, a particular app enabled. +A feature or test scenario might only be relevant to run on a system-under-test that has a particular environment. +For example, a particular app enabled. -To allow the test runner script to run the features and scenarios relevant to -the system-under-test the feature file or individual scenarios are tagged. -The test runner script can then filter by tags to select the relevant features and/or scenarios. +To allow the test runner script to run the features and scenarios relevant to the system-under-test the feature file or individual scenarios are tagged. +The test runner script can then filter by tags to select the relevant features or scenarios. -For general information on tagging features and scenarios see http://behat.org/en/latest/user_guide/organizing.html#tags[the Behat tags documentation]. +TIP: For general information on tagging features and scenarios see {behat-tags-docs-url}[the Behat tags documentation]. === Tagging Features By API, CLI and webUI -Tag every feature with its major acceptance test type `api`, `cli` or `webUI`: +Tag every feature with its major acceptance test type `api`, `cli` or `webUI`, as in the following examples. +Doing so allows the tests of a particular major type to be quickly run or skipped. + +==== `@api` [source,gherkin] ---- @@ -507,6 +509,8 @@ Feature: add groups So that I can more easily manage access to resources by groups rather than individual users ---- +==== `@cli` + [source,gherkin] ---- @cli @@ -516,6 +520,8 @@ Feature: add group So that I can more easily manage access to resources by groups rather than individual users ---- +==== `@webUI` + [source,gherkin] ---- @webUI @@ -525,7 +531,6 @@ Feature: login users So that I have access to my files ---- -This allows the the tests of a particular major type to be easily run or skipped. === Tagging Scenarios That Require An App @@ -533,33 +538,34 @@ When a feature or scenario requires a core app to be enabled then tag it like: [source,gherkin] ---- - @comments-app-required - @federation-app-required - @files_trashbin-app-required - @files_versions-app-required - @notifications-app-required - @provisioning-app-required - @systemtags-app-required +@comments-app-required +@federation-app-required +@files_trashbin-app-required +@files_versions-app-required +@notifications-app-required +@provisioning-app-required +@systemtags-app-required ---- -The above apps might be disabled on a system-under-test. Tagging the feature or scenario allows -all tests for the app to be easily run or skipped. +The above apps might be disabled on a system-under-test. +Tagging the feature or scenario allows all tests for the app to be quickly run or skipped. -For tests in an app repository, do not tag them with the app name (e.g. `files_texteditor-app-required`). +For tests in an app repository, do not tag them with the app name (e.g., `files_texteditor-app-required`). It is already a given that the app in the repository is required for running the tests! === Tagging Scenarios That Need to Be Skipped ==== Skip UI Tests On A Particular Browser -Some browsers have difficulty with some automated test actions. To skip scenarios for a browser tag them with the relevant tags: +Some browsers have difficulty with some automated test actions. +To skip scenarios for a browser tag them with the relevant tags: [source,gherkin] ---- - @skipOnCHROME - @skipOnFIREFOX - @skipOnINTERNETEXPLORER - @skipOnMICROSOFTEDGE +@skipOnCHROME +@skipOnFIREFOX +@skipOnINTERNETEXPLORER +@skipOnMICROSOFTEDGE ---- === Skip Tests On A Particular Version Of ownCloud @@ -570,41 +576,60 @@ Use tag formats like the following to skip on a particular major, minor or patch [source,gherkin] ---- - @skipOnOcV10 - @skipOnOcV10.0 - @skipOnOcV10.0.10 - @skipOnOcV10.1 +@skipOnOcV10 +@skipOnOcV10.0 +@skipOnOcV10.0.10 +@skipOnOcV10.1 ---- === Skip Tests In Other Environments -`@skipOnLDAP` - skip the scenario if the test is running with the LDAP backend. For example, some user provisioning features may not be relevant when LDAP is the backend for authentication. - -`@skipOnStorage:ceph` - skip the scenario if the test is running with `ceph` backend storage. - -`@skipOnStorage:scality` - skip the scenario if the test is running with `scality` backend storage. - -`@skipOnEncryption` - skip the scenario if the test is running with encryption enabled. - -`@skipOnEncryptionType:masterkey` - skip the scenario if the test is running with `masterkey` encryption enabled. - -`@skipOnEncryptionType:user-keys` - skip the scenario if the test is running with `user-keys` encryption enabled. +[cols="20,80",options="header"] +|=== +|Annotation +|Description +|`@skipOnLDAP` +|skip the scenario if the test is running with the LDAP backend. For example, some user provisioning features may not be relevant when LDAP is the backend for authentication. +|`@skipOnStorage:ceph` +|skip the scenario if the test is running with `ceph` backend storage. +|`@skipOnStorage:scality` +|skip the scenario if the test is running with `scality` backend storage. +|`@skipOnEncryption` +|skip the scenario if the test is running with encryption enabled. +|`@skipOnEncryptionType:masterkey` +|skip the scenario if the test is running with `masterkey` encryption enabled. +|`@skipOnEncryptionType:user-keys` +|skip the scenario if the test is running with `user-keys` encryption enabled. +|=== == Tags For Tests To Run In Special Environments -`@smokeTest` - this scenario has been selected as part of a base set of smoke tests. - -`@TestAlsoOnExternalUserBackend` - this scenario is selected as part of a base set of tests to run when a special user backend is in place (e.g. LDAP). - -`@local_storage` - this scenario requires and tests the local storage feature. +[cols="25,75",options="header"] +|=== +|Annotation +|Description +|`@smokeTest` +|this scenario has been selected as part of a base set of smoke tests. +|`@TestAlsoOnExternalUserBackend` +|this scenario is selected as part of a base set of tests to run when a special user backend is in place (e.g., LDAP). +|`@local_storage` +|this scenario requires and tests the local storage feature. +|=== == Special Tags for UI Tests -`@insulated` - this makes the browser driver restart the browser session between each scenario. It helps isolate browser state. -When the browser session is recording there is a separate video for each scenario. Use this tag on all UI scenarios. - -`@disablePreviews` - generating previews/thumbnails takes time. Use this tag on UI test scenarios that do not need to test thumbnail behavior. - +[cols="25,75",options="header"] +|=== +|Annotation +|Description +|`@insulated` +|this makes the browser driver restart the browser session between each scenario. +It helps isolate the browser state. +When the browser session is recording, there is a separate video for each scenario. +Use this tag on all UI scenarios. +|`@disablePreviews` +|generating previews/thumbnails takes time. Use this tag on UI test scenarios that do not need to test thumbnail behavior. +|=== [[how-to-add-a-new-feature]] == How to Add New Test Steps