Evaluate Cypress for Web deliverables

[awentzel]

Cypress provides a comprehensive testing suite with CLI/App/Dashboard that allows testing functional UI components. The thought here is that with this we can wire up functional testing with ease easy to maintain and review issues with the team from the portal. Applitools, may also still be used on top to test Visual UI scenarios or sites.

1. Learn and implement a sample Cypress implementation and get feedback from Collaborators

Deliverables:
Include a remote branch with example implementation for packages that have a UI/UX.

1. Discuss with team the folder structure
2. Discuss with team the pros/cons
3. Discuss with team if there are other requirements we should consider.

[awentzel]

Folder Structure

Recommended default though, this does not take into account a mono-repository. Will determine the proposed structure once get into the code to try this out.

Pros (Cypress.io)

Cypress.io is a free, open-source front-end testing tool. Unlike the two previous frameworks, it’s a complete testing solution. It makes no claims of being a general test automation solution because its main focus is performing end-to-end testing of a web application’s user experience. In my research, there is no true end-to-end solution. Even if there was, there is an argument to be made that it's best to choose very specific frameworks, even if they add a dependency, because you can get best testing. Could have one thing for API testing and another for UX Testing.

Cypress.io’s main selling point is its blazing speed. It achieves this speed by running inside the browser. This means that it runs in the same execution loop as your application’s source code (as opposed to Puppeteer and Selenium, which run in a different process than the browser process that renders the application code). This means that Cypress.io doesn’t need to support the control protocols that are required for inter-process communication. This also resolves any system or network latency that slows down sending and receiving commands and responses.

1. Key Features (fast to iterate, easy to write, easy to debug, quick setup)
2. Why cypress
3. Key Differences
4. Trade offs
5. Best practices
6. Works well alongside our existing Jest, Enzyme, React environment
7. Supports Circle CI, Sauce Labs (coming first and soon)
8. Cross-browser support coming soon
9. TypeScript support here
10. Supports Jquery matchers

Cons

1. Current browser support not yet complete.No current cross-browser support, not planned. In fact, it is deprioritized for Edge/Safari/Firefox browsers refer.
2. No current support for vdiff, but, planned and proposal in progress by Applitools. Once available, we could drop Applitools.
3. Cypress.io’s method makes it harder to communicate with backend servers, which will mean that you’ll need to install additional modules.
4. Second, it doesn’t support testing multiple browser tabs.
5. It can only control a single browser instance at a time.
6. Visual regression testing not built in currently although applitools has a Cypress SDK that allows for this integration and their entire system is built around comprehensive visual regression testing or Percy

Resources

• https://medium.com/@wataruoguchi/running-e2e-tests-with-selenium-and-cypress-1de69b95d936
• https://medium.com/@shivambharadwaj/https-medium-com-shivambharadwaj-why-you-should-switch-to-cypress-for-modern-web-testing-5d3739a19e6
• https://www.slideshare.net/latinov/qa-challenge-accepted-cypress-vs-selenium
• https://pusher.com/sessions/meetup/js-monthly-london/testing-made-easy-better-faster-stronger-unit-tests-and-integration-tests

User scenarios

A good example of how Jest/Cypress can be used together for functional testing. (in this example he's functionally testing the login/logout user experience in real time during development). 15 minutes for that part.I don't actually think we have a need for functional testing currently. But, hopefully this will help explain a good use case for Cypress. https://frontendmasters.com/courses/testing-react/

Spec

• Circle CI configuration details for storing artifacts
• Examples

[awentzel]

Example Implementations
Cypress, React, Jest, Babel, Webpack

[awentzel]

Wiki

To install locally

$ npm install cypress --save-dev
$ npm install @cypress/webpack-preprocessor --save-dev

Configure tsconfig.json to include cypress type

  "compilerOptions": {
       "types": ["cypress"]
    }

Add new configuration file cyupress.json for cypress

{
  "projectId": "1csr87"
}

Create an .env file at root of package with the "record key". Ensure the .env is ignored by .gitignore.

CYPRESS_RECORD_KEY=<record key>

Update package.json scripts

"scripts": {
    "build:app": "webpack --progress",
    "integration-tests": "webpack --output-filename out.js --entry ./cypress/integration/*.ts",
    "integration-tests:open": "cypress open",
    "integration-tests:run": "node build/run-cypress.js",
}

Create a script file to execute both Webpack development servers. One for the App and one for Cypress testing.

const spawn = require('child_process').spawn
const webpack = require('webpack');
const webpackConfig = require('../webpack.config');
const WebpackDevServer = require('webpack-dev-server');
const envs = require('dotenv').config();
const cypress = require('cypress');

// Validate environment variables are secure and properly set
if(envs.error) {
  if(process.env.CYPRESS_RECORD_KEY.length == 0)
    throw envs.error;
  else
    console.log('Need to add CYPRESS_RECORD_KEY to CI');
}
console.log(envs.parsed);

// Start Webpack development server for the application as required by Cypress
function startServer() {
  return new Promise((resolve, reject) => {
    new WebpackDevServer(webpack(webpackConfig), {})
    .listen(7001, 'localhost', function(err) {
      resolve()
    }).on('error', (error) => {
      reject(error)
    })
  })
}

var server = startServer();

server.then(function() {

  setTimeout(() => {
    const cypress = spawn("cypress", ["run", "--record", "--key", process.env.CYPRESS_RECORD_KEY], {stdio: "inherit"});
    console.log(cypress.stdout);
  }, 10000);
})

Startup Web server for App and for Cypress and auto-run tests with dashboard integration
```bash
$ npm run integration-tests:run

Open Cypress while Web server is running for manual testing

$ npm run integration-tests:open

[awentzel]

Ready for review @chrisdholt, @nicholasrice , @janechu

[chrisdholt]

@awentzel Thanks! A few questions:

1. What determines the project Id? Is there a way that it can be more intuitive and project based?
2. Can you share some of the purpose for running two webpack dev servers simultaneously?
3. On the script definitions side of things, does build:test actually build the test or does it functionally run? I would suggest naming the npm task specific to what is happening. Typically when we use build it is packaging it up and not executing the file with webpack. Want to be sure I understand what is happening so we can be consistent.

[awentzel]

@chrisdholt

1. Project ID is created by the dashboard on Cypress upon initial sign-up, which I've already done.
2. The first Webpack dev server is to run the application (msft-docs, for example). The second is to run Cypress integration tests against that first server. It may be possible to combine, but, it depends on what we're planning on testing against. For example, we could test against development site or an already hosted production website.
3. build:test has been removed to bring inline with other tests. Code updated for package.json above.

[chrisdholt]

@awentzel I notice that Cypress stores video files in the repo locally. Is that correct? That would add a ton of weight to the repo itself. Can we store it outside the repo as an alternative? If we want the video functionality and the only way to have it is stored in the repo - I'm not sure we want to include it.

[awentzel]

@chrisdholt It does store when it runs locally during development testing run and so would be included in the source. It does delete all of these files at the start of execution so it's doesn't yet do visual regressing diff testing. When running on the CI it would be available and stored on the CI with the history of that job. While these tend to be large files, the way GitHub is architected this is not a huge perf cost when uploading and downloading. The benefits I believe would outweigh the cost. There is a feature to to turn off video feature for local testing and only use that in CI.

More details https://docs.cypress.io/guides/guides/screenshots-and-videos.html

[awentzel]

@janechu @chrisdholt @nicholasrice last chance to provide thoughts on this Proposal before it's approved. Thanks

[chrisdholt]

Don't we need approvals in order for this to be approved?