E2E Test Setup

[smeubank]

Idea

There is a gap in our tests, that we do not do a proper E2E test of our consumers using new releases of our SDK. E2E tests should catch those errors that we're unlikely to notice during normal development.

Requirements:

Examples of what an E2E test of an individual package should do:

• Upload current version of SDK to a local fake registry (this is to test tarball contents) (we could use verdaccio as registry)
• Download SDK package from the registry
• Build an example app using the downloaded SDK (this is to test typing and eventual build-time stuff we do, ie. webpack plugin)
• Run the example app
• Intercept any outgoing requests to sentry from that example app
• Do some stuff in the example app (cause errors, trigger transactions)
• Check payloads of intercepted requests and their content

Consider:

• This will have maintenance costs, changes in SDKs need to be reflected in our example apps

Prior Work:

• React Native: https://github.com/getsentry/sentry-react-native/blob/main/.github/workflows/e2e.yml
• Electron: https://github.com/getsentry/sentry-electron/blob/6ed4441d67e9002e3dcb7d5c2db263fa455d48a5/.github/workflows/build.yml#L71-L95

---

Things to discuss and figure out

• Do we intend these tests to be run locally? It would make the whole thing much less complicated running this only in CI because we don't have to worry about cleanup. The downside is that writing tests and adjusting the E2E test becomes a little bit more cumbersome.
  • Yes, it's a requirement to be run locally. It is definitely worth the additional effort.
• When do we run E2E tests? As part of CI or with a cron-job?
  • Let's start off running the E2E as part of every PR. If we notice that they slow us down, we demote the checks to only run on master pushes or even only before doing a release.
  • Cron jobs will only become relevant when we test canaries for major frameworks.
• Will E2E tests be a required check?
  • Discussed: Most likely yes. However, might turn into "no" depending on how fast the tests run.
• Will we run E2E tests on different kinds of OSes? (Checking filepaths, etc.)
  • Yes, we should.
• For step 2 (validating SDK behavior), would it make sense to actually send events to prod Sentry and query for the event IDs to verify the payloads?
  • Flakiness?
  • What if Sentry goes down? Bigger things to worry about.
  • What if Sentry simply has an implementation error?
  • What if the SDK has an error and doesn't return event IDs?
  • Upside: Constant stream of data flowing into Sentry for us to check out and verify things.
  • Decision: Let's send the events to prod Sentry. If we notice it's too flakey, we reevaluate.

Tasks

• Set up fake registry infrastructure (test(e2e): Set up fake registry for E2E tests #5806)
• Set up recipe-based test suites framework (test(e2e): Add E2E Test recipe framework #5836)
• Add first tests (for @sentry/react) that test the typical build-pipeline steps (test(e2e): Test building a create-react-app app with Sentry SDK #5848)
  • What to test?: Installing package, building the app (create-react-app) check typings (run tsc)
• Create guide in README on how to set up new tests

Next up

Verifying SDK behavior with E2E tests. Tracked in #5855

[smeubank]

I see these for electron all the time getsentry/sentry-electron#562

Could we achieve something similar her?

[AbhiPrasad]

I see these for electron all the time getsentry/sentry-electron#562

Could we achieve something similar her?

This would be nice to have when we look at Test multiple versions of major frameworks, which we set up as a stretch goal for now.

[smeubank]

Out of scope of this issue:

Discussion: this would be functionally different, new releases of react would be something like a canary release issue to test not block a release of our SDK