Skip to content

TESTING.md

Latest commit

 

History

History
106 lines (65 loc) · 4.18 KB

TESTING.md

File metadata and controls

106 lines (65 loc) · 4.18 KB

UI/UX Testing

We have a lot of test cases we need to automate. While we work on getting those made we maintain this list of known cases you might want to test. You do not need to test all of these for every PR - use your best judgement.

This is not an exhaustive list by any means. As you find weird edge cases please add them here.

Unit Testing

We have some unit tests but not much. Usually we prefer to just unit test helper/utils that have a more simple input <-> output format and are easy to test.

E2E Testing

Our E2E tests are currently in the playwright-tests directory. The goal is eventually this becomes a stand alone project within the Estuary organization. As these tests are trully testing the entire application and not just the UI.

Connectors to test

You will want to test a solid combination of different connectors that all can trigger different edges cases. You can also fake these locally my altering the Endpoint and Resource Configs. However, this will make saving them not work.

Both

  1. requires no endpoint config (???)
  2. allows oAuth for authentication (Google Sheets)
  3. uses a descriminator for a anyOf of oneOf (???)

Captures

Bindings

  1. returns a single binding (HelloWorld)
  2. returns several bindings (Postgres)
  3. returns inferred schema (WebHook)
  4. returns TONS of bindings to check performance (Postgres w/ table that has +1k tables)

Materializations

Source settings

  1. supports deltaUpdates only (???)
  2. supports targetSchema only (???)
  3. supports both deltaUpdates targetSchema (???)
  4. supports neither deltaUpdates targetSchema (Amazon S3 Parquet)

Create Scenarios

During create you will need to worry about when you are editing with just local state stored in the browser vs when you are actually updating the draft spec. Normally you are updating the draft if there is a Test and Save cta visible in the header.

Both

  1. happy path save
  2. edit collection schema (just schema and read/write ones)

Capture

Captures are slightly more straight forward as they do not have a draft until things are discovered. So you do not need to worry as much about testing these and then clicking next like Materializations.

  1. edit collection keys
  2. edit and then re-discover
  3. failed discover
  4. rename field on collection

Materialization

Materializations have a draft right away. However, we do not auto generate the spec so there is still a next cta showing like when you enter Capture.

You will want to test all these before AND after clicking next cta.

  1. happy path save with materialize cta
  2. happy path save but WITHOUT bindings
  3. refresh field select
  4. set field select groupBy
  5. setting source.capture
  6. adding bindings
  7. setting source defaults and adding bindings
  8. removing bindings
  9. happy path save with old collection using trial storage

Edit Scenarios

You will want to do all the same testing as you do on Create - but just in the edit flow. During edit the user can also reload at anytime - so make sure you test with reloading at different times and ensuring your changes are still there and things are hydrating correctly

Both

  1. happy path edit and save
  2. edit endpoint config
  3. backfill a single binding
  4. backfill all bindings

Capture

Materialization

Special Cases

There are some connectors that require special handling and have been known to break our assumptions.

If you are adding this kind of handling ensure you add a comment with some details on why it is added so we can more easily find them in the future.

Pydantic

Pydantic is a Python tool that is used by the Connectors team. It tends to output schemas that are technically correct but written in a manner that seems a bit "off" from how a person might write them. The main thing that has cause bugs before is how Pydantic handles oneOf, anyOf, and discriminator.

These connectors can be found here and should always be included in your testing.

You can also find a lot of the special handling by just seraching for Pydantic in the code.