Skip to content

Fix screenshot test for Russian & Thai - #15099

Merged
rachelmcr merged 5 commits into
developfrom
fix/15074-test-login-failures
Oct 14, 2020
Merged

Fix screenshot test for Russian & Thai#15099
rachelmcr merged 5 commits into
developfrom
fix/15074-test-login-failures

Conversation

@rachelmcr

@rachelmcr rachelmcr commented Oct 14, 2020

Copy link
Copy Markdown
Contributor

Fixes #15074

As noted in the issue, for some reason typing text works in most text fields and languages except secure text entry fields in Russian and Thai (of the languages we run the test in).

That issue prevented the test from logging in when the device is set to those languages. To work around that, this PR checks the language code on the device and pastes (instead of typing) the password in those languages. I didn't switch to using a paste method for all languages because in repeated test runs I found pasting to be a bit flaky, so this limits it to the languages where it's really needed.

We also have a small mocks improvement in wordpress-mobile/WordPressMocks#21 so I've updated the Podfile here so we can test that change.

To test:

  • Run rake mocks.
  • Set your simulator to Russian and Thai and confirm the WordPressScreenshotGeneration test passes.
  • Confirm the WordPressScreenshotGeneration test continues to pass in another language (e.g. English).

(You can run the test at least once on an iPad Pro, 12.9 inch, 2nd or 3rd gen to test the mocks change.)

PR submission checklist:

  • I have considered adding unit tests where possible.
  • I have considered adding accessibility improvements for my changes.
  • I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

@rachelmcr rachelmcr self-assigned this Oct 14, 2020
@peril-wordpress-mobile

peril-wordpress-mobile Bot commented Oct 14, 2020

Copy link
Copy Markdown

You can trigger an installable build for these changes by visiting CircleCI here.

@rachelmcr

Copy link
Copy Markdown
Contributor Author

I see there's a Podfile.lock conflict; I'll fix that once the mocks PR is merged and I update the Podfile to use the published pod.

@peril-wordpress-mobile

peril-wordpress-mobile Bot commented Oct 14, 2020

Copy link
Copy Markdown

You can trigger optional UI/connected tests for these changes by visiting CircleCI here.

@jkmassel jkmassel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a note about a small change for your consideration.

I tested in English, Russian, and Thai.

English worked the first time; but for whatever reason Russian and Thai both failed the first time but worked on subsequent runs.

Additionally, it doesn't seem to work on iOS 14, just on 13.5.

I don't think either of these things are blockers – snapshot will auto-retry if there's a failure so worst-case the generation just takes slightly longer.

Comment thread WordPress/WordPressUITests/Utils/XCTest+Extensions.swift
@rachelmcr

Copy link
Copy Markdown
Contributor Author

Thanks for the review!

English worked the first time; but for whatever reason Russian and Thai both failed the first time but worked on subsequent runs.

Yeah, if it failed on the paste action I'm not too surprised because it was pretty flaky when I was trying it out. It's the most straightforward workaround I came up with for now but we can think about more stable options to try (maybe using language-specific text so we can type instead of paste).

Additionally, it doesn't seem to work on iOS 14, just on 13.5.

Good point. We haven't made any changes for tests to run on iOS 14 but it's on my radar. Agreed that it isn't a blocker for this but I'll see if I can follow up on it soon.

I made the changes mentioned above (the suggested change to the paste implementation and updated the WordPressMocks pod) so I'll go ahead and merge this PR once tests are passing.

@rachelmcr
rachelmcr merged commit 2f7b687 into develop Oct 14, 2020
@rachelmcr
rachelmcr deleted the fix/15074-test-login-failures branch October 14, 2020 17:32
@peril-wordpress-mobile

Copy link
Copy Markdown
Messages
📖 This PR has the 'Releases' label: some checks will be skipped.

Generated by 🚫 dangerJS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Testing: UI tests fail with device set to certain languages

2 participants