Skip to content

Run Xcode 15 in CI and skip or disable tests that fail #21921

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mokagio wants to merge 47 commits into from
Closed

Run Xcode 15 in CI and skip or disable tests that fail #21921

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
4fae8c7
Use Xcode 15.0.1 in CI
mokagio Oct 31, 2023
e5ea649
Update Fastlane to version 2.217.0 to get Xcode 15 trainer fix
mokagio Nov 19, 2023
d9ca84f
Update `TimeZoneFormatterTests` to pass on Xcode 15
mokagio Oct 12, 2023
0404e96
Remove a useless reassignment in `WKCookieJarTests`
mokagio Oct 31, 2023
0eb5fca
Mark `WKCookieJarTests` as expected to fail
mokagio Oct 31, 2023
b57be1b
Remove redundant custom test assertion, which was also out of sync
mokagio Oct 31, 2023
615bffb
Skip two tests failing because of a changed `WKWebView` behavior
mokagio Oct 31, 2023
16e9b7a
[UI Tests] Replaced "4 AM" with "4 AM".
pachlava Nov 20, 2023
53ea327
[Temp Experiment] See if turning off parallelization helps with UI te…
pachlava Nov 21, 2023
244da46
Revert "[Temp Experiment] See if turning off parallelization helps wi…
pachlava Nov 21, 2023
380d23e
Use Xcode 15.0.1 in CI
mokagio Oct 31, 2023
9c9683b
Update Fastlane to version 2.217.0 to get Xcode 15 trainer fix
mokagio Nov 19, 2023
874131c
Update `TimeZoneFormatterTests` to pass on Xcode 15
mokagio Oct 12, 2023
aae0381
Remove a useless reassignment in `WKCookieJarTests`
mokagio Oct 31, 2023
8f80dff
Mark `WKCookieJarTests` as expected to fail
mokagio Oct 31, 2023
0dbbf5d
Remove redundant custom test assertion, which was also out of sync
mokagio Oct 31, 2023
dc63002
Skip two tests failing because of a changed `WKWebView` behavior
mokagio Oct 31, 2023
ecd712d
[UI Tests] Replaced "4 AM" with "4 AM".
pachlava Nov 20, 2023
5a68855
[Temp Experiment] See if turning off parallelization helps with UI te…
pachlava Nov 21, 2023
0d4c1f7
Revert "[Temp Experiment] See if turning off parallelization helps wi…
pachlava Nov 21, 2023
a20e69b
Remove git-cache plugin to run on Apple Silicon CI
mokagio Nov 22, 2023
23b99e9
Run UI tests on "flagship" Simulators for iOS 17
mokagio Nov 22, 2023
689b775
Run UI tests on iPhone 15
mokagio Nov 23, 2023
7e2042f
Revert "Revert "[Temp Experiment] See if turning off parallelization …
mokagio Nov 23, 2023
5ee54f6
Use UTC date formatter in `BloggingPromptsServiceTests`
mokagio Nov 23, 2023
b6f2e54
Always track screen captures in UI tests
mokagio Nov 23, 2023
b9ee167
Disable parallel testing in Fastlane, too
mokagio Nov 23, 2023
4b38795
Set unit tests to run in English and with region US
mokagio Nov 23, 2023
0648c0b
Merge branch 'mokagio/fix-code-15-tests' of https://github.com/wordpr…
pachlava Nov 23, 2023
d6ae539
Wait for comment text to load for longer.
pachlava Nov 23, 2023
0f32cb9
[UI Tests] On CI it takes even longer than 10s to load the comment.
pachlava Nov 23, 2023
655a424
[UI Tests] Changed an element locator.
pachlava Nov 26, 2023
692dace
Merge 'origin/trunk' into mokagio/fix-code-15-tests
mokagio Dec 14, 2023
9f744d0
Use Xcode 15.1
mokagio Dec 14, 2023
c233e83
Fix indentation in `TestObserver`
mokagio Dec 15, 2023
779b70d
Merge branch 'trunk' into mokagio/fix-code-15-tests
jostnes Dec 15, 2023
b328aec
Add workaround for iOS 17.2 Simulator breaking passwords settings
mokagio Dec 15, 2023
51b50aa
temp remove TestObserver
jostnes Dec 15, 2023
f7ae151
temp remove auto retries
jostnes Dec 15, 2023
4e0ef37
added workarounds around login
jostnes Dec 15, 2023
ebff3e7
Merge branch 'mokagio/fix-code-15-tests' of https://github.com/wordpr…
jostnes Dec 15, 2023
85f8c5d
decrease timeout
jostnes Dec 18, 2023
fd5c800
remove unnecessary checks
jostnes Dec 18, 2023
67e3c73
fix logic for tapUntilCondition
jostnes Dec 18, 2023
257e78f
turn retry back on
jostnes Dec 18, 2023
d8ec5d2
fix stats test by making year dynamic
jostnes Dec 18, 2023
4c3ca35
add retry to dismiss save password prompt
jostnes Dec 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .buildkite/cache-builder.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ common_params:
# Common environment values to use with the `env` key.
- &common_env
# Be sure to also update the `.xcode-version` file when updating the Xcode image/version here
IMAGE_ID: xcode-14.3.1
IMAGE_ID: xcode-15.1

steps:

Expand Down
9 changes: 3 additions & 6 deletions .buildkite/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,10 @@ common_params:
# Common plugin settings to use with the `plugins` key.
- &common_plugins
- automattic/a8c-ci-toolkit#2.18.1
- automattic/git-s3-cache#1.1.4:
bucket: "a8c-repo-mirrors"
repo: "automattic/wordpress-ios/"
# Common environment values to use with the `env` key.
- &common_env
# Be sure to also update the `.xcode-version` file when updating the Xcode image/version here
IMAGE_ID: xcode-14.3.1
IMAGE_ID: xcode-15.1

# This is the default pipeline – it will build and test the app
steps:
Expand Down Expand Up @@ -80,7 +77,7 @@ steps:
- group: "🔬 UI Tests"
steps:
- label: "🔬 :jetpack: UI Tests (iPhone)"
command: .buildkite/commands/run-ui-tests.sh 'iPhone SE (3rd generation)'
command: .buildkite/commands/run-ui-tests.sh 'iPhone 15'
depends_on: "build_jetpack"
env: *common_env
plugins: *common_plugins
Expand All @@ -92,7 +89,7 @@ steps:
context: "UI Tests (iPhone)"

- label: "🔬 :jetpack: UI Tests (iPad)"
command: .buildkite/commands/run-ui-tests.sh 'iPad Air (5th generation)'
command: .buildkite/commands/run-ui-tests.sh 'iPad Pro (12.9-inch) (6th generation)'
depends_on: "build_jetpack"
env: *common_env
plugins: *common_plugins
Expand Down
2 changes: 1 addition & 1 deletion .buildkite/release-builds.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ common_params:
# Common environment values to use with the `env` key.
- &common_env
# Be sure to also update the `.xcode-version` file when updating the Xcode image/version here
IMAGE_ID: xcode-14.3.1
IMAGE_ID: xcode-15.1

steps:

Expand Down
2 changes: 1 addition & 1 deletion .xcode-version
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
14.3.1
15.1
81 changes: 9 additions & 72 deletions API-Mocks/WordPressMocks/src/main/assets/mocks/mappings/wpcom/stats/stats_visits-year.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
"response": {
"status": 200,
"jsonBody": {
"date": "2019-07-16",
"date": "{{now format='yyyy-MM-dd'}}",
"unit": "year",
"fields": [
"period",
Expand All @@ -33,7 +33,7 @@
],
"data": [
[
"2005-01-01",
"{{now offset='-7 years' format='yyyy-MM-dd'}}",
0,
0,
0,
Expand All @@ -42,7 +42,7 @@
0
],
[
"2006-01-01",
"{{now offset='-6 years' format='yyyy-MM-dd'}}",
0,
0,
0,
Expand All @@ -51,7 +51,7 @@
0
],
[
"2007-01-01",
"{{now offset='-5 years' format='yyyy-MM-dd'}}",
0,
0,
0,
Expand All @@ -60,7 +60,7 @@
0
],
[
"2008-01-01",
"{{now offset='-4 years' format='yyyy-MM-dd'}}",
0,
0,
0,
Expand All @@ -69,70 +69,7 @@
0
],
[
"2009-01-01",
0,
0,
0,
0,
0,
0
],
[
"2010-01-01",
0,
0,
0,
0,
0,
0
],
[
"2011-01-01",
0,
0,
0,
0,
0,
0
],
[
"2012-01-01",
0,
0,
0,
0,
0,
0
],
[
"2013-01-01",
0,
0,
0,
0,
0,
0
],
[
"2014-01-01",
0,
0,
0,
0,
0,
0
],
[
"2015-01-01",
0,
0,
0,
0,
0,
0
],
[
"2016-01-01",
"{{now offset='-3 years' format='yyyy-MM-dd'}}",
48,
12,
0,
Expand All @@ -141,7 +78,7 @@
13
],
[
"2017-01-01",
"{{now offset='-2 years' format='yyyy-MM-dd'}}",
788,
465,
0,
Expand All @@ -150,7 +87,7 @@
3
],
[
"2018-01-01",
"{{now offset='-1 years' format='yyyy-MM-dd'}}",
1215,
632,
0,
Expand All @@ -159,7 +96,7 @@
3
],
[
"2019-01-01",
"{{now format='yyyy-MM-dd'}}",
9148,
4216,
1351,
Expand Down
3 changes: 2 additions & 1 deletion Gemfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ gem 'cocoapods', '~> 1.14'
gem 'commonmarker'
gem 'danger-dangermattic', git: 'https://github.com/Automattic/dangermattic'
gem 'dotenv'
gem 'fastlane', '~> 2.216'
# 2.217.0 includes a fix for Xcode 15 test results parsing in CI
gem 'fastlane', '~> 2.217'
gem 'fastlane-plugin-appcenter', '~> 2.1'
gem 'fastlane-plugin-sentry'
# This comment avoids typing to switch to a development version for testing.
Expand Down
2 changes: 1 addition & 1 deletion Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -405,7 +405,7 @@ DEPENDENCIES
commonmarker
danger-dangermattic!
dotenv
fastlane (~> 2.216)
fastlane (~> 2.217)
fastlane-plugin-appcenter (~> 2.1)
fastlane-plugin-sentry
fastlane-plugin-wpmreleasetoolkit (~> 9.1)
Expand Down
2 changes: 0 additions & 2 deletions WordPress/UITests/JetpackUITests-Info.plist
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,5 @@
<string>????</string>
<key>CFBundleVersion</key>
<string>1</string>
<key>NSPrincipalClass</key>
<string>${PRODUCT_NAME}.TestObserver</string>

@mokagio mokagio Dec 15, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jostnes I'd recommend keeping this setup and skipping the executeWithRetries(disableAutoFillPasswords) in testBundleWillStart from TestObserver.

I'm saying this because it's easier to find the TestObserver setup in the code than in this Info.plist and hopefully we'll be able to go back to that setup when Apple fixes their Simulator bug (assuming my theory about it being a simulator bug is correct)

@jostnes jostnes Dec 18, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup, this is a temporary removal to save testing time and skip the part where the test tries to turn it off and fails every time because of the simulator bug. this will be re-added before merging.

</dict>
</plist>
1 change: 0 additions & 1 deletion WordPress/UITests/JetpackUITests.xctestplan
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@
},
"testTargets" : [
{
"parallelizable" : true,
"skippedTests" : [
"EditorAztecTests",
"LoginTests\/testEmailMagicLinkLogin()",
Expand Down
1 change: 0 additions & 1 deletion WordPress/UITests/Tests/LoginTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ class LoginTests: XCTestCase {
siteUrl: WPUITestCredentials.testWPcomPaidSite
)
.continueWithSelectedSite()
.dismissNotificationAlertIfNeeded()
try TabNavComponent()
.goToMeScreen()
.logoutToPrologue()
Expand Down
59 changes: 30 additions & 29 deletions WordPress/UITests/Tests/StatsTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,42 +22,43 @@ class StatsTests: XCTestCase {
takeScreenshotOfFailedTest()
}

let insightsStats: [String] = [
"Your views in the last 7-days are -9 (-82%) lower than the previous 7-days. ",
"Thursday",
"34% of views",
"Best Hour",
"4 AM",
"25% of views"
]

let yearsStats: [String] = [
"9,148",
"+7,933 (653%)",
"United States, 60",
"Canada, 44",
"Germany, 15",
"France, 14",
"United Kingdom, 12",
"India, 121"
]

let yearsChartBars: [String] = [
"Views, 2019: 9148",
"Visitors, 2019: 4216",
"Views, 2018: 1215",
"Visitors, 2018: 632",
"Views, 2017: 788",
"Visitors, 2017: 465"
]

func testInsightsStatsLoadProperly() throws {
let insightsStats: [String] = [
"Your views in the last 7-days are -9 (-82%) lower than the previous 7-days. ",
"Thursday",
"34% of views",
"Best Hour",
"4 AM",
"25% of views"
]

try StatsScreen()
.switchTo(mode: "insights")
.assertStatsAreLoaded(insightsStats)
}

func testYearsStatsLoadProperly() throws {
let yearsStats: [String] = [
"9,148",
"+7,933 (653%)",
"United States, 60",
"Canada, 44",
"Germany, 15",
"France, 14",
"United Kingdom, 12",
"India, 121"
]

let currentYear = Calendar.current.component(.year, from: Date())
let yearsChartBars: [String] = [
"Views, \(currentYear): 9148",
"Visitors, \(currentYear): 4216",
"Views, \(currentYear - 1): 1215",
"Visitors, \(currentYear - 1): 632",
"Views, \(currentYear - 2): 788",
"Visitors, \(currentYear - 2): 465"
]

try StatsScreen()
.switchTo(mode: "years")
.assertStatsAreLoaded(yearsStats)
Expand Down
21 changes: 15 additions & 6 deletions WordPress/UITestsFoundation/Globals.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,20 +62,29 @@ public func waitAndTap( _ element: XCUIElement, maxRetries: Int = 20) {
}
}

public func tapUntilCondition(element: XCUIElement, condition: Bool, description: String, maxRetries: Int = 10) {
public func waitForElementAndTap(_ tapElement: XCUIElement, untilConditionOn conditionElement: XCUIElement, condition: String, errorMessage: String, maxRetries: Int = 10) {

var retries = 0
while retries < maxRetries {
if !condition {
element.tap()
break
tapElement.tap()

switch condition {
case "exists":
if conditionElement.exists { return }
case "dismissed":
if !conditionElement.isHittable { return }
case "selected":
if conditionElement.isSelected { return }
default:
XCTFail("\(condition) is invalid! Please choose 'exists', 'dismissed' or 'selected'")
}

usleep(500000) // a 0.5 second delay before retrying
sleep(1) // a 1 second delay before retrying
retries += 1
}

if retries == maxRetries {
XCTFail("Condition \(description) still not met after \(maxRetries) tries.")
XCTFail("\(errorMessage) after \(maxRetries) tries.")
}
}

Expand Down
2 changes: 1 addition & 1 deletion WordPress/UITestsFoundation/Screens/Editor/EditorPostSettings.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ public class EditorPostSettings: ScreenObject {
// To ensure that the day tap happens on the correct month
let nextMonth = monthLabel.value as! String
if nextMonth != currentMonth {
tapUntilCondition(element: firstCalendarDayButton, condition: firstCalendarDayButton.isSelected, description: "First Day button selected")
waitForElementAndTap(firstCalendarDayButton, untilConditionOn: firstCalendarDayButton, condition: "selected", errorMessage: "First Day button not selected!")
}

doneButton.tap()
Expand Down
6 changes: 4 additions & 2 deletions WordPress/UITestsFoundation/Screens/Login/LoginUsernamePasswordScreen.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,13 +40,11 @@ public class LoginUsernamePasswordScreen: ScreenObject {

public func proceedWithSelfHostedSiteAddedFromSitesList(username: String, password: String) throws -> MySitesScreen {
fill(username: username, password: password)

return try MySitesScreen()
}

public func proceedWithSelfHosted(username: String, password: String) throws -> MySiteScreen {
fill(username: username, password: password)

return try MySiteScreen()
}

Expand All @@ -66,6 +64,10 @@ public class LoginUsernamePasswordScreen: ScreenObject {
passwordTextField.typeText(password)
}
nextButton.tap()

if #available(iOS 17.2, *) {
app.dismissSavePasswordPrompt()
}
}

private func dismissQuickStartPromptIfNeeded() throws {
Expand Down
Loading