Skip to content

AB testing: implement the first A/B experiment - #16129

Merged
leandroalonso merged 12 commits into
developfrom
task/first_ab_test
Mar 19, 2021
Merged

AB testing: implement the first A/B experiment#16129
leandroalonso merged 12 commits into
developfrom
task/first_ab_test

Conversation

@leandroalonso

@leandroalonso leandroalonso commented Mar 18, 2021

Copy link
Copy Markdown
Contributor

Related PR: Automattic/Automattic-Tracks-iOS#174

After our successful A/A test (#15789) it's time for our first A/B test!

This PR implements the first A/B test within the FAB button (under My Site).

Also, it fixes a bug identified in the previous version (which was requesting the assignments endpoint even without any active tests)

Control Treatment
Simulator Screen Shot - iPhone 11 - 2021-03-18 at 16 22 06 Simulator Screen Shot - iPhone 11 - 2021-03-18 at 16 23 23

To test

  1. Open the app
  2. Tap the FAB, you should see the following order: "Blog post", "Site page" and "Story post"
  3. Close the app

Now you'll need to use a tool like Charles to intercept the proxy.

  1. Open the app again
  2. Intercept the assignments request and change wpios_create_menu_story_first to treatment
    1. Tap the FAB
  3. You should see the following order: "Story post", "Blog post" and "Site page"

You can see this test in the following video:

verifying_treatment.mov

Exposure event

The PR is still missing the exposure event, I'll add it before merging it.

@peril-wordpress-mobile

peril-wordpress-mobile Bot commented Mar 18, 2021

Copy link
Copy Markdown

You can trigger an installable build for these changes by visiting CircleCI here.

@leandroalonso

Copy link
Copy Markdown
Contributor Author

@aaronyan I hope that by looking at the video and the images you would be able to validate this PR. Also, let me know once we define the exposure event so I can update the code with it. :)

@peril-wordpress-mobile

peril-wordpress-mobile Bot commented Mar 18, 2021

Copy link
Copy Markdown

You can trigger optional UI/connected tests for these changes by visiting CircleCI here.

@momo-ozawa momo-ozawa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Comment on lines +19 to +20
let storyAction = actions.remove(at: 2)
actions.insert(storyAction, at: 0)

@momo-ozawa momo-ozawa Mar 19, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL there's a swapAt(::) method
https://developer.apple.com/documentation/swift/array/2893281-swapat

Totally up to you if you want to do remove/insert or swapAt 😀

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL too! Thanks! Changed in 3502df5

@leandroalonso
leandroalonso merged commit 9f12f6c into develop Mar 19, 2021
@leandroalonso
leandroalonso deleted the task/first_ab_test branch March 19, 2021 21:30
@mkevins mkevins mentioned this pull request Jan 28, 2022
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants