Skip to content

AB testing: implements an A/A experiment - #15789

Merged
leandroalonso merged 14 commits into
developfrom
task/explat_a_a_experiment_test
Feb 15, 2021
Merged

AB testing: implements an A/A experiment#15789
leandroalonso merged 14 commits into
developfrom
task/explat_a_a_experiment_test

Conversation

@leandroalonso

@leandroalonso leandroalonso commented Feb 4, 2021

Copy link
Copy Markdown
Contributor

Ready for review after #15770 is merged

Automattic Tracks: Automattic/Automattic-Tracks-iOS#172

This PR implements the A/A experiment to validate that our ExPlat implementation is not biased.

To test

  1. Add a breakpoint in ExPlatService.swift:70
  2. Add a breakpoint in WPTabBarController+Swift.swift:89
  3. Run the app
  4. The first breakpoint in ExPlatService should be hit
  5. Tap the Reader tab
  6. The second breakpoint should be hit and the property variation should be control

Closing and reopening the app

  1. Stop the app
  2. Run it again
  3. The first breakpoint in ExPlatService should be hit

Coming back from the background

  1. Put the app in the background
  2. Put the app in the foreground
  3. The first breakpoint in ExPlatService should not be hit

PR submission checklist:

  • I have considered adding unit tests where possible.
  • I have considered adding accessibility improvements for my changes.
  • I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

@peril-wordpress-mobile

peril-wordpress-mobile Bot commented Feb 4, 2021

Copy link
Copy Markdown

You can trigger an installable build for these changes by visiting CircleCI here.

@peril-wordpress-mobile

peril-wordpress-mobile Bot commented Feb 4, 2021

Copy link
Copy Markdown

You can trigger optional UI/connected tests for these changes by visiting CircleCI here.

@leandroalonso leandroalonso changed the title AB testing: implements the A/A experiment AB testing: implements an A/A experiment Feb 4, 2021
@leandroalonso

Copy link
Copy Markdown
Contributor Author

@emilylaguna can I bother you to review this one?

@leandroalonso

Copy link
Copy Markdown
Contributor Author

@renanferrari JFYI, here's the iOS PR for the A/A experiment.

@aaronyan aaronyan left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does everyone that open an app get assigned an experiment variation? If so, that's okay, I just want to understand how this works. :)

Comment thread WordPress/Classes/ViewRelated/System/WPTabBarController+Swift.swift Outdated
@leandroalonso

Copy link
Copy Markdown
Contributor Author

@aaronyan It works like this:

  • If there's any active experiment (aka an enum added here), the request is made
  • The returned experiments values are saved
  • When the variation for an experiment is consulted, we either return "treatment" (if the API returned treatment) or "control" (in any other case: missing value, null, etc - as per suggested here)

@leandroalonso
leandroalonso removed the request for review from emilylaguna February 8, 2021 19:27
@jkmassel

jkmassel commented Feb 8, 2021

Copy link
Copy Markdown
Contributor

👋 We're freezing 16.7 today, so this PR is being bumped to 16.8. If you need this to be part of the 16.7 release, please merge it into the release/16.7 branch and DM me – I'll be happy to cut a new beta release!

@jkmassel jkmassel modified the milestones: 16.7, 16.8 Feb 8, 2021
@aaronyan

Copy link
Copy Markdown
  • If there's any active experiment (aka an enum added here), the request is made
  • The returned experiments values are saved
  • When the variation for an experiment is consulted, we either return "treatment" (if the API returned treatment) or "control" (in any other case: missing value, null, etc - as per suggested here)

This looks great!

@emilylaguna emilylaguna left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Works as described.

@leandroalonso

Copy link
Copy Markdown
Contributor Author

I tested the scenarios for the A/A testing as described in pbxNRc-Fx-p2#comment-1656

@leandroalonso
leandroalonso merged commit 37ec0d8 into develop Feb 15, 2021
@leandroalonso
leandroalonso deleted the task/explat_a_a_experiment_test branch February 15, 2021 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants