Skip to content

ExPlat: Add A/B experiment - #14286

Merged
renanferrari merged 7 commits into
developfrom
feature/add-explat-ab-experiment
Mar 21, 2021
Merged

ExPlat: Add A/B experiment#14286
renanferrari merged 7 commits into
developfrom
feature/add-explat-ab-experiment

Conversation

@renanferrari

@renanferrari renanferrari commented Mar 19, 2021

Copy link
Copy Markdown
Contributor

This PR is a follow up to #13960 and introduces a A/B experiment using our ExPlat client.

The experiment is pretty simple and consists of changing the position of the "Story post" action inside the menu that appears when the main FAB is tapped.

When the variation is Control, the action is shown at its default position, at the bottom of the list. When the variation is Treatment, the action is shown at the top of the list:

Control Treatment
story-last story-first

To test

Notes:

  • You'll need a tool like Charles to intercept the ExPlat endpoint response.
  • The ExPlat endpoint looks like this: /wpcom/v2/experiments/0.1.0/assignments/wpandroid/.
  1. Open the app.
  2. On Charles, make sure you see the normal response from the ExPlat endpoint.
  3. On the My Site screen, tap the FAB.
  4. Notice that the menu actions have the following order: "Blog post", "Site page" and "Story post".
  5. Close the app and make sure to stop it.
  6. Open the app again.
  7. On Charles, intercept and edit the response so it looks like this:
{
	"variations": {
		"wpandroid_create_menu_story_first": "treatment"
	},
	"ttl": 60
}
  1. On the My Site screen, tap the FAB.
  2. Notice that the menu actions have the following order: "Story post", "Blog post", "Site page".

Here's a video demonstrating this process:

story-first.mov

Here are my breakpoint settings on Charles:

breakpoint-settings-1 breakpoint-settings-2

Let me know if you need additional assistance with any of the steps above:


PR submission checklist:

  • I have considered adding unit tests where possible.
  • I have considered adding accessibility improvements for my changes.
  • I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

@peril-wordpress-mobile

peril-wordpress-mobile Bot commented Mar 19, 2021

Copy link
Copy Markdown

You can trigger optional UI/connected tests for these changes by visiting CircleCI here.

@peril-wordpress-mobile

peril-wordpress-mobile Bot commented Mar 19, 2021

Copy link
Copy Markdown

You can test the changes on this Pull Request by downloading the APK here.

@develric develric left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @renanferrari 👋 ! I tested this as per test steps and all works as described 👍 ! I'm not familiar with the project or this specific experiment but AFAIU also code looks good. Checked the analytics tracking as well and it is consistent!

Side notes: placing a couple of notes I happened to think about while reviewing; probably not relevant to this specific PR so in case feel free to skip them 🙇

  1. I played a bit with the ttl response parameter setting it to like 10 seconds, changing the endpoint response from having the variant or not having it and forcing a refresh putting the app in background and back again in foreground. The endpoint was correctly fetched with the new experiment configuration, but the behaviour in the menus list order was kept as the original one (only closing/opening back the app triggers a change in the list order as per your test steps). AFAIU I think this is a wanted behaviour given we are keeping the cached value in ExPlat > activeVariations, but thought to better mention in case I'm missing anything.
  2. I see we are not changing the order of the actions in the FAB inside the Posts section. I guess this may be also wanted to keep the experiment scope more focused, but again I thought I better mention than not 😄

Not sure if you need other people in the list to look into it (especially on the data/experiment side) so I'm approving without merging right now; if this is not the case please just let me know or given you are targeting upcoming milestone feel free to merge if the side notes above are not relevant to this PR as I would expect 🙇

@mzorz

mzorz commented Mar 21, 2021

Copy link
Copy Markdown
Contributor

Not sure if you need other people in the list to look into it (especially on the data/experiment side) so I'm approving without merging right now; if this is not the case please just let me know or given you are targeting upcoming milestone feel free to merge if the side notes above are not relevant to this PR as I would expect

👋 @develric @renanferrari I'm planning to review this one tomorrow morning but as @develric said, given the plans to merge to this milestone please feel free to merge given @develric already gave it a pass. I went through the code briefly and it looks good to me 👍 .

@renanferrari

Copy link
Copy Markdown
Contributor Author

@develric Thanks a lot for the review! Both of your assumptions are correct: if the variation is active, the behavior shouldn't change and this experiment only affects the main FAB menu, not the one in the Posts section 👍

@mzorz Thank you for looking into this as well 🙂

I'm going to go ahead and merge this, since @aaronyan has also looked into it already.

@renanferrari
renanferrari merged commit 018f674 into develop Mar 21, 2021
@renanferrari
renanferrari deleted the feature/add-explat-ab-experiment branch March 21, 2021 13:10
@renanferrari renanferrari mentioned this pull request Apr 20, 2021
3 tasks
@antonis antonis mentioned this pull request Jan 20, 2022
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants