ExPlat: Add A/B experiment - #14286
Conversation
|
You can trigger optional UI/connected tests for these changes by visiting CircleCI here. |
|
You can test the changes on this Pull Request by downloading the APK here. |
There was a problem hiding this comment.
Hi @renanferrari 👋 ! I tested this as per test steps and all works as described 👍 ! I'm not familiar with the project or this specific experiment but AFAIU also code looks good. Checked the analytics tracking as well and it is consistent!
Side notes: placing a couple of notes I happened to think about while reviewing; probably not relevant to this specific PR so in case feel free to skip them 🙇
- I played a bit with the
ttlresponse parameter setting it to like 10 seconds, changing the endpoint response from having the variant or not having it and forcing a refresh putting the app in background and back again in foreground. The endpoint was correctly fetched with the new experiment configuration, but the behaviour in the menus list order was kept as the original one (only closing/opening back the app triggers a change in the list order as per your test steps). AFAIU I think this is a wanted behaviour given we are keeping the cached value inExPlat > activeVariations, but thought to better mention in case I'm missing anything. - I see we are not changing the order of the actions in the FAB inside the Posts section. I guess this may be also wanted to keep the experiment scope more
focused, but again I thought I better mention than not 😄
Not sure if you need other people in the list to look into it (especially on the data/experiment side) so I'm approving without merging right now; if this is not the case please just let me know or given you are targeting upcoming milestone feel free to merge if the side notes above are not relevant to this PR as I would expect 🙇
👋 @develric @renanferrari I'm planning to review this one tomorrow morning but as @develric said, given the plans to merge to this milestone please feel free to merge given @develric already gave it a pass. I went through the code briefly and it looks good to me 👍 . |
|
@develric Thanks a lot for the review! Both of your assumptions are correct: if the variation is active, the behavior shouldn't change and this experiment only affects the main FAB menu, not the one in the Posts section 👍 @mzorz Thank you for looking into this as well 🙂 I'm going to go ahead and merge this, since @aaronyan has also looked into it already. |
This PR is a follow up to #13960 and introduces a A/B experiment using our ExPlat client.
The experiment is pretty simple and consists of changing the position of the "Story post" action inside the menu that appears when the main FAB is tapped.
When the variation is
Control, the action is shown at its default position, at the bottom of the list. When the variation isTreatment, the action is shown at the top of the list:To test
Notes:
/wpcom/v2/experiments/0.1.0/assignments/wpandroid/.{ "variations": { "wpandroid_create_menu_story_first": "treatment" }, "ttl": 60 }Here's a video demonstrating this process:
story-first.mov
Here are my breakpoint settings on Charles:
Let me know if you need additional assistance with any of the steps above:
PR submission checklist:
RELEASE-NOTES.txtif necessary.