A/B test Land In the Editor

[antonis]

Description

This PR adds an A/B experiment that opens the home page for editing after the last step in site creation.

This flow is conditioned on an ExPlat experiment named "wpandroid_land_in_the_editor_phase1_v3" and should leave the current behavior intact for the control variation.

To test

Note: To change the experiment variation between Control(default) and Treatment(experiment):

1. Browse to the Abacus experiment wpandroid_land_in_the_editor_phase1_v3 (internal ref: pbxNRc-1fJ-p2)
2. Follow the instructions on the Bookmarklet tooltip to set the Control or Treatment variation for your user.

Alternatively you may use a proxy (e.g. Charlie) to intercept the call to the experiments endpoint (/wpcom/v2/experiments/) and return the desired variation (detailed instructions in this previous experiment). E.g.

{
	"variations": {
		"wpandroid_land_in_the_editor_phase1_v3": "treatment"
	},
	"ttl": 60
}

Control: Login or New user

1. Login to the app with a user that has the Control variation assigned (check note above)
2. Press the Create new site option at the bottom
3. Select a design and a domain to create a site
4. Press OK on the next screen
5. Verify that you are presented with the Want a little help to manage your site dialog
6. Verify that you land on the My Site screen after your selection

Video

login-control.mp4

Treatment: Login or New user

1. Login to the app with a user that has the Treatment variation assigned (check note above)
2. Press the Create new site option at the bottom
3. Select a design and a domain to create a site
4. Press OK on the next screen
5. Verify that you are presented with the home page editor
6. Verify that the landing_editor_shown analytics event is emitted
7. Press the Continue button and save any changes
8. Verify that you are presented with the Want a little help to manage your site dialog
9. Verify that you land on the My Site screen after your selection

Video

login-treat.mp4

Control: Existing or Logged in user

1. Login to the app with a user that has the Control variation assigned (check note above)
2. On the My Site screen press the site picker selector
3. Press the Plus(+) button at the top-right
4. Choose the Create a WordPress.com site option
5. Select a design and a domain to create a site
6. Press OK on the next screen
7. Verify that you are presented with the Want a little help to manage your site dialog
8. Verify that you land on the My Site screen after your selection

Video

existing-control.mp4

Treatment: Existing or Logged in user

1. Login to the app with a user that has the Treatment variation assigned (check note above)
2. On the My Site screen press the site picker selector
3. Press the Plus(+) button at the top-right
4. Choose the Create a WordPress.com site option
5. Select a design and a domain to create a site
6. Press OK on the next screen
7. Verify that you are presented with the home page editor
8. Verify that the landing_editor_shown analytics event is emitted
9. Press the Continue button and save any changes
10. Verify that you are presented with the Want a little help to manage your site dialog
11. Verify that you land on the My Site screen after your selection

Video

existing-treat.mp4

Regression Notes

1. Potential unintended areas of impact
   N/A

2. What I did to test those areas of impact (or what existing automated tests I relied on)
   Relied on existing tests

3. What automated tests I added (or what prevented me from doing so)
   Added tests in MySiteViewModelTest, EditorActionsProviderTest and GetButtonUiStateUseCaseTest

PR submission checklist:

• I have completed the Regression Notes.
• I have considered adding accessibility improvements for my changes.
• I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

[peril-wordpress-mobile]

You can trigger optional UI/connected tests for these changes by visiting CircleCI here.

[peril-wordpress-mobile]

You can test the changes on this Pull Request by downloading the APKs:

• Jetpack-pr-15627-build-118427.apk

• WordPress-pr-15627-build-118427.apk

[mkevins]

Curious why we are checking this twice?

[antonis]

Good catch @mkevins 👍
The call hierarchy was initially different and this made sense but now it is not needed.
Fixed with 96e08c4

[mkevins]

The code looks good.

I tested each user path in the description on a Pixel 3a (physical device) using the staging bookmarklet tool to switch between control and treatment groups for my user account. Each case works as described.

Also, I ran all the additional tests locally (thanks for adding the extra tests 😄 ) and everything is working as expected as well.

Nice work on this Antonis! 👍