A/B test landing in the editor

[mkevins]

Related PR:

wordpress-mobile/WordPress-iOS-Shared#306

Description

This PR adds a custom GutenbergViewController for editing the home page. This is used after the last step in site creation to bring the user directly into the editor after the new site is launched. It provides a modified UI (with a custom navigation bar), and a completion callback to launch quickstart when editing is finished.

This flow is conditioned on an ExPlat experiment named "wpios_land_in_the_editor_phase1_v3", and should leave the current behavior intact for the control variation.

Fixes tracks event regression

This PR also addresses a minor regression in which the tracks event enhanced_site_creation_preview_ok_button_tapped was not emitted at the end of the site creation flow for users who entered that path via the authentication screen.

Known Issue(s)

Animations

At the end of the existing user flow for the treatment variation, there is an issue where multiple dismiss animations occur before the quick start dialog is presented. It may be worth exploring ways to make this smoother (i.e. faster).

Potential for stale starter-site content

I did not encounter this, however, I foresee that this could cause potential confusion in the future: when Gutenberg is updated to somehow modify the format of a particular block that is used in the content of one of the starter-sites, the editor will parse this content when initializing, and automatically migrate the existing content to the new format. A minor issue is that this causes the content to have changed, prompting the user to save unsaved changes when they're continuing to complete the flow. For users who have not made any changes, such a prompt may be confusing, imo, especially if this is their first experience with the app / editor.

To test

Note: To change the experiment variation between Control (default) and Treatment (experiment):

1. Browse to the Abacus experiment wpios_land_in_the_editor_phase1_v3 (internal ref: pbxNRc-1fJ-p2)
2. Follow the instructions on the Bookmarklet tooltip to set the Control or Treatment variation for your user.

• Note that this seems to only work with an A8C user account (separate test accounts may not work)

Alternatively you may use a proxy (e.g. Charlie) to intercept the call to the experiments endpoint (/wpcom/v2/experiments/) and return the desired variation (detailed instructions in this previous experiment). E.g.

{
	"variations": {
		"wpios_land_in_the_editor_phase1_v3": "treatment"
	},
	"ttl": 60
}

For each of the tests below, please also verify that the tracks event: enhanced_site_creation_preview_ok_button_tapped is emitted.

Control: Login or New user

1. Login to the app with a user that has the Control variation assigned (check note above)
2. Press the Create new site option at the bottom
3. Select a design and a domain to create a site
4. Press OK on the next screen
5. Verify that you are presented with the Want a little help to manage your site dialog
6. Verify that you land on the My Site screen after your selection

Video

login-control.mp4

Treatment: Login or New user

1. Login to the app with a user that has the Treatment variation assigned (check note above)
2. Press the Create new site option at the bottom
3. Select a design and a domain to create a site
4. Press OK on the next screen
5. Verify that you are presented with the home page editor
6. Verify that the landing_editor_shown analytics event is emitted
7. Press the Continue button and save any changes
8. Verify that you are presented with the Want a little help to manage your site dialog
9. Verify that you land on the My Site screen after your selection

Video

login-treatment.mp4

Control: Existing or Logged in user

1. Login to the app with a user that has the Control variation assigned (check note above)
2. On the My Site screen press the site picker selector
3. Press the Plus(+) button at the top-right
4. Choose the Create a WordPress.com site option
5. Select a design and a domain to create a site
6. Press OK on the next screen
7. Verify that you are presented with the Want a little help to manage your site dialog
8. Verify that you land on the My Site screen after your selection

Video

existing-control.mp4

Treatment: Existing or Logged in user

1. Login to the app with a user that has the Treatment variation assigned (check note above)
2. On the My Site screen press the site picker selector
3. Press the Plus(+) button at the top-right
4. Choose the Create a WordPress.com site option
5. Select a design and a domain to create a site
6. Press OK on the next screen
7. Verify that you are presented with the home page editor
8. Verify that the landing_editor_shown analytics event is emitted
9. Press the Continue button and save any changes
10. Verify that you are presented with the Want a little help to manage your site dialog
11. Verify that you land on the My Site screen after your selection

Video

existing-treatment.mp4

Regression Notes

1. Potential unintended areas of impact
   Site creation flow

2. What I did to test those areas of impact (or what existing automated tests I relied on)

Manual testing of site creation flows
3. What automated tests I added (or what prevented me from doing so)

I began writing tests for the ExPlat A/B test, but was unable to cleanly stub the necessary dependency since the variation is provided via an enum getter (Swift enums cannot be inherited 😞 )

PR submission checklist:

• I have completed the Regression Notes.
• I have considered adding unit tests for my changes.
• I have considered adding accessibility improvements for my changes.
• I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

[peril-wordpress-mobile]

You can trigger an installable build for these changes by visiting CircleCI here.

[peril-wordpress-mobile]

You can trigger optional UI/connected tests for these changes by visiting CircleCI here.

[antonis]

Great work @mkevins 👍
I went through the draft PR and I think you implementation looks and behaves really good 🎉
I've only added a few comments which are more of suggestions or request for feedback 🙇

[antonis]

Thank you for your work on this @mkevins 🙇
The code LGTM and I think we can proceed with the final testing with an ExPlat experiment.
A potential enhancement would be to add some analytics to track the land on the editor flow. On Android I added one new event for this.

[mkevins]

Thank you for your work on this @mkevins bow The code LGTM and I think we can proceed with the final testing with an ExPlat experiment. A potential enhancement would be to add some analytics to track the land on the editor flow. On Android I added one new event for this.

Good idea, I will add this as well! 👍

[mkevins]

Thank you for your work on this @mkevins bow The code LGTM and I think we can proceed with the final testing with an ExPlat experiment. A potential enhancement would be to add some analytics to track the land on the editor flow. On Android I added one new event for this.

Good idea, I will add this as well! +1

I've added an analytics event w/ 2d38fe3

[antonis]

Great work @mkevins 👍
I tested the app on an iPhone SE 2020 (iOS 14.7.1) and everything worked as expected. The code changes are consistent and overall LGTM 🎉

[mkevins]

Thank you for reviewing and testing this Antonis!