Widgets: Track loaded widgets when application is opened

[staskus]

Fixes #21699

Current Widget analytics are hard to interpret. We use separate events for each widget kind and add a number of different widget families to properties. We only track the event for the second time when properties change. The tracking of these events is also triggered unpredictably and driven by the system.

I noticed from the experience that the way we track widgets makes it hard to understand both the overall usage of widgets as well the most popular widget types. This makes those events hardly useful.

Changes

My goal for widget analytics is to understand two questions:

• The percentage of users who use any widgets
• The proportional usage of different widget kinds and families

Possible solutions

Track added and removed widgets

As described in this article, we can use getTimeline and getCurrentConfigurations to keep track of previous and current widget configurations and try to track what type of widgets have been added and removed.

I experimented with this approach. The drawback remains the unpredictable triggering of events. getTimeline is usually not triggered when the widgets are removed, so we could only reliably track added widgets. Also, we would still need to do additional calculations to determine the overall usage of widgets.

Track all widgets when the application is opened

WooCommerce iOS tracks widgets when an application is opened (woocommerce/woocommerce-ios#7717).

Although it does not allow to understand a more granular interaction with widgets, it does give a straightforward view into the proportional usage of different kinds of widgets (?eventname=woocommerceios_application_opened ). Moreover, we can derive the overall usage of widgets by looking at how many application open events do not contain any widgets at all.

This approach satisfies our current needs.

Solution:

1. Created WidgetAnalytics with trackLoadedWidgetsOnApplicationOpened method which loads widget configuration and adds any different widgetConfig to "widgets" property of widgets_loaded_on_application_opened event
2. Calling the method when application is opened
3. Removed an older widget tracking code

I think this solution is much more simple to understand and interpret.

To test:

1. Launch Jetpack app
2. Add home and/or lock screen widgets
3. Open app
4. Confirm that added widget types are tracked
5. Repeat steps 2-3 couple of times

I noticed that it can take getCurrentConfigurations a few seconds to return a newest configuration.

Regression Notes

1. Potential unintended areas of impact

I removed existing widget tracks. I don't think anyone found them valuable but I could be wrong.

2. What I did to test those areas of impact (or what existing automated tests I relied on)

3. What automated tests I added (or what prevented me from doing so)

PR submission checklist:

• I have completed the Regression Notes.
• I have considered adding unit tests for my changes.
• I have considered adding accessibility improvements for my changes.
• I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

UI Changes testing checklist:

• Portrait and landscape orientations.
• Light and dark modes.
• Fonts: Larger, smaller and bold text.
• High contrast.
• VoiceOver.
• Languages with large words or with letters/accents not frequently used in English.
• Right-to-left languages. (Even if translation isn’t complete, formatting should still respect the right-to-left layout)
• iPhone and iPad.
• Multi-tasking: Split view and Slide over. (iPad)

[wpmobilebot]

📲 You can test the changes from this Pull Request in WordPress Alpha by scanning the QR code below to install the corresponding build.

	App Name	WordPress Alpha
	Configuration	Release-Alpha
	Build Number	pr21710-204f4a3
	Version	23.5
	Bundle ID	org.wordpress.alpha
	Commit	204f4a3
	App Center Build	WPiOS - One-Offs #7501

Automatticians: You can use our internal self-serve MC tool to give yourself access to App Center if needed.

[wpmobilebot]

📲 You can test the changes from this Pull Request in Jetpack Alpha by scanning the QR code below to install the corresponding build.

	App Name	Jetpack Alpha
	Configuration	Release-Alpha
	Build Number	pr21710-204f4a3
	Version	23.5
	Bundle ID	com.jetpack.alpha
	Commit	204f4a3
	App Center Build	jetpack-installable-builds #6526

Automatticians: You can use our internal self-serve MC tool to give yourself access to App Center if needed.

[guarani]

Thanks @staskus! I think this approach is great.
There's a typo in the mapping of widget kinds and I left an idea about making the Tracks event property easier to analyze by breaking out the comma-separated list into separate properties.

[guarani]

The comma-separated list that this produces could be tricky to analyze or filter since all the widget names are together in a single property:

🔵 Tracked: widgets_loaded_on_application_opened <widgets: homeThisWeekWidget_systemMedium,homeAllTimeWidget_systemMedium,lockScreenTodayViewsVisitorsWidget_accessoryRectangular>

Should multiple properties be used instead? Something like this:

🔵 Tracked: widgets_loaded_on_application_opened <today_home_extension_widget_systemmedium: true, today_home_extension_widget_systemsmall: true>

The idea is that any unique widget is included as a property of the event. Any widgets not present aren't mapped to a property.

I tested this code to do this:

            return [:]
        }

        let widgetAnalyticNames: [String] = installedWidgets.map { widgetInfo in
            guard let eventKind = AppConfiguration.Widget.Stats.Kind(rawValue: widgetInfo.kind) else {
                DDLogWarn("⚠️ Make sure the widget: \(widgetInfo.kind), has the correct kind.")
                return "\(widgetInfo.kind)_\(widgetInfo.family)"
            }
            return "\(Events.eventPrefix(for: eventKind).rawValue)_\(widgetInfo.family.description.lowercased())"
        }

        let dict = Dictionary(uniqueKeysWithValues: widgetAnalyticNames.map { name in
            return (name, "true")
        })

        return dict
    }

Here I used "true" as a value for lack of a better alternative (there's no "false" value because widgets that aren't added won't be included in the dictionary). Tracks also wanted the event property to use lowercase.

What do you think?

[staskus]

Yes, you're right. I used the example of Woo, and I thought that the tracks system interprets commas more cleverly, allowing us to see the total usage of all the widgets separately under one property. However, that is not the case.

Your proposal definitely makes sense in this instance. I also updated property keys so they would all start with widget_ prefix, for easier visibility within tracks.

[staskus]

@guarani Thank you for the review! I used your advice to split into properties. I also registered an event to add descriptions for properties for clarity https://github.com/Automattic/tracks-events-registration/pull/1905

[guarani]

Thanks @staskus!

I re-tested and at first, the removal of widgets was not being reported. I waited around 2-3 minutes in case I ran into the issue you mentioned above where it can take a few seconds for the widget usage to be reported properly but no luck. I then reset the simulator and it worked as expected 👍

I don't think this is a blocker since I wasn't able to reproduce it, but sharing it in case it comes up again. Here's the video of it:

cut.mp4

[staskus]

I re-tested and at first, the removal of widgets was not being reported. I waited around 2-3 minutes in case I ran into the issue you mentioned above where it can take a few seconds for the widget usage to be reported properly but no luck. I then reset the simulator and it worked as expected 👍

Yes, I couldn't find information on how much time it takes these configurations to reload. For me, I didn't need to reset the simulator to get the configuration updated. I think it's fine given we want to understand just the overall usage of widgets and we can tolerate the numbers not being 100% up-to-date.