API changes tracking

[Udumft]

Adding a script to resolve tracking Public API changes using CI script.

Currently at MVP stage and there are some known upgrades incoming:

• storing the APIDiffReport separately and checking out it during the build
• caching and restoring Base API log to avoid building it every time
• parameterizing Diff utility to control which changes exactly we are interested, which are blocking, which are not
• need a way to specify Base version for comparison

[Udumft]

These steps should be run only if cache above was not restored. I could not find a working solution, so any suggestions are highly welcome.

[Udumft]

These basically log changes to output and fail the CI. Should we Report the diff some other way?

[1ec5]

This is fine for now. When we work on a major version, we can make the build bot optional.

[Udumft]

Here we can configure which type of changes exactly we are interested right now.

[Udumft]

There we control accessibility level of interest. Also, we can filter out undocumented or explicit ':nodoc:' items.

[Udumft]

A sample diff output with v.0.40 could be found here.

@kshehadeh, it would be also nice to receive any feedback or a use-cases from the Directions team so we could adjust the tool.

[kshehadeh]

A sample diff output with v.0.40 could be found here.

@kshehadeh, it would be also nice to receive any feedback or a use-cases from the Directions team so we could adjust the tool.

This is very promising. I don't yet have access to CircleCI, but I'll check it out as soon as I do.

[Udumft]

Here's one as attachment:
log.txt

[kshehadeh]

I'm having some trouble reading the output. Could you help explain what we're looking at there?

[Udumft]

The output lists detected updated made to public symbols, which are considered as breaking change. It should detect code-related points mentioned in this comment. Log contains the symbol name and it's location

[Udumft]

I've added some options to control the diff generation process from the command line. Now you can choose symbol access level you are interested in or if only documented symbols are included (useful for cases when only documented entities are considered publicly supported).

For example, for added section of code like this:

public var newUndocumentedProperty = false
    
    /// There is a valid doc on this property
    public var newDocumentedProperty = true
    
    /// :nodoc:
    public var newNODOCProperty = false
    
    internal var newInternalVar = false
    
    /// Another valid doc here
    internal var newInternalDocumentedVar = false

Only newDocumentedProperty property will be detected as a braking change:

**** BREAKING CHANGES DETECTED ****

Breaking changes in 'NavigationViewController'
new var: newDocumentedProperty in NavigationViewController
Program ended with exit code: 1

I would also like to provide ability for user to select exact items to check (like only detect new functions, or only check structs and so on), but it is unclear how to provide such description. Such points are controlled by filtering api log JSON keys, and keys have form like key.parsed_declaration. To control list of allowed keys it would be handy to pass in some config file, but that seems like out of scope for current task. Or at least I should not design it with knowing some real use cases for that.

[1ec5]

I would also like to provide ability for user to select exact items to check (like only detect new functions, or only check structs and so on), but it is unclear how to provide such description. Such points are controlled by filtering api log JSON keys, and keys have form like key.parsed_declaration.

This would be an interesting capability to have, but it could be tail work as long as the default is reasonable. To avoid a false sense of security, I think we’d want the default to check anything that could be a backwards-compatible change, at the risk of overcommunicating. We can make the check non-required at first.

To control list of allowed keys it would be handy to pass in some config file, but that seems like out of scope for current task. Or at least I should not design it with knowing some real use cases for that.

Yeah, I don’t have a concrete use case in mind for filtering. Perhaps with the churn that’ll come with v2.0, we may want to sort changes by type or severity, but I don’t see us sometimes caring about method names and sometimes not caring.

[1ec5]

In case it helps, here’s a list I wrote up last year with help from @JThramer, before we dropped Objective-C support in #2230, of changes that would be considered backwards-incompatible:

• Upgrading a dependency to a new major version (?)
• Migrating to a new major version of Swift or Xcode
  Mitigation: Conditionally supporting the older version
• Increasing the minimum deployment target
• Renaming a public, documented type or constant (or its runtime name)
  Mitigation: Leaving the old type name as a type alias or @compatibility_alias
• Renaming a public, documented method or function (or its Objective-C selector name)
  Mitigation: Leaving a deprecated stub under the old name (does renamed: count?)
• Removing a public, documented type, method, function, constant, or enumeration case
• Adding a public enumeration case (?)
• Adding a field to a public, documented C struct
• adding a required method to a protocol
• Changing the optionality (nullability) of a public, documented property, or an argument or return value of a public, documented method or function
• Adding an argument to a method or function
  • Giving the argument a default value (not in Objective-C)
• Adding an argument to a case of an associated-value enumeration
• Changing the default value of a public, documented property, constant, or argument (?)

[zugaldia]

@Udumft I think the CLI that @kshehadeh was interested in is the one that landed with mapbox/mapbox-directions-swift#469 (to validate Directions API responses) as opposed to this one (which is used to validate the SDK API contract).

[S2Ler]

Closing in favour of #3103