[TA] AI generated failed test solution

[Adal3n3]

Problem

Debugging failed tests is a time-intensive and frustrating task for developers. With our current Test Analytics solution, when tests fail, developers can easily access the error message, the file and test where the error occurred, and the stack traces. This helps them focus on their test-fixing workflow without needing to dig through logs to identify the failure reason.

However, manually analyzing root-cause information can be time-consuming when determining a solution for fixing failed tests. Often, stack traces contain overwhelming information and can be challenging to interpret without providing a more direct path to a solution.

Proposed Solution

Leverage AI to generate suggestions for fixing failed tests. This would help developers reduce the time spent analyzing failures, enabling faster debugging. It would make the test-fixing workflow more developer-friendly and less frustrating while aligning seamlessly with the existing development process.

We understand that there isn’t always a one-size-fits-all solution. For the initial design iteration, we can narrow the scope to provide one or two AI-generated suggestions and focus on ensuring the quality of these suggestions.

Design Exploration

Exploration 1: Provide one single solution only

Exploration 2: Provide > 1 solution

[Adal3n3]

cc @rohan-at-sentry for product planning

[Adal3n3]

12/31 Feedback on slack:

• Eli: change the ''Steps to fix..." copy to "use AI Autofix" or something to leverage that branding from Sentry (and it likely will use autofix under the hood)
• Rohit: it feels a little "hidden" imo, like an extended part of the stack trace. Maybe we can use an emoji or image to communicate that "hey we have a potential solution here" ?

[rohan-at-sentry]

Defering till Q2 or until AI test generation ships and we have in-market performance

[Adal3n3]

2/11 Meeting with @loretta:

• We need to implement a UI consent flow that allows org admins to opt-in or opt-out of AI consent for their users. AI solutions should not be activated for users without their admin's consent.
• We will need to clarify what data we will use to generate the AI solution, will we be displaying users' code snippets? We will go through these questions with legal, after we start implementing this feature. 
• One idea is to include a link in PR comments that explains the need for consent to enable AI solutions. This link could direct members to communicate with their admins about consent.
• Note that in Sentry, Auto-fix consent covered all AI features. For now, we'll tailor AI consent to TA only, but we may expand it to all AI features in Codecov in the future.

[Adal3n3]

2/14 Meeting with @rohan-at-sentry @rohitvinnakota-codecov @suejung-sentry:

1. Admins must install the Codecov AI app (but the backend remains the same as test generation). Members will see a message if the AI integration has not been enabled so they can communicate to their admins or if the AI doesn't have access to the repo.
2. The AI consent, including source code information, is part of the Codecov AI app—so it remains the same. No need in-app AI consent. (also we can't glue and support uninstall the app with consent within the Codecov UI)
3. ToDo: I'll mock out the installation flow for the Codecov AI app as the next step for review. Pr comment only no change in the TA UI.
4. @rohitvinnakota-codecov will talk to @katia-sentry to plan this work for Q1.