Our normalization system depends on being able to parse failure messages. Investigate and implement improvements for messages output by each of the test runners.
Improvements should help
- Improve visually how failures look on the UI (focus on pytest first, aim for vitest and jest by EOQ)
### Optimize Test Failure Message Normalization
- [ ] For pytest
- [ ] For vitest
- [ ] For jest
- [ ] For ruby (we have customers using test analytics)
- Optimize how we store error messages for flake comparisons
### Optimize how we store failure messages to improve efficiency of error message comparison to determine flakes
- [ ] Investigate ways to detect and improve efficiency
- [ ] Investigate and implement ways to remove stop words
- [ ] Recommend other techniques
- [ ] Implement them
- [ ] Talk to Sentry ML team - to determine how to get feedback for model performance
- [ ] Investigate if we need our own test runner plugins - helpful in getting us the right error message details from source
Our normalization system depends on being able to parse failure messages. Investigate and implement improvements for messages output by each of the test runners.
Improvements should help