Test failure should not be reported as a compile error

[isidorn]

Have a test failing.
Notice that python reports this as an Error. This shows up in the Problems panel and with a squigle.
This is not the right thing to do. Test errors should not be reported as compile errors.

I am not sure if there is a better alternative to use
fyi Johannes Rieken (@jrieken) Luciana Abud (@luabud)

Also we now support rendering emojis so test failures could be rendered a bit better. Johannes Rieken (@jrieken) can provide more details.

[luabud]

FWIW this was implemented by an external contributor for a feature request: #120, PR: #3303.

If I'm not mistaken the Problems window was used because there isn't a way to jump to the failed lines through the Output window?! But I may be wrong.

Anyway, we should then look into a better solution to solve the problem other than using the Problems window. As you pointed out, this isn't a syntax or compilation error.

[DonJayamanne]

IMHO I think the problems window is fine.
Though reporting test issues in problems window while we have a test explorer (dedicated UI for tests) does seem awkward.

The user requirement is that when tests fail, they expect to see the failures and be able to jump to the failed lines easily. Currently this isn't possible from the test explorer.
I.e. the test explorer doesn't provide a view for failures in a list. Agreed we can provide a custom view. But users wouldn't know about such a toggle, or we'd need to create another UI list view with exact same functionality as problems window, however that doesn't seem right.

Test errors should not be reported as compile errors.

Not sure this is correct. Today we have spell checkers reporting errors in problems window and they aren't compile errors. I wouldn't be surprised if there were others.
That may have been the intent, however I believe the community is now using the problems window to surface other types of problems.

[luabud]

Isidor Nikolic (@isidorn) I talked to the team about it and the sentiment was that we always thought having that report in the problems window would be fine (after all, having failed tests in your project could be seen as a problem).

Are there any guidelines from VS Code that we could refer to in terms of what should or shouldn't be displayed in the Problems window?

I think Don raised a good point that the community may be using the Problems window to surface various types of problems, so having some guidelines could help on this.

[isidorn]

Adding Johannes Rieken (@jrieken) and Sandeep Somavarapu (@sandy081) for Prolbems view guidelines

[sandy081]

Problems view is for code diagnostics which are generated by a compiler and Tests view is for managing and running tests. Former is compile time and later is during run time. I think this is the standard in general everywhere. As always, we never enforced this but it is expected that extensions follow this so that users will get consistent experience.

Agreed that before having tests view, authors were using different approaches (output, problems, code-decorations) to show tests. But after introducing this, we recommended all to create a test view and show test results there. You can define your own actions to run, debug the tests and more which you cannot do from Problems view where you can only do quick fixes (which is to fix your code to compile).

Today we have spell checkers reporting errors in problems window and they aren't compile errors. I wouldn't be surprised if there were others.

I do not think I agree with this and they are compile time errors.

[jrieken]

The problems view is for syntax or semantic errors and for lint'ing messages. You can obviously bend the definition of a "problem" but when we design and plan features for diagnostics then we do not think about test running, e.g "Run & Debug" will prompt you in the presence of diagnostics - that makes sense for compile errors but not for test errors, esp. not when "run" means "re-run the tests". That means there is a chance of future (semantic) breakage and for non-python users this might be confusing

Still, we like the idea of showing test failures in-line with sources (we just don't like the use of diagnostics for this) and this issue should also be an invitation to brain storming how VS Code (as a platform) should offer better test reporting UX and API. The Jest extension, while also using diagnostics 😞, uses line-end decorations which I think is pretty cool.

So, lets think further than the test explorer and lets start to collect requirements and ideas for better test integration