CommonMark compatibility (Tracker bug)

[susnux]

Background

Currently there are "commonmark" tests, but they only test markdown-it to output correct HTML for markdown (which is already tested upstream), but they are not testing our pipeline to keep the markdown valid.
Meaning validating that the input and output are at least equivalent (both result in same HTML, as the mapping is surjective but not injective).

md ==> markdown-it ==> HTML ==> TipTap ==> prosemirror-markdown ==> md

Problems

I created a branch with adjusted tests, currently there are a lot of failures, which need to be triaged (real bugs, whitespace problems, test issues...), so this is a list of all bugs blocking full CommonMark support:

• 🐞 Link titles are removed #2699
• 🐞 Multiple (allowed) (white)spaces are removed, Better whitespace handling, parse and keep where allowed #2720
• 🐞 Link text escaping of backslash is removed #2703
• 🆙 ⏲️ ) in URL is not escaped ProseMirror/prosemirror-markdown#78 (waiting for release)
• 🆙 Link is split into multiple ones ProseMirror/prosemirror-markdown#77
• 🆙 Lines that start with a colon are escaped upon save #969 (Waiting for release of Unneeded escaping of colons at the start of a line ProseMirror/prosemirror-markdown#76 )
• 🆙 Fenced code blocks does not check for backticks ProseMirror/prosemirror-markdown#88 (we need to upgrade to prosemirror-markdown >= 1.10.1)
• 🆙 Wrong parsing of emphasis and strong emphasis formatting ProseMirror/prosemirror-markdown#82
• ➕ Parse HTML tags #2170 (HTML is required for full CommonMark support)

Opinions

There are already issues about breaking the formatting of markdown files, like #593 and #2577 but from my point of view it is hard to provide that kind of compatibility, at least the way text is build.
But I do not think it is needed, if I need a markdown editor with that behavior, I would recommend to use https://github.com/icewind1991/files_markdown
Of cause you as the developers of this app should decide the scope of application, but for me text is a collaborative text editor.

[susnux]

This test results show where CommonMark markdown is still changed, so it results in semantically unequal code:

CI results (broken-tests-log.txt)

[mejo-]

We're on ProseMirror 0.10.1 now, so "Lines that start with a colon are escaped upon save" is resolved. Thanks again for debugging, reporting and fixing @susnux ❤️