Syntax highlighting: Textmate grammars vs. inbuilt mechanism?

[Matthias247]

The officially documented way to contribute syntax highlighting support for new languages seems to be by wrapping Textmate .tmlanguage grammars.

I'm curious if this is also the mechanism that vscode uses to support the inbuilt languages. There also seems to be various tokenizers for those, like in https://github.com/Microsoft/vscode/blob/master/src/vs/editor/standalone-languages/csharp.ts .

I'm now wondering if vscode uses two different approaches.
If yes, which advantages and disadvantages gives the tokenizer based approach and is it also available for extensions?
If no, what else is the use-case for this integrated tokenizers?

[aeschli]

@Matthias247 We started with hand written tokenizers as well as a tokenizers using a library called 'Monarch'. Some modes still use these, but our goal is to move to text mate tokenizers.
Text mate tokenizers are well established and are have a lively community keeping them up-to-date. They also play well with text mate color themes which we also support.
There are still some old tokenizers around, as we ran out of time replacing these. We are working on that.

The /standalone-languages folder contains the old implementations, but they are not used in VS Code.

We still have them for other usages of the editor, for example when the editor widget (the 'standalone editor' aka 'Monaco editor') is used in a web page. We can't use Text mate there as text mate tokenizers depend on a native regex library called 'oniguruma'.

There is more to come on how to bundle and use the standalone editor.
Hope that answers your questions.

[Matthias247]

@aeschli Thanks for the detailed explanation!
I was a little bit wondering if textmate schemes are technically the better approach, because they rely very much on single-line regexes, and as soon as you put some keywords on another line the regex and thereby the highlighting breaks. I found the Codemirror/Brackets approach with a complete lexer/parser for the highlighting better suited for this problem. But I totally see the point in reusing the whole ecosystem of existing textmate schemes.

Is 0.10.1 already using a textmate scheme for typescript? I am wondering because I have a typescript source file where the highlight is totally broken up from a specific line (without a syntax error) in Atom (which is also using textmate schemes), but it looks fine in VS Code - good for VS code :)

[aeschli]

Yes, Typescript uses textmate tokenization for a while now. You can file issues against the grammar at https://github.com/Microsoft/TypeScript-TmLanguage