Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ Available themes: Aurora X, Dracula, GitHub Dark, GitHub Light, Monokai, Nord, O

## Languages

`apache`, `css`, `diff`, `dockerfile`, `ejs`, `env`, `html`, `http`, `js`, `json`, `jsx`, `markdown`, `mermaid`, `nginx`, `plaintext`, `python`, `scheme`, `shell`, `sql`, `svelte`, `toml`, `ts`, `tsrx`, `tsx`, `vue`, and `yaml`.
`apache`, `css`, `diff`, `dockerfile`, `ejs`, `env`, `go`, `html`, `http`, `js`, `json`, `jsx`, `markdown`, `mermaid`, `nginx`, `plaintext`, `python`, `scheme`, `shell`, `sql`, `svelte`, `toml`, `ts`, `tsrx`, `tsx`, `vue`, and `yaml`.

Each language is available from `@tanstack/highlight/languages/<name>`. The aggregate `@tanstack/highlight/languages` entry can tree-shake, while direct subpaths make isolation explicit.

Expand All @@ -199,7 +199,7 @@ Local browser bundles, minified with esbuild and compressed independently:
| Core + TSX | 9.3 KB | 3.9 KB | 3.5 KB |
| Octane MDX + TypeScript | 12.5 KB | 4.9 KB | 4.5 KB |
| Nine-language docs set | 15.3 KB | 5.8 KB | 5.3 KB |
| All 25 languages | 22.2 KB | 8.0 KB | 7.2 KB |
| All 27 languages | 24.0 KB | 8.6 KB | 7.7 KB |

On 80 real JavaScript/TypeScript/JSX/TSX TanStack docs fixtures repeated across 5,040 blocks:

Expand Down
2 changes: 1 addition & 1 deletion docs/guides/performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Small and fast are product constraints, not README adjectives. The repository me
| TSX | TSX | 3.86 KB | 4.1 KB |
| Octane | TypeScript plus Octane MDX adapter | 4.93 KB | 5.2 KB |
| Docs | CSS, HTML, JS, JSON, JSX, Markdown, Shell, TS, TSX | 5.83 KB | 6.1 KB |
| All | All 25 definitions | 7.96 KB | 8.3 KB |
| All | All 27 definitions | 8.60 KB | 8.9 KB |

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

25 was already outdated, it was missing tsrx which was added recently too.


Selective profiles are the primary metric. The all-language profile exists to prevent convenience-entry growth from becoming invisible.

Expand Down
1 change: 1 addition & 0 deletions docs/language-support.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ Every language is an isolated definition imported from `@tanstack/highlight/lang
| Dockerfile | `dockerfile` | `docker` | Common directives, variables, commands |
| EJS | `ejs` | - | HTML plus optional JavaScript delegation |
| Env | `env` | `dotenv` | Properties, values, comments |
| Go | `go` | `golang` | Raw strings, runes, comments, declarations |
| HTML | `html` | `htm`, `xml`, `angular-html` | Optional JavaScript/TypeScript and CSS delegation |
| HTTP | `http` | - | Methods, headers, protocol, paths |
| JavaScript | `js` | `javascript`, `mjs`, `cjs`, `js-vue` | Templates, interpolation, regex literals |
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/default-entry.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Returns the canonical registered language name for a name or alias. Names are tr
function listLanguages(): Array<HighlightLanguage>
```

Returns the 25 canonical language names registered in `defaultHighlighter`.
Returns the 27 canonical language names registered in `defaultHighlighter`.

### `tokenize`

Expand Down
3 changes: 2 additions & 1 deletion docs/reference/languages.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ const highlighter = createHighlighter({
| `dockerfile` | `@tanstack/highlight/languages/dockerfile` | `docker` |
| `ejs` | `@tanstack/highlight/languages/ejs` | None |
| `env` | `@tanstack/highlight/languages/env` | `dotenv` |
| `go` | `@tanstack/highlight/languages/go` | `golang` |
| `html` | `@tanstack/highlight/languages/html` | `htm`, `xml`, `angular-html` |
| `http` | `@tanstack/highlight/languages/http` | None |
| `js` | `@tanstack/highlight/languages/js` | `javascript`, `mjs`, `cjs`, `js-vue` |
Expand All @@ -48,6 +49,6 @@ const highlighter = createHighlighter({
| `vue` | `@tanstack/highlight/languages/vue` | None |
| `yaml` | `@tanstack/highlight/languages/yaml` | `yml` |

`@tanstack/highlight/languages` re-exports `apache`, `css`, `diff`, `dockerfile`, `ejs`, `env`, `html`, `http`, `js`, `json`, `jsx`, `markdown`, `mermaid`, `nginx`, `plaintext`, `python`, `scheme`, `shell`, `sql`, `svelte`, `toml`, `ts`, `tsrx`, `tsx`, `vue`, and `yaml`. The barrel is convenient but individual subpaths make bundle intent explicit.
`@tanstack/highlight/languages` re-exports `apache`, `css`, `diff`, `dockerfile`, `ejs`, `env`, `go`, `html`, `http`, `js`, `json`, `jsx`, `markdown`, `mermaid`, `nginx`, `plaintext`, `python`, `scheme`, `shell`, `sql`, `svelte`, `toml`, `ts`, `tsrx`, `tsx`, `vue`, and `yaml`. The barrel is convenient but individual subpaths make bundle intent explicit.

See the [language support matrix](../language-support) for the context-aware behavior and current scope of each registration.
2 changes: 1 addition & 1 deletion docs/test-strategy.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ The suite protects the package's actual product boundary: valid code commonly pu
| TSX | TSX | 4.1 KB |
| Octane | TypeScript plus Octane MDX adapter | 5.2 KB |
| Docs | CSS, HTML, JS, JSON, JSX, Markdown, Shell, TS, TSX | 6.1 KB |
| All | All 25 definitions | 8.3 KB |
| All | All 27 definitions | 8.9 KB |

The selective profiles are the primary product metric. The all-language profile protects the convenience entry from unbounded growth.

Expand Down
2 changes: 2 additions & 0 deletions scripts/language-utils.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ export const supportedLanguages = [
'dockerfile',
'ejs',
'env',
'go',
'html',
'http',
'js',
Expand Down Expand Up @@ -39,6 +40,7 @@ const aliases = {
cmd: 'shell',
console: 'shell',
dotenv: 'env',
golang: 'go',
htm: 'html',
javascript: 'js',
'js-vue': 'js',
Expand Down
2 changes: 1 addition & 1 deletion scripts/measure-size.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ const profiles = {
import { defaultHighlighter } from './src/index.ts'
globalThis.highlighter = defaultHighlighter
`,
limits: { minified: 23_000, gzip: 8_300, brotli: 7_500 },
limits: { minified: 24_500, gzip: 8_900, brotli: 8_000 },
},
}

Expand Down
3 changes: 3 additions & 0 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import { diff } from './languages/diff.js'
import { dockerfile } from './languages/dockerfile.js'
import { ejs } from './languages/ejs.js'
import { env } from './languages/env.js'
import { go } from './languages/go.js'
import { html } from './languages/html.js'
import { http } from './languages/http.js'
import { js } from './languages/js.js'
Expand Down Expand Up @@ -39,6 +40,7 @@ export type HighlightLanguage =
| 'dockerfile'
| 'ejs'
| 'env'
| 'go'
| 'html'
| 'http'
| 'js'
Expand Down Expand Up @@ -107,6 +109,7 @@ export const allLanguages = [
dockerfile,
ejs,
env,
go,
html,
http,
js,
Expand Down
40 changes: 40 additions & 0 deletions src/languages/go.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
import { defineLanguage } from '../core.js'
import { patternTokenizer } from '../internal/patterns.js'

export const go = defineLanguage({
name: 'go',
aliases: ['golang'],
tokenize: patternTokenizer([
{
className: (match) =>
match[0].startsWith('/') ? 'comment' : 'string',
regex:
/\/\/[^\n]*|\/\*[\s\S]*?\*\/|`[\s\S]*?`|"(?:\\.|[^"\\\n])*"|'(?:\\.|[^'\\\n])*'/g,
},
{
className: 'keyword',
regex:
/\b(?:break|case|chan|const|continue|default|defer|else|fallthrough|for|func|go|goto|if|import|interface|map|package|range|return|select|struct|switch|type|var)\b/g,
},
{ className: 'literal', regex: /\b(?:false|iota|nil|true)\b/g },
{ className: 'type', regex: /\btype\s+([A-Za-z_]\w*)/g, group: 1 },
{
className: 'type',
regex:
/\b(?:any|bool|byte|comparable|complex64|complex128|error|float32|float64|int|int8|int16|int32|int64|rune|string|uint|uint8|uint16|uint32|uint64|uintptr)\b/g,
},
{ className: 'function', regex: /\b[A-Za-z_]\w*(?=\s*\()/g },
{
className: 'number',
regex:
/(?:^|[^\w.])((?:0[bB]_?[01](?:_?[01])*|0[oO]_?[0-7](?:_?[0-7])*|0[xX](?:(?:_?[\da-fA-F](?:_?[\da-fA-F])*(?:\.(?:[\da-fA-F](?:_?[\da-fA-F])*)?)?|\.[\da-fA-F](?:_?[\da-fA-F])*)[pP][+-]?\d(?:_?\d)*|_?[\da-fA-F](?:_?[\da-fA-F])*)|\d(?:_?\d)*\.(?:\d(?:_?\d)*)?(?:[eE][+-]?\d(?:_?\d)*)?|\d(?:_?\d)*[eE][+-]?\d(?:_?\d)*|\.\d(?:_?\d)*(?:[eE][+-]?\d(?:_?\d)*)?|\d(?:_?\d)*)(?:i)?)(?![\w.])/g,
group: 1,
},
{ className: 'property', regex: /\.\s*([A-Za-z_]\w*)/g, group: 1 },
{
className: 'operator',
regex:
/\.\.\.|<<=?|>>=?|&\^=?|:=|<-|\+\+|--|==|!=|<=|>=|&&|\|\||[+\-*/%&|^!<>=~]=?/g,
},
]),
})
1 change: 1 addition & 0 deletions src/languages/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ export { diff } from './diff.js'
export { dockerfile } from './dockerfile.js'
export { ejs } from './ejs.js'
export { env } from './env.js'
export { go } from './go.js'
export { html } from './html.js'
export { http } from './http.js'
export { js } from './js.js'
Expand Down
29 changes: 29 additions & 0 deletions test/fixtures.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,35 @@ export function Counter() @{
'th-number',
],
},
{
lang: 'golang',
normalized: 'go',
code: `package main

import "fmt"

type Greeter struct {
Name string
}

func (g Greeter) Greet() {
count := 2
url := \`https://go.dev\`
initial := 'G'
fmt.Println(url, initial, g.Name, count, true) // greet
}`,
expectedClasses: [
'th-keyword',
'th-string',
'th-type',
'th-function',
'th-property',
'th-number',
'th-literal',
'th-comment',
'th-operator',
],
},
{
lang: 'tsx',
normalized: 'tsx',
Expand Down
7 changes: 7 additions & 0 deletions test/highlight.contract.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,17 @@ describe('language inventory', () => {

it('lists the normalized support targets once', () => {
const languages = listLanguages()
const fixtureLanguages = new Set(
languageFixtures.map((fixture) => fixture.normalized),
)
expect(new Set(languages).size).toBe(languages.length)

for (const fixture of languageFixtures) {
expect(languages).toContain(fixture.normalized)
}
for (const language of languages) {
expect(fixtureLanguages, `missing representative fixture for ${language}`).toContain(language)
}
})
})

Expand Down Expand Up @@ -82,6 +88,7 @@ describe('output contract', () => {
it('keeps comment markers inside strings and quotes inside comments', () => {
const cases = [
['ts', `const url = "https://example.com" // "comment"`],
['go', `url := "https://go.dev" // "comment"`],
['python', `value = "# not a comment" # "comment"`],
['shell', `echo "# not a comment" # "comment"`],
['yaml', `url: "https://example.com/#hash" # "comment"`],
Expand Down
7 changes: 5 additions & 2 deletions test/real-doc-fixtures.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,14 @@ type DocFixture = {
}

const fixtures = fixtureData.fixtures as Array<DocFixture>
const languagesWithoutRealDocFixtures = new Set(['go'])

describe('real TanStack docs fixtures', () => {
it('covers every normalized language target with sampled docs code', () => {
it('covers normalized language targets available in TanStack docs', () => {
const fixtureLanguages = new Set(fixtures.map((fixture) => fixture.lang))

for (const language of listLanguages()) {
if (languagesWithoutRealDocFixtures.has(language)) continue
expect(fixtureLanguages, `missing real docs fixture for ${language}`).toContain(language)
}
})
Expand Down Expand Up @@ -53,8 +55,9 @@ describe('real TanStack docs fixtures', () => {
}
}

const fixtureLanguages = new Set(fixtures.map((fixture) => fixture.lang))
for (const language of listLanguages()) {
if (language === 'plaintext') continue
if (language === 'plaintext' || !fixtureLanguages.has(language)) continue
expect(highlightedLanguages, `${language} should highlight at least one real docs fixture`).toContain(language)
}
})
Expand Down
42 changes: 42 additions & 0 deletions test/regressions.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import { describe, expect, it } from 'vitest'
import { createHighlighter } from '../src/core'
import { css } from '../src/languages/css'
import { dockerfile } from '../src/languages/dockerfile'
import { go } from '../src/languages/go'
import { html } from '../src/languages/html'
import { http } from '../src/languages/http'
import { js } from '../src/languages/js'
Expand All @@ -19,6 +20,7 @@ const highlighter = createHighlighter({
languages: [
css,
dockerfile,
go,
html,
http,
js,
Expand Down Expand Up @@ -305,6 +307,46 @@ export function View() @{ <p>Ready</p> }`,
})

describe('docs language regressions', () => {
it('handles Go raw strings, runes, and numeric forms', () => {
const numbers = [
'0b1010',
'0o755',
'0x1.fp2',
'1.',
'1.e2',
'1.e2i',
'.5',
'.5i',
'.5e2',
'.5e2i',
'1e2',
'1e2i',
'0x1.p2',
'0x.8p2',
'0x_1p2',
'0x1.p2i',
'1_000.5e-2i',
]
const result = highlighter.tokenize(
[
'raw := `https://go.dev/a//b`',
"runeValue := '\\n'",
`values := []complex128{${numbers.join(', ')}}`,
'// done',
].join('\n'),
{ lang: 'go' },
)

expect(exactClassesFor(result, '`https://go.dev/a//b`')).toEqual(['string'])
expect(exactClassesFor(result, "'\\n'")).toEqual(['string'])
expect(exactClassesFor(result, 'complex128')).toEqual(['type'])
for (const number of numbers) {
expect(exactClassesFor(result, number), number).toEqual(['number'])
}
expect(exactClassesFor(result, '// done')).toEqual(['comment'])
expect(reconstruct(result)).toBe(result.code)
})

it('keeps Python triple strings together and highlights literals and numbers', () => {
const result = highlighter.tokenize(
'text = """hello\nworld"""\nempty = None\nenabled = True\ncount = 42',
Expand Down