Skip to content
Draft
Show file tree
Hide file tree
Changes from 93 commits
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
3beff85
[WIP] Simplify core
LeaVerou Apr 26, 2025
46806b6
highlightAll
LeaVerou Apr 26, 2025
e4f6634
Just use properties on env, not a separate `state` property
LeaVerou Apr 26, 2025
183817c
Move singleton, eliminate state further
LeaVerou Apr 26, 2025
d1097d8
Finish removing HookState
LeaVerou Apr 26, 2025
e3e5db2
Update prism-class.ts
LeaVerou Apr 26, 2025
c866261
Delete hook-state.ts
LeaVerou Apr 26, 2025
8ce6c69
Move more functions out of prism-class
LeaVerou Apr 26, 2025
1a2c60b
Merge branch 'v2' into simplify
LeaVerou Apr 26, 2025
52fca5d
Move tokenize
LeaVerou Apr 26, 2025
1158aa9
Update src/core.ts
LeaVerou Apr 27, 2025
9acac36
Update src/types.d.ts
LeaVerou Apr 27, 2025
8a8bc18
Update src/global.ts
LeaVerou Apr 27, 2025
e0ca584
Update hooks.ts
LeaVerou Apr 27, 2025
2727307
Update hooks.ts
LeaVerou Apr 27, 2025
96e93ea
Update src/core/prism-class.ts
LeaVerou Apr 27, 2025
dbbb0dd
Update src/global.ts
LeaVerou Apr 27, 2025
4dfff4c
Update registry.ts
LeaVerou Apr 27, 2025
a2a5d23
Update eslint.config.mjs
LeaVerou Apr 27, 2025
1e31274
Remove known plugins
LeaVerou Apr 27, 2025
ca6df67
Update prism.ts
LeaVerou Apr 27, 2025
051e0e6
Update src/core/highlight.ts
LeaVerou Apr 27, 2025
2147cae
Update src/core/highlight.ts
LeaVerou Apr 27, 2025
aac6bf7
Merge branch 'v2' into simplify
LeaVerou Apr 27, 2025
6513275
Merge branch 'simplify' of https://github.com/PrismJS/prism into simp…
LeaVerou Apr 27, 2025
de0b0bd
Update src/plugins/keep-markup/prism-keep-markup.ts
LeaVerou Apr 27, 2025
83fad95
[build] Delete `known-plugins.d.ts`
DmitrySharabin Apr 27, 2025
b4366b2
[WIP] Make language definitions more declarative, simplify registry, …
LeaVerou Apr 28, 2025
73ef579
Merge branch 'simplify' of https://github.com/PrismJS/prism into simp…
LeaVerou Apr 28, 2025
0762142
Fixes (#3915)
LeaVerou Apr 28, 2025
60cadeb
`extends` -> `base` and a few other things
LeaVerou Apr 29, 2025
c7ce545
Fix `c`
LeaVerou Apr 29, 2025
abf6b71
Concept for language extensions
LeaVerou Apr 29, 2025
52fe2ae
Move $insertBefore and $delete to extend()
LeaVerou Apr 29, 2025
e12f6df
Separate language patches from other eval, attempt to move types to m…
LeaVerou Apr 29, 2025
42312a5
Update actionscript.ts
LeaVerou Apr 29, 2025
4d8b9ea
Update bison.ts
LeaVerou Apr 29, 2025
f997850
Type wrangling
LeaVerou Apr 29, 2025
5bf9582
Merge branch 'v2' into simplify
LeaVerou Apr 29, 2025
93bbe20
Some suggestions (#3919)
DmitrySharabin Apr 29, 2025
0e674d2
Add proper typing for the `Grammar` type (#3918)
DmitrySharabin Apr 29, 2025
107632e
Preserve context
DmitrySharabin Apr 29, 2025
bf68a19
Merge branch 'simplify' of https://github.com/PrismJS/prism into simp…
LeaVerou Apr 30, 2025
b7623b6
Move iterable helpers to separate file
LeaVerou May 1, 2025
b850074
TS
LeaVerou May 1, 2025
bc5144f
Merge branch 'v2' into simplify
LeaVerou May 1, 2025
1deb867
Update eslint.config.mjs
LeaVerou May 1, 2025
257e650
WIP, improve API around languages
LeaVerou May 1, 2025
552c9e6
First pass on transforming `php` and `php-extras` (#3922)
DmitrySharabin May 1, 2025
1c3531c
First pass on transforming ASP.NET (C#) (#3920)
DmitrySharabin May 1, 2025
269f5d4
First pass (#3921)
DmitrySharabin May 1, 2025
9a70161
Move grammar back into types
LeaVerou May 1, 2025
9ffbe35
Languages
LeaVerou May 1, 2025
6a14538
Drop tokenize and rest symbols in favor of $tokenize and $rest proper…
LeaVerou May 1, 2025
0ea3547
Update tokenize.ts
LeaVerou May 1, 2025
586bd55
Sample rewrite to show what group matching + $language can do
LeaVerou May 1, 2025
69f8bf3
Remove effect from languages
LeaVerou May 1, 2025
097f21a
Suggestions to the `simplify` branch (#3925)
DmitrySharabin May 2, 2025
6fb945c
Allow implicit any
LeaVerou May 2, 2025
f7c21f6
Move language utils to `util/` and separate them out
LeaVerou May 2, 2025
1baaebf
Add aliases to C++, C#, F# so markdown doesn't need to handle them sp…
LeaVerou May 2, 2025
b5d562e
Drop xml-doc
LeaVerou May 2, 2025
ff26b79
Drop shared file, not worth it
LeaVerou May 2, 2025
42ebc79
Some more work on dynamic langs (still very WIP)
LeaVerou May 2, 2025
9fb2ed1
`resolveGrammar` -> `grammarPatch`
LeaVerou May 2, 2025
3d760a5
insertAfter, deep keys
LeaVerou May 2, 2025
9dc1677
Make php-extras just a part of PHP
LeaVerou May 2, 2025
ba3a8b4
Add media types and extensions
LeaVerou May 2, 2025
dcf018c
Fix typo
DmitrySharabin May 2, 2025
577eb31
Some more suggestions to the `simplify` branch (#3926)
DmitrySharabin May 3, 2025
6661905
Preserve accessors when extending/copying objects
LeaVerou May 5, 2025
b76f673
Suggestions (next iteration) (#3928)
DmitrySharabin May 7, 2025
a08fd18
Simplify hooks, add type safety
LeaVerou May 7, 2025
e2d15e9
Update util.ts
LeaVerou May 7, 2025
dedc06d
WIP
LeaVerou May 7, 2025
abc87fd
Update src/core/classes/language-registry.ts
LeaVerou May 7, 2025
1b7e413
Update highlight.ts
LeaVerou May 7, 2025
c08e814
Merge branch 'simplify' of https://github.com/PrismJS/prism into simp…
LeaVerou May 7, 2025
5f1da53
Update src/core/classes/hooks.ts
LeaVerou May 7, 2025
97a28fd
Drop `$language`
LeaVerou May 13, 2025
0d0ea00
Break down `tokenize.ts` into separate modules
LeaVerou May 15, 2025
ff0a08b
Prettier
LeaVerou May 15, 2025
4a631e9
Registry -> ComponentRegistry
LeaVerou May 15, 2025
4e3f178
async util
LeaVerou May 15, 2025
c03ab64
Update eslint.config.mjs
LeaVerou May 17, 2025
7c8205a
Update async.ts
LeaVerou May 17, 2025
4a71b8d
Update async.ts
LeaVerou May 17, 2025
3e1e9c7
Update language.ts
LeaVerou May 17, 2025
2f5c194
Update language.ts
LeaVerou May 17, 2025
eb5e425
Update types.d.ts
LeaVerou May 17, 2025
f069a78
tokenize
LeaVerou May 17, 2025
1888f4a
Move Token class to classes
LeaVerou May 17, 2025
3a25f8e
First stab at functional $inside/$rest
LeaVerou May 18, 2025
630b6d4
Some suggestions to the `simplify` branch (#3939)
DmitrySharabin May 18, 2025
7a7979c
Update the `Language` class (#3941)
DmitrySharabin May 25, 2025
a1112bc
Some fixes to `tokenize` (#3942)
DmitrySharabin May 25, 2025
bc34dde
Partially fix tests (in the `simplify` branch) (#3940)
DmitrySharabin May 25, 2025
e07c8b7
Another attempt to avoid the “too deep” structure error and not fall…
DmitrySharabin May 26, 2025
ca7fc26
Update Prettier plugins (#3945)
DmitrySharabin May 27, 2025
ccbc10b
Bug fixes (the `simplify` branch) (#3943)
DmitrySharabin May 28, 2025
3935b1a
Correctly parse CLI arguments (#3949)
DmitrySharabin May 29, 2025
0725758
Make `nextTick()` work in environments other than browsers (#3952)
DmitrySharabin May 29, 2025
b86c4e0
Resolve the language if its definition is already loaded, and it is n…
DmitrySharabin May 29, 2025
830fdff
Fix the Markdown language (#3950)
DmitrySharabin May 29, 2025
5660d05
Transform languages (#3948)
DmitrySharabin May 31, 2025
37bdeeb
Fix the gradle language (#3955)
DmitrySharabin Jun 1, 2025
7c7c598
In addition to waiting for all promises to settle, handle post-hoc ad…
DmitrySharabin Jun 17, 2025
d8246d5
Integrate changes from the `v2` branch (#3982)
DmitrySharabin Jul 11, 2025
54a2dfa
Merge branch 'v2' into simplify
DmitrySharabin Jul 11, 2025
d110fa7
Fix issues after resolving conflicts
DmitrySharabin Jul 11, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .prettierignore
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
/src/plugins
tsconfig.json
30 changes: 6 additions & 24 deletions benchmark/benchmark.ts
Original file line number Diff line number Diff line change
@@ -1,17 +1,18 @@
import Benchmark from 'benchmark';
import fetch from 'cross-fetch';
import crypto from 'crypto';
import fs from 'fs';
import path from 'path';
import { fileURLToPath } from 'url';
import Benchmark from 'benchmark';
import fetch from 'cross-fetch';
import { gitP } from 'simple-git';
import yargs from 'yargs';
import { hideBin } from 'yargs/helpers';
import { toArray } from '../src/util/iterables';
import { parseLanguageNames } from '../tests/helper/test-case';
import { config as baseConfig } from './config';
import type { Prism } from '../src/core';
import type { Prism } from '../src/types';
import type { Config, ConfigOptions } from './config';
import type { Options, Stats } from 'benchmark';
import { fileURLToPath } from 'url';

const __dirname = path.dirname(fileURLToPath(import.meta.url));

Expand Down Expand Up @@ -129,7 +130,6 @@ async function runBenchmark (config: Config) {
const name = candidates[i].name.padEnd(maxCandidateNameLength, ' ');
const best = String(s.best).padStart('best'.length);
const worst = String(s.worst).padStart('worst'.length);
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
const relative = ((s.avgRelative! / minAvgRelative).toFixed(2) + 'x').padStart(
'relative'.length
);
Expand Down Expand Up @@ -279,8 +279,7 @@ async function getFilePath (uri: string) {

// file path
const hash = crypto.createHash('md5').update(uri).digest('hex');
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
const localPath = path.resolve(downloadDir, hash + '-' + /[-\w\.]*$/.exec(uri)![0]);
const localPath = path.resolve(downloadDir, hash + '-' + /[-.\w]*$/.exec(uri)![0]);

if (!fs.existsSync(localPath)) {
// download file
Expand Down Expand Up @@ -364,7 +363,6 @@ function createTestFunction (
if (testFunction === 'tokenize') {
return code => {
const grammar = Prism.components.getLanguage(mainLanguage);
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
Prism.tokenize(code, grammar!);
};
}
Expand Down Expand Up @@ -406,7 +404,6 @@ async function getCandidates (config: Config): Promise<Candidate[]> {
const baseGit = gitP(remoteBaseDir);

for (const remote of config.remotes) {
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
const user = /[^/]+(?=\/prism.git)/.exec(remote.repo)![0];
const branch = remote.branch || 'main';
const remoteName = `${user}@${branch}`;
Expand Down Expand Up @@ -440,19 +437,4 @@ async function getCandidates (config: Config): Promise<Candidate[]> {
return candidates;
}

/**
* A utility function that converts the given optional array-like value into an array.
*/
function toArray<T extends {}> (value: T[] | T | undefined | null): readonly T[] {
if (Array.isArray(value)) {
return value;
}
else if (value != null) {
return [value];
}
else {
return [];
}
}

runBenchmark(getConfig()).catch(error => console.error(error));
32 changes: 15 additions & 17 deletions eslint.config.mjs
Original file line number Diff line number Diff line change
@@ -1,17 +1,15 @@
import path from 'path';
import { fileURLToPath } from 'url';
import js from '@eslint/js';

import tsEslintPlugin from '@typescript-eslint/eslint-plugin';
import tsEslintParser from '@typescript-eslint/parser';
import { defineConfig } from 'eslint/config';
import eslintConfigPrettier from 'eslint-config-prettier/flat';
import eslintCommentsPlugin from 'eslint-plugin-eslint-comments';
import jsdocPlugin from 'eslint-plugin-jsdoc';
import regexpPlugin from 'eslint-plugin-regexp';
import { defineConfig } from 'eslint/config';
import globals from 'globals';

import path from 'path';
import { fileURLToPath } from 'url';

const __dirname = path.dirname(fileURLToPath(import.meta.url));

const config = [
Expand All @@ -27,6 +25,10 @@ const config = [
languageOptions: {
ecmaVersion: 'latest',
sourceType: 'module',
globals: {
...globals.browser,
...globals.node,
},
},
rules: {
'no-use-before-define': ['warn', { 'functions': false, 'classes': false }],
Expand All @@ -37,7 +39,7 @@ const config = [
'object-shorthand': ['warn', 'always', { avoidQuotes: true }],
'one-var': ['warn', 'never'],
'prefer-arrow-callback': 'warn',
'prefer-const': ['warn', { 'destructuring': 'all' }],
'prefer-const': 'off',
'prefer-spread': 'warn',

// JSDoc
Expand Down Expand Up @@ -150,21 +152,17 @@ const config = [

'@typescript-eslint/no-unsafe-call': 'off',
'@typescript-eslint/no-explicit-any': 'off',
},
},
{
// Core
files: ['src/core/**/*.ts'],
languageOptions: {
globals: {
...globals.browser,
...globals.node,
},
'@typescript-eslint/no-unsafe-assignment': 'off',
'@typescript-eslint/no-unsafe-member-access': 'off',
'@typescript-eslint/no-unsafe-return': 'off',
'@typescript-eslint/no-unsafe-argument': 'off',
'@typescript-eslint/no-floating-promises': 'off',
'@typescript-eslint/await-thenable': 'off',
},
},
{
// Browser-specific parts
files: ['src/auto-start.ts'],
files: ['src/global.ts'],
languageOptions: {
globals: {
...globals.browser,
Expand Down
37 changes: 21 additions & 16 deletions scripts/build.ts
Original file line number Diff line number Diff line change
@@ -1,20 +1,19 @@
/* eslint-disable @typescript-eslint/no-non-null-assertion */
import ts from 'typescript';
import fs from 'fs';
import { copyFile, mkdir, readdir, readFile, rm, writeFile } from 'fs/promises';
import path from 'path';
import { fileURLToPath } from 'url';
import commonjs from '@rollup/plugin-commonjs';
import rollupTerser from '@rollup/plugin-terser';
import rollupTypescript from '@rollup/plugin-typescript';
import CleanCSS from 'clean-css';
import fs from 'fs';
import { mkdir, readFile, readdir, rm, writeFile, copyFile } from 'fs/promises';
import MagicString from 'magic-string';
import path from 'path';
import { rollup } from 'rollup';
import { fileURLToPath } from 'url';
import ts from 'typescript';
import { webfont } from 'webfont';
import { toArray } from '../src/shared/util';
import { toArray } from '../src/util/iterables';
import { components } from './components';
import { parallel, runTask, series } from './tasks';
import type { ComponentProto } from '../src/types';
import type { ComponentProto, LanguageProto } from '../src/types';
import type { OutputOptions, Plugin, RollupBuild, RollupOptions, SourceMapInput } from 'rollup';

const __dirname = path.dirname(fileURLToPath(import.meta.url));
Expand Down Expand Up @@ -141,7 +140,7 @@
const data = await Promise.all(
[...languageIds, ...pluginIds].map(async id => {
const proto = await loadComponent(id);
return { id, alias: toArray(proto.alias) };
return { id, alias: toArray((proto as LanguageProto).alias) };
})
);
return Object.fromEntries(data.flatMap(({ id, alias }) => alias.map(a => [a, id])));
Expand All @@ -163,7 +162,7 @@
const data = (
await Promise.all(
languageIds.map(async id => {
const proto = await loadComponent(id);
const proto = (await loadComponent(id)) as LanguageProto;
const title = rawTitles.get(id);
if (!title) {
throw new Error(`No title for ${id}`);
Expand Down Expand Up @@ -238,7 +237,7 @@
renderChunk (code) {
const str = new MagicString(code);
str.replace(
/\/((?:[^\n\r[\\\/]|\\.|\[(?:[^\n\r\\\]]|\\.)*\])+)\/\s*\.\s*source\b/g,

Check warning on line 240 in scripts/build.ts

View workflow job for this annotation

GitHub Actions / lint

Unnecessary escape character: \/
(m, source: string) => {
// escape backslashes
source = source.replace(/\\(.)|\[(?:\\s\\S|\\S\\s)\]/g, (m, g1: string) => {
Expand Down Expand Up @@ -280,7 +279,7 @@
renderChunk(code) {
const str = new MagicString(code);
str.replace(
/^(?<indent>[ \t]+)grammar: (\{[\s\S]*?^\k<indent>\})/m,

Check warning on line 282 in scripts/build.ts

View workflow job for this annotation

GitHub Actions / lint

The quantifier '[\s\S]*?' is always entered despite having a minimum of 0. This is because the assertion '^' contradicts with the element(s) after the quantifier. Either set the minimum to 1 (+?) or change the assertion
(m, _, grammar: string) => `\tgrammar: () => (${grammar})`
);
return toRenderedChunk(str);
Expand Down Expand Up @@ -319,7 +318,7 @@
]);
}

async function copyComponentsJson () {
async function copyComponentsJson() {
const from = path.join(SRC_DIR, 'components.json');
const to = path.join(__dirname, '../dist/components.json');
await copyFile(from, to);
Expand All @@ -329,7 +328,7 @@
await mkdir('./types');

// Copy existing type definitions
const typeFiles = ['types.d.ts', 'known-plugins.d.ts'];
const typeFiles = ['types.d.ts'];

await Promise.all(
typeFiles.map(file => copyFile(path.join(SRC_DIR, file), path.join('./types', file)))
Expand Down Expand Up @@ -410,11 +409,11 @@
rollupOptions: {
...defaultRollupOptions,
input: {
'prism': path.join(SRC_DIR, 'auto-start.ts'),
}
'prism': path.join(SRC_DIR, 'global.ts'),
},
},
outputOptions: defaultOutputOptions,
}
},
};

try {
Expand All @@ -430,4 +429,10 @@
}
}

runTask(series(clean, parallel(buildTypes, buildJS, series(treeviewIconFont, minifyCSS)), copyComponentsJson));
runTask(
series(
clean,
parallel(buildTypes, buildJS, series(treeviewIconFont, minifyCSS)),
copyComponentsJson
)
);
65 changes: 0 additions & 65 deletions src/auto-start.ts

This file was deleted.

8 changes: 0 additions & 8 deletions src/components.json
Original file line number Diff line number Diff line change
Expand Up @@ -896,10 +896,6 @@
"title": "PHPDoc",
"owner": "RunDevelopment"
},
"php-extras": {
"title": "PHP Extras",
"owner": "milesj"
},
"plant-uml": {
"title": "PlantUML",
"owner": "RunDevelopment"
Expand Down Expand Up @@ -1297,10 +1293,6 @@
},
"owner": "freakmaxi"
},
"xml-doc": {
"title": "XML doc (.net)",
"owner": "RunDevelopment"
},
"xojo": {
"title": "Xojo (REALbasic)",
"owner": "Golmote"
Expand Down
69 changes: 69 additions & 0 deletions src/config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
const hasDOM = typeof document !== 'undefined' && typeof window !== 'undefined';
const scriptElement: HTMLOrSVGScriptElement | null = hasDOM ? document.currentScript : null;
const globalConfig: Record<string, PrismConfig[keyof PrismConfig] | null> =
// @ts-ignore

Check warning on line 4 in src/config.ts

View workflow job for this annotation

GitHub Actions / lint

Use "@ts-expect-error" instead of "@ts-ignore", as "@ts-ignore" will do nothing if the following line is error-free
globalThis.Prism?.constructor?.name === 'Object' ? globalThis.Prism : {};

function getGlobalSetting (name: string) {
// eslint-disable-next-line regexp/no-unused-capturing-group
let camelCaseName = name.replace(/-([a-z])/g, g => g[1].toUpperCase());

if (camelCaseName in globalConfig) {
return globalConfig[camelCaseName];
}
else if (name in globalConfig) {
return globalConfig[name];
}
else if (hasDOM) {
return (
scriptElement?.dataset[camelCaseName] ??
document.querySelector(`[data-prism-${name}]`)?.getAttribute(`data-prism-${name}`)
);
}
}

function getGlobalBooleanSetting (name: string, defaultValue: boolean): boolean {
const value = getGlobalSetting(name);

if (value === null || value === undefined) {
return defaultValue;
}

return !(value === false || value === 'false');
}

function getGlobalArraySetting (name: string): string[] {
const value = getGlobalSetting(name);
if (value === null || value === undefined || value === false || value === 'false') {
return [];
}
else if (typeof value === 'string') {
return value.split(',').map(s => s.trim());
}
else if (Array.isArray(value)) {
return value;
}

return [];
}

export interface PrismConfig {
manual?: boolean;
silent?: boolean;
errorHandler?: (reason: any) => PromiseLike<never>;
plugins?: string[];
languages?: string[];
pluginPath?: string;
languagePath?: string;
}

export const globalDefaults: PrismConfig = {
manual: getGlobalBooleanSetting('manual', !hasDOM),
silent: getGlobalBooleanSetting('silent', false),
languages: getGlobalArraySetting('languages'),
plugins: getGlobalArraySetting('plugins'),
languagePath: (getGlobalSetting('language-path') ?? './languages/') as string,
pluginPath: (getGlobalSetting('plugin-path') ?? './plugins/') as string,
};

export default globalDefaults;
Loading
Loading