Skip to content

Commit 687c7f4

Browse files
EagleoutIceEllpeck
EagleoutIce
and
Ellpeck
authored
New Parser Backend (Tree-Sitter) (#1221)
* feat: initial tree-sitter setup * wip: stuff we talked about * refactor: move tree sitter wasm to git lfs * wip: lots of work, lots of issues * refactor: use the recommended name for tree sitter vars * wip: more work * feat: a lot more work on supporting tree-sitter types * feat: support more tree-sitter types * feat: finished basic tree sitter normalization * test: automatically test for equality between the two ast variations * test: check out lfs when testing to fix wasm loading * test-fix: no that was awful * feat: full parity between RShell and TS outputs * refactor: remove dangling true * feat: separate tree sitter and r shell normalize contexts * refactor: switch from deprecated external marker * refactor(tests): avoid ast reparse * refactor: synchronize step names * feat: build a facade around rshell and treesitterexecutor * feat: use the tree sitter executor when sourcing * refactor: revert an unneeded change * feat: basic engine config * refactor: don't enforce dragging string | Tree along * refactor: cool config stuff (and todo) * refactor: specify which parsers and parser return types are known * feat: integrate engine configs into the repl * wip: todo comment about config parsing * feat: enable all engines by default in config file * feat: amend config with commandline args if they have an equivalent * feat: use specified engine on repl and server * test-fix: fixed tests oops * feat: copy tree-sitter binaries into the appropriate locations for build * refactor: remove todo, this seems to work as-is in all cases * refactor: clean up benchmark slicer initialization * feat: allow performance benchmarks to run using tree-sitter * test: also run slicing tests with tree-sitter * feat-fix: allow explicit returns in tree-sitter * feat-fix: unescape backticks for identifiers * test: also compare asts in slicing tests * feat-fix: tree-sitter returns * feat-fix: small fixes for comment issues * feat-fix: various parity fixes for comment linking * test: print mermaid urls if tree comparisons fail * test: skip invalid compare test, see #1209 * test: skip tests with inconsistent comment placement, see #1208 * feat: allow specifying a custom tree-sitter wasm location * ci: run benchmarks for both parser types * ci-fix: probably fix benchmarks * ci-fix: fix uploads and upload graph data as artifact for testing * doc: use the new naming for the old data * ci-fix: fixed performance test always running with r-shell * Revert "doc: use the new naming for the old data" This reverts commit 5481904. * ci: use old naming for r-shell suites? * dep: update nanoid to remove security vulnerability * ci-fix(flowr-run): pass on arguments * feat-fix: disallow running arbitrary R code through :execute * feat-fix: ensureSessionClosed throws if the session was never opened * dep: update tree-sitter and rebuild tree-sitter-r using tree-sitter cli * dep: rebuild tree-sitter-r with the most recent version * dep: use a local version of tree-sitter * ci: timeout-minutes reduction * ci: actually check out lfs omg * refactor: deduplicate r shell execution * doc(tree-sitter): document implications for flowr --------- Co-authored-by: Ellpeck <[email protected]>
1 parent fb9c3af commit 687c7f4

File tree

95 files changed

+1826
-448
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

95 files changed

+1826
-448
lines changed

‎.gitattributes‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,4 @@
33
*.png filter=lfs diff=lfs merge=lfs -text
44
*.jpg filter=lfs diff=lfs merge=lfs -text
55
*.gif filter=lfs diff=lfs merge=lfs -text
6+
*.wasm filter=lfs diff=lfs merge=lfs -text

‎.github/workflows/broken-links-and-wiki.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,7 @@ jobs:
8585
update_wiki_page "Dataflow Graph" wiki:df-graph
8686
update_wiki_page "Query API" wiki:query-api
8787
update_wiki_page "Search API" wiki:search-api
88+
update_wiki_page "Engines" wiki:engines
8889
update_wiki_page "Interface" wiki:interface
8990
update_wiki_page "Normalized AST" wiki:normalized-ast
9091
update_wiki_page "Linting and Testing" wiki:linting-and-testing

‎.github/workflows/qa.yaml‎

Lines changed: 74 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,7 @@ jobs:
9595
uses: actions/checkout@v4
9696
with:
9797
submodules: true
98+
lfs: true
9899

99100
- name: "🌍 Load Versions to Use"
100101
id: doc-global-versions
@@ -196,17 +197,19 @@ jobs:
196197
)
197198
}}
198199
runs-on: ubuntu-22.04
199-
timeout-minutes: 400
200+
timeout-minutes: 60
200201
strategy:
201202
fail-fast: true
202203
matrix:
203204
# currently we have a duplication, add your benchmark names here and in `performance-test-upload`
204205
name: ['artificial', 'social-science']
206+
parser: ["tree-sitter", "r-shell"]
205207
steps:
206208
- name: "🛒 Checkout Repository"
207209
uses: actions/checkout@v4
208210
with:
209211
submodules: true
212+
lfs: true
210213

211214
- name: "🌍 Load Versions to Use"
212215
id: doc-global-versions
@@ -228,12 +231,12 @@ jobs:
228231
run: install.packages("xmlparsedata", repos="https://cloud.r-project.org/")
229232

230233
- name: "⏱️ Run the performance benchmarks"
231-
run: bash .github/workflows/scripts/run-flowr-command.sh performance-test -- 1 1 "${{ matrix.name }}"
234+
run: bash .github/workflows/scripts/run-flowr-command.sh performance-test -- 1 1 "${{ matrix.name }}" "${{ matrix.parser }}"
232235

233236
- name: "⬆️ Upload Benchmark Results"
234237
uses: actions/upload-artifact@v4
235238
with:
236-
name: benchmark-results-${{ matrix.name }}
239+
name: benchmark-results-${{ matrix.name }}-${{ matrix.parser }}
237240
path: test/performance/results/
238241

239242
performance-test-upload:
@@ -248,12 +251,6 @@ jobs:
248251
lfs: true
249252
token: ${{ secrets.RELEASE_TOKEN }}
250253

251-
- name: "⬇️ Get benchmark Artifacts (artificial)"
252-
uses: actions/download-artifact@v4
253-
with:
254-
name: benchmark-results-artificial
255-
path: benchmark-ai/
256-
257254
- name: "⚙️ Name and Email for Git (config) and update"
258255
# we pull just to be totally sure
259256
run: |
@@ -262,12 +259,18 @@ jobs:
262259
git pull
263260
264261
# currently we have a duplication, add your benchmark names here and in `performance-test` and check the release.yaml
265-
- name: "🥄 Merge and Produce Performance Results (artificial)"
262+
- name: "⬇️ Get benchmark Artifacts (artificial, r-shell)"
263+
uses: actions/download-artifact@v4
264+
with:
265+
name: benchmark-results-artificial-r-shell
266+
path: benchmark-ai-r-shell/
267+
268+
- name: "🥄 Merge and Produce Performance Results (artificial, r-shell)"
266269
uses: benchmark-action/github-action-benchmark@v1
267270
with:
268271
name: '"artificial" Benchmark Suite'
269272
tool: 'customSmallerIsBetter'
270-
output-file-path: benchmark-ai/artificial/artificial-summarized-graph.json
273+
output-file-path: benchmark-ai-r-shell/artificial/artificial-summarized-graph.json
271274
github-token: ${{ secrets.GITHUB_TOKEN }}
272275
fail-on-alert: false
273276
summary-always: false
@@ -280,18 +283,18 @@ jobs:
280283
benchmark-data-dir-path: wiki/stats/benchmark/
281284
auto-push: false
282285

283-
- name: "⬇️ Get benchmark Artifacts (social-science)"
286+
- name: "⬇️ Get benchmark Artifacts (social-science, r-shell)"
284287
uses: actions/download-artifact@v4
285288
with:
286-
name: benchmark-results-social-science
287-
path: benchmark-ss/
289+
name: benchmark-results-social-science-r-shell
290+
path: benchmark-ss-r-shell
288291

289-
- name: "🥄 Merge and Produce Performance Results (social-science)"
292+
- name: "🥄 Merge and Produce Performance Results (social-science, r-shell)"
290293
uses: benchmark-action/github-action-benchmark@v1
291294
with:
292295
name: '"social-science" Benchmark Suite'
293296
tool: 'customSmallerIsBetter'
294-
output-file-path: benchmark-ss/social-science/social-science-summarized-graph.json
297+
output-file-path: benchmark-ss-r-shell/social-science/social-science-summarized-graph.json
295298
github-token: ${{ secrets.GITHUB_TOKEN }}
296299
fail-on-alert: false
297300
summary-always: false
@@ -304,6 +307,61 @@ jobs:
304307
benchmark-data-dir-path: wiki/stats/benchmark/
305308
auto-push: false
306309

310+
- name: "⬇️ Get benchmark Artifacts (artificial, tree-sitter)"
311+
uses: actions/download-artifact@v4
312+
with:
313+
name: benchmark-results-artificial-tree-sitter
314+
path: benchmark-ai-tree-sitter/
315+
316+
- name: "🥄 Merge and Produce Performance Results (artificial, tree-sitter)"
317+
uses: benchmark-action/github-action-benchmark@v1
318+
with:
319+
name: '"artificial" Benchmark Suite (tree-sitter)'
320+
tool: 'customSmallerIsBetter'
321+
output-file-path: benchmark-ai-tree-sitter/artificial/artificial-summarized-graph.json
322+
github-token: ${{ secrets.GITHUB_TOKEN }}
323+
fail-on-alert: false
324+
summary-always: false
325+
comment-on-alert: true
326+
comment-always: true
327+
skip-fetch-gh-pages: true
328+
max-items-in-chart: 100
329+
ref: "refs/heads/main"
330+
gh-repository: ${{ github.repository }}
331+
benchmark-data-dir-path: wiki/stats/benchmark/
332+
auto-push: false
333+
334+
- name: "⬇️ Get benchmark Artifacts (social-science, tree-sitter)"
335+
uses: actions/download-artifact@v4
336+
with:
337+
name: benchmark-results-social-science-tree-sitter
338+
path: benchmark-ss-tree-sitter
339+
340+
- name: "🥄 Merge and Produce Performance Results (social-science, tree-sitter)"
341+
uses: benchmark-action/github-action-benchmark@v1
342+
with:
343+
name: '"social-science" Benchmark Suite (tree-sitter)'
344+
tool: 'customSmallerIsBetter'
345+
output-file-path: benchmark-ss-tree-sitter/social-science/social-science-summarized-graph.json
346+
github-token: ${{ secrets.GITHUB_TOKEN }}
347+
fail-on-alert: false
348+
summary-always: false
349+
comment-on-alert: true
350+
comment-always: true
351+
skip-fetch-gh-pages: true
352+
max-items-in-chart: 100
353+
ref: "refs/heads/main"
354+
gh-repository: ${{ github.repository }}
355+
benchmark-data-dir-path: wiki/stats/benchmark/
356+
auto-push: false
357+
358+
- name: "⬆️ Upload Benchmark Graph Data"
359+
uses: actions/upload-artifact@v4
360+
with:
361+
name: benchmark-results-graph-data
362+
path: wiki/stats/benchmark/
363+
retention-days: 1
364+
307365
deploy-doc:
308366
needs: [ test, performance-test ]
309367
name: "🚀 Build and Deploy Documentation (only on main)"

‎.github/workflows/release.yaml‎

Lines changed: 67 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -228,12 +228,13 @@ jobs:
228228
startsWith(github.event.head_commit.message, '[release:major]') ||
229229
startsWith(github.event.head_commit.message, '[release:patch]')
230230
runs-on: ubuntu-22.04
231-
timeout-minutes: 400
231+
timeout-minutes: 60
232232
strategy:
233233
fail-fast: true
234234
matrix:
235235
# currently we have a duplication, add your benchmark names here and in `performance-test-upload`
236236
name: ['artificial', 'social-science']
237+
parser: ["tree-sitter", "r-shell"]
237238
steps:
238239
- name: "🛒 Checkout Repository"
239240
uses: actions/checkout@v4
@@ -260,12 +261,12 @@ jobs:
260261
run: install.packages("xmlparsedata", repos="https://cloud.r-project.org/")
261262

262263
- name: "⏱️ Run the performance benchmarks"
263-
run: bash .github/workflows/scripts/run-flowr-command.sh performance-test -- 1 1 "${{ matrix.name }}"
264+
run: bash .github/workflows/scripts/run-flowr-command.sh performance-test -- 1 1 "${{ matrix.name }}" "${{ matrix.parser }}"
264265

265266
- name: "⬆️ Upload Benchmark Results"
266267
uses: actions/upload-artifact@v4
267268
with:
268-
name: benchmark-results-${{ matrix.name }}
269+
name: benchmark-results-${{ matrix.name }}-${{ matrix.parser }}
269270
path: test/performance/results/
270271
outputs:
271272
version: ${{ needs.release.outputs.version }}
@@ -284,27 +285,27 @@ jobs:
284285
# check out the release ref so that the upload action finds the commit
285286
ref: "refs/tags/v${{ needs.performance-test.outputs.version }}"
286287

287-
- name: "⬇️ Get benchmark Artifacts (artificial)"
288-
uses: actions/download-artifact@v4
289-
with:
290-
name: benchmark-results-artificial
291-
path: benchmark-ai/
292-
293288
- name: "⚙️ Name and Email for Git (config) and update"
294289
# we fetch just to be totally sure
295290
run: |
296291
git config --local user.email "[email protected]"
297292
git config --local user.name "GitHub Action"
298293
git fetch --tags
299294
295+
- name: "⬇️ Get benchmark Artifacts (artificial, r-shell)"
296+
uses: actions/download-artifact@v4
297+
with:
298+
name: benchmark-results-artificial-r-shell
299+
path: benchmark-ai-r-shell/
300+
300301
# currently we have a duplication, add your benchmark names here and in `performance-test` and check the qa.yaml
301302
# Furthermore, you have to update the git reset command when pushing!
302-
- name: "🥄 Merge and Produce Performance Results (artificial)"
303+
- name: "🥄 Merge and Produce Performance Results (artificial, r-shell)"
303304
uses: benchmark-action/github-action-benchmark@v1
304305
with:
305306
name: '"artificial" Benchmark Suite'
306307
tool: 'customSmallerIsBetter'
307-
output-file-path: benchmark-ai/artificial/artificial-summarized-graph.json
308+
output-file-path: benchmark-ai-r-shell/artificial/artificial-summarized-graph.json
308309
github-token: ${{ secrets.RELEASE_TOKEN }}
309310
fail-on-alert: false
310311
summary-always: false
@@ -317,18 +318,68 @@ jobs:
317318
benchmark-data-dir-path: wiki/stats/benchmark/
318319
auto-push: false
319320

320-
- name: "⬇️ Get benchmark Artifacts (social-science)"
321+
- name: "⬇️ Get benchmark Artifacts (social-science, r-shell)"
321322
uses: actions/download-artifact@v4
322323
with:
323-
name: benchmark-results-social-science
324-
path: benchmark-ss/
324+
name: benchmark-results-social-science-r-shell
325+
path: benchmark-ss-r-shell/
325326

326-
- name: "🥄 Merge and Produce Performance Results (social-science)"
327+
- name: "🥄 Merge and Produce Performance Results (social-science, r-shell)"
327328
uses: benchmark-action/github-action-benchmark@v1
328329
with:
329330
name: '"social-science" Benchmark Suite'
330331
tool: 'customSmallerIsBetter'
331-
output-file-path: benchmark-ss/social-science/social-science-summarized-graph.json
332+
output-file-path: benchmark-ss-r-shell/social-science/social-science-summarized-graph.json
333+
github-token: ${{ secrets.RELEASE_TOKEN }}
334+
fail-on-alert: false
335+
summary-always: false
336+
comment-on-alert: true
337+
comment-always: true
338+
skip-fetch-gh-pages: true
339+
max-items-in-chart: 100
340+
ref: "refs/tags/v${{ needs.performance-test.outputs.version }}"
341+
gh-repository: ${{ github.repository }}
342+
benchmark-data-dir-path: wiki/stats/benchmark/
343+
auto-push: false
344+
345+
- name: "⬇️ Get benchmark Artifacts (artificial, tree-sitter)"
346+
uses: actions/download-artifact@v4
347+
with:
348+
name: benchmark-results-artificial-tree-sitter
349+
path: benchmark-ai-tree-sitter/
350+
351+
# currently we have a duplication, add your benchmark names here and in `performance-test` and check the qa.yaml
352+
# Furthermore, you have to update the git reset command when pushing!
353+
- name: "🥄 Merge and Produce Performance Results (artificial, tree-sitter)"
354+
uses: benchmark-action/github-action-benchmark@v1
355+
with:
356+
name: '"artificial" Benchmark Suite (tree-sitter)'
357+
tool: 'customSmallerIsBetter'
358+
output-file-path: benchmark-ai-tree-sitter/artificial/artificial-summarized-graph.json
359+
github-token: ${{ secrets.RELEASE_TOKEN }}
360+
fail-on-alert: false
361+
summary-always: false
362+
comment-on-alert: true
363+
comment-always: true
364+
skip-fetch-gh-pages: true
365+
max-items-in-chart: 100
366+
ref: "refs/tags/v${{ needs.performance-test.outputs.version }}"
367+
gh-repository: ${{ github.repository }}
368+
benchmark-data-dir-path: wiki/stats/benchmark/
369+
auto-push: false
370+
371+
- name: "⬇️ Get benchmark Artifacts (social-science, tree-sitter)"
372+
uses: actions/download-artifact@v4
373+
with:
374+
name: benchmark-results-social-science-tree-sitter
375+
path: benchmark-ss-tree-sitter/
376+
377+
- name: "🥄 Merge and Produce Performance Results (social-science, tree-sitter)"
378+
uses: benchmark-action/github-action-benchmark@v1
379+
with:
380+
name: '"social-science" Benchmark Suite (tree-sitter)'
381+
tool: 'customSmallerIsBetter'
382+
output-file-path: benchmark-ss-tree-sitter/social-science/social-science-summarized-graph.json
332383
github-token: ${{ secrets.RELEASE_TOKEN }}
333384
fail-on-alert: false
334385
summary-always: false

‎.github/workflows/scripts/run-flowr-command.sh‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ if [[ -z "${1:-}" ]]; then
2626
fi
2727

2828
ACTION=$1
29+
OTHER_ARGS=${*:2}
2930

3031
set -eu
3132

@@ -93,7 +94,7 @@ if [ "$ACTION" == "doc" ]; then
9394
fi
9495

9596
group "Run action $ACTION"
96-
$NPM_CMD run $ACTION
97+
$NPM_CMD run $ACTION $OTHER_ARGS
9798
end_group
9899

99100
if [ "$ACTION" == "doc" ]; then

‎README.md‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,4 +73,3 @@ We welcome every contribution! Please check out the [contributing guidelines](ht
7373
[GPLv3 License](LICENSE).
7474

7575
----
76-
# abstract-interpretation-ltx

‎package-lock.json‎

Lines changed: 7 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)