Skip to content

Refactor web application and other infra fixes #83

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
MBkkt wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Refactor web application and other infra fixes #83

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
*.iml
.idea
outputs
results
results.json
benchmark/target
tantivy/target
lucene/build
Expand All @@ -25,3 +25,6 @@ corpus.json
engines/**/.project
engines/**/.classpath
engines/**/.settings
.vscode/
results-/
results/
12 changes: 8 additions & 4 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,16 @@ export

WIKI_SRC = "https://www.dropbox.com/s/wwnfnu441w1ec9p/wiki-articles.json.bz2"

COMMANDS ?= TOP_100_COUNT TOP_100 COUNT
COMMANDS ?= TOP_100_COUNT TOP_100 COUNT

# ENGINES ?= tantivy-0.13 lucene-8.4.0 pisa-0.8.2 rucene-0.1 bleve-0.8.0-scorch rucene-0.1 tantivy-0.11 tantivy-0.14 tantivy-0.15 tantivy-0.16 tantivy-0.17 tantivy-0.18 tantivy-0.19
# ENGINES ?= tantivy-0.16 lucene-8.10.1 pisa-0.8.2 bleve-0.8.0-scorch bluge-0.2.2 rucene-0.1
# ENGINES ?= tantivy-0.16 tantivy-0.17 tantivy-0.18 tantivy-0.19
ENGINES ?= tantivy-0.22 tantivy-0.24 tantivy-0.25 tantivy-main lucene-10.3.0 lucene-10.3.0-bp
PORT ?= 8080
WARMUP_TIME ?= 60
NUM_ITER ?= 10
SERVE_TYPE ?= serve

help:
@grep '^[^#[:space:]].*:' Makefile
Expand All @@ -32,15 +34,17 @@ index:

bench:
@echo "--- Benchmarking ---"
@rm -fr results
Copy link
Collaborator

@PSeitz PSeitz Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why was this removed?

Copy link
Author

@MBkkt MBkkt Mar 20, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because in the PR it works this way:
ENIGNES=1 3 make bench => produce files engines/1/results/count.json, engines/3/results/count.json, etc and <benchmark-root>/results.json. Last file is needed to keep old behavior.

Also make merge merge all engines results/ into new <benchmakr-root>/results.json.

So there's no results dir in the root

@mkdir results
@python3 src/client.py queries.txt $(ENGINES)

merge:
@echo "--- Merging results ---"
@python3 src/client.py --merge

compile:
@echo "--- Compiling binaries ---"
@for engine in $(ENGINES); do cd ${shell pwd}/engines/$$engine && make compile ; done

serve:
@echo "--- Serving results ---"
@cp results.json web/build/results.json
@cd web/build && python3 -m http.server $(PORT)
@cd web/build && python3 ../../web/serve.py $(PORT)
Copy link
Author

@MBkkt MBkkt Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just to serve gzip file instead of full large json (I sometimes had slow internet connection between me and the server, so it was fix suggest by claude and it actually allows to download less data (500kb vs 3.5 MB) when open this web view).

Loading