Running ecraven's benchmarks

[amirouche]

What about porting ecraven's benchmarks and run them using the containers found in this project so that it is easy for anyone to reproduce the results?

[lassik]

Sounds excellent. That benchmark suite is impressive but haven't had time to try and run it. Do you know how to hook it up to our containers?

[amirouche]

It seems straight forward to run the tests with docker. I will look into it.

[amirouche]

ref: https://docs.docker.com/storage/bind-mounts/#start-a-container-with-a-bind-mount

[amirouche]

After cloning the benchmark repository, change directory inside it then:

% docker run -it --mount type=bind,source="$(pwd)",target=/benchmark schemers/chibi bash
$ cd /benchmark/
$ ./bench chibi all

Testing browse under Chibi
Including prelude /benchmark/src/Chibi-prelude.scm
Including postlude /benchmark/src/Chibi-postlude.scm
Compiling...
chibi_comp /tmp/larcenous/Chibi/browse.scm /tmp/larcenous/Chibi/browse.scm
Running...
WARNING: importing undefined variable: chibi-version
Running browse:2000

...

[amirouche]

That is the idea. I tried chicken first, but it seems like the docker container is missing the command chicken-csc or chicken5 has changed its command line. Anyway, I will further investigate.

[lassik]

it seems like the docker container is missing the command chicken-csc

Almost all of our containers should install the Scheme implementation under /usr/local. Try looking in /usr/local/bin. The Chicken compiler may be simply csc.

[amirouche]

The Chicken compiler may be simply csc.

That is exactly the case.

[lassik]

Bind mounts are one way, or you can do git clone --depth 1 or wget/curl https://github.com/ecraven/r7rs-benchmarks/archive/master.tar.gz inside the container. Bind mounts are find for local development, but may not be easy to use reliably in production.

Most of the containers are based on debian, so apt-get update && apt-get install -y curl works to install packages.

[amirouche]

Based on my investigations, we need to fork the benchmarks to adapt it.

What do you think @ecraven?

[ecraven]

Based on my investigations, we need to fork the benchmarks to adapt it. What do you think @ecraven?

I've been planning to containerise things for a long time, but haven't had the time. I've also started work on a chibi scheme test runner, that correctly captures compilation time, stdout, stderr, and so on, but that isn't close to finishing either. I won't be able to finish these projects anytime soon, so feel free to give it a go. One thing to keep in mind is that containers *do* have a performance hit, so there will be people who think that benchmarks run inside a container are worse than those run on bare metal ;) Greetings, Peter