initial commit

Author: klemense1

.bazelrc

@@ -0,0 +1,6 @@
+test --test_output=errors --action_env="GTEST_COLOR=1"
+
+# Force bazel output to use colors (good for jenkins) and print useful errors.
+common --color=yes
+
+build --cxxopt='-std=c++17' --define planner_rules_mcts=true --define ltl_rules=true

.github/workflows/CI.yml

@@ -0,0 +1,21 @@
+name: CI
+
+on:
+  push:
+  schedule:
+  - cron: "0 2 * * *"
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    container:
+      image: docker://barksim/bark:latest
+    steps:
+      - uses: actions/checkout@v1
+      - name: Setting up virtual environment
+        run: virtualenv -p python3 ./tools/python/venv  --system-site-packages
+      - name: Getting into venv
+        run: . ./tools/python/venv/bin/activate
+      - name: Runing merging_test_specific
+        run: bazel test //src/run:merging_test_specific

README.md

@@ -0,0 +1,61 @@
+# Example Benchmark
+
+Repository showing how to use BARK for research, e.g., conducting reproducible experiments. The core of this example repository is to run a benchmark of a single agent Monte Carlo Tree Search and compare it with MCTS variants, that incorporate penalties for violating traffic rules to their reward function. The study focuses on merging -- an arbitrary number of scenarios can be generated, where the initial positions and velocities of the vehicles are sampled randomly from a normal distribution.
+
+## Getting started:
+If you are not familar with BARK yet, check out our BARK paper [paper](https://arxiv.org/abs/2003.02604) and the code documentation [documentation](https://bark-simulator.readthedocs.io/en/latest/).
+This repository uses a virtual environment just as BARK does (install via `bash tools/python/setup_venv.sh` and then source it via `source tools/python/into_venv.sh`)
+
+## What is in there: 
+There are three targets ready for execution 
+* `merging_test_random`: runs N scenarios and sample the initial states randomly
+* `merging_test_specific`: runs a specific scenario (you can tune them)
+* `run_benchmark`: runs a full benchmark
+* `scenario_tuning`: runs a lightweight benchmark with visualization to find suitable scenario generation parameters
+
+## Benchmark
+First, execute `bazel run //src/run:run_benchmark`. The results of the benchmark will be saved at 
+`example_benchmark/bazel-bin/src/run/run_benchmark.runfiles/example_benchmark`. Copy the file to `results/benchmark`. The result file consists of a pandas dataframe, that represents the outcome of each benchmark simulation run. To visualize that, we have prepared an ipython notebook. You can start the notebook server via `bazel run //src/create_figures:run_notebooks` and then select the `plot_benchmark_results` notebook. The code will generate the following figure:
+
+![Benchmark Results](benchmark_results.png)
+
+### Changing Parameters:
+Parametrization in BARK is done via the `ParameterServer`, which reads json-files. You can tweak the existing ones:
+* for the viewer in `viewer_config/params/` 
+* for the behavior models in `mcts_config/params/`
+
+Of course, those json-files are quite nested, so we provide scripts to generate them with default values. 
+* for the viewer: `src/viewer_config/viewer_config_test.py`
+* for the behavior models: `src/mcts_config/mcts_config_test.py`
+The concept for creating parameter files can be transferred to any other object.
+
+### Evaluators:
+Evaluators can be used as a metric to analyze the benchmark, but also as a termination criterion for a simulation run. You can choose from any Evaluator in https://github.com/bark-simulator/bark/tree/master/bark/world/evaluation. Using EvaluatorLTL for example allows you to use a wide range of traffic rules.
+
+### Scenarios:
+The scenarios are generated based on the config files in `src/database/scenario_sets`. 
+
+## Dependency Management using Bazel
+As you can see in the [bark-simulator Github Group](https://github.com/bark-simulator/), the BARK ecosystem is split over multiple Github repositories. One reason for this was to keep the core functionalities light-weight and reasonably fast to build. Specifically, a lot of planning modules are placed in seperate repositories. Using Bazel as our build environment enables the reproducibility of our experiments, as dependency versions of the repositories can be tracked easily.
+
+For example, have a look to `tools/deps.bzl`, where the specific dependencies and either their commit hashs or a specific branch can be selected. In order to try two different versions of your planner (located in another repo), you do not build or install them manually, you just need to change the commit hash.
+
+## Cite us
+
+This repository contains work from multiple publications: 
+* Traffic Rules as Evaluators: [Formalizing Traffic Rules for Machine Interpretability](https://arxiv.org/abs/2007.00330)
+* MCTS with Traffic Rules: [Modeling and Testing Multi-Agent Traffic Rules within Interactive Behavior Planning](https://arxiv.org/abs/2009.14186)
+  
+If you use them, please cite them. 
+
+For everything else, please cite us using the following [paper](https://arxiv.org/abs/2003.02604):
+
+```
+@inproceedings{Bernhard2020,
+    title = {BARK: Open Behavior Benchmarking in Multi-Agent Environments},
+    author = {Bernhard, Julian and Esterle, Klemens and Hart, Patrick and Kessler, Tobias},
+    booktitle = {2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
+    url = {https://arxiv.org/pdf/2003.02604.pdf},
+    year = {2020}
+}
+```

WORKSPACE

@@ -0,0 +1,35 @@
+workspace(name = "example_benchmark")
+
+load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive", "http_file")
+load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
+load("//tools:deps.bzl", "example_benchmark_dependencies")
+
+example_benchmark_dependencies()
+
+load("@bark_project//tools:deps.bzl", "bark_dependencies")
+bark_dependencies()
+
+load("@com_github_nelhage_rules_boost//:boost/boost.bzl", "boost_deps")
+boost_deps()
+
+# -------- Benchmark Database -----------------------
+git_repository(
+    name = "benchmark_database",
+    commit = "422b0ddd316ab46ac79dcd72a45645e197cf7da1",
+    remote = "https://github.com/bark-simulator/benchmark-database",
+)
+
+load("@benchmark_database//util:deps.bzl", "benchmark_database_dependencies")
+benchmark_database_dependencies()
+
+load("@benchmark_database//load:load.bzl", "benchmark_database_release")
+benchmark_database_release()
+
+load("@rule_monitor_project//util:deps.bzl", "rule_monitor_dependencies")
+rule_monitor_dependencies()
+
+load("@planner_rules_mcts//util:deps.bzl", "planner_rules_mcts_dependencies")
+planner_rules_mcts_dependencies()
+
+load("@pybind11_bazel//:python_configure.bzl", "python_configure")
+python_configure(name = "local_config_python") 

benchmark_results.png

@@ -0,0 +1,5 @@
+filegroup(
+   name="benchmark_results",
+   srcs=glob(["*.zip"]),
+   visibility = ["//visibility:public"],
+)

results/BUILD

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c8934a9c77d8b59c40a22a4dcb40a7c2680a36fa3c736639e49631bb174f318b
+size 2063700

results/benchmark/BUILD

@@ -0,0 +1,6 @@
+py_library(
+  name = "custom_lane_corridor_config",
+  srcs = ["custom_lane_corridor_config.py"],
+  data = ["@bark_project//bark/python_wrapper:core.so"],
+  visibility = ["//visibility:public"],
+)