Skip to content

Commit fd91fa8

Browse files
SchmlukSchmluk
authored
Feature/introspection (#63)
* start by setting up settings * update readme * add testing infrastructure * add introspection arg parsing * more infrastructure * add infrastructure for introspection * WIP: parse substitutions * implement introspection logging for CLI parsing * some initial docs bulletpoints * add programmatic tracking and tests * more descriptions * refactor data structures * WIP: render history tests * refactor merge logic to work for larger trees * finalize implementation of set introspection * fix up introspection get and tests * WIP: First stab at viewer infrastructure * WIP: Virtual Config operator== and introspection utests * refactor virtual config visits and properly log type param as meta params * refactor introspection logging and rendering for clarity * implement proper getValues logging * fiddle around with an initial rendering * implement initial rendering pipeline * implement introspection viewer first version * fix and test defauls and lookups * update viewer and data * untrack data * update install and readme * fix bugs, docs, and tests * log more info and refactor display logic * add more rendering options and source lookup * update tests to new info * add more verbose info to GUI * address review comments * update docs * clean up dangling readme ref
1 parent 94a72fc commit fd91fa8

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

48 files changed

+3682
-501
lines changed

‎.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,3 +66,10 @@ Notes.txt
6666
tmp/
6767

6868
compile_commands.json
69+
70+
# Doxygen
71+
.doxygen/
72+
73+
# Introspection Data
74+
config_introspection_results/
75+
config_utilities/introspection_viewer/data.json

‎README.md‎

Lines changed: 104 additions & 85 deletions
Original file line numberDiff line numberDiff line change
@@ -1,125 +1,138 @@
11
![CMake: Build](https://github.com/MIT-SPARK/config_utilities/actions/workflows/cmake_build.yml/badge.svg)
22

33
# config_utilities
4-
`config_utilities` is a minimal but powerful C++ library, providing tools for automated modular software configuration. Parse, verify, and print C++ config structs and run-time configuration of object-oriented modular systems.
4+
5+
`config_utilities` is a minimal but powerful C++ library, providing tools for automated modular software configuration. Parse, verify, modify, and print C++ config structs to build modular object-oriented software systems at run-time.
56

67
## Table of contents
8+
79
- [Credits](#credits)
810
- [Why `config_utilities`?](#why-config_utilities)
911
- [Installation](#installation)
1012
- [How to `config_utilities`?](#how-to-config_utilities)
1113
- [Example Projects using `config_utilities`](#example-projects-using-config_utilities)
1214

1315
## Credits
16+
1417
This library was developed by [Lukas Schmid](https://schmluk.github.io/) and [Nathan Hughes](http://mit.edu/sparklab/people.html) at the [MIT-SPARK Lab](http://mit.edu/sparklab), based on functionalities in [ethz-asl/config_utilities](https://github.com/ethz-asl/config_utilities) and [Hydra](https://github.com/MIT-SPARK/Hydra), and is released under a [BSD-3-Clause License](LICENSE)! Additional contributions welcome! This work was supported in part by the Swiss National Science Foundation and Amazon.
1518

1619
## Why `config_utilities`?
20+
1721
Among many other, the key features of `config_utilities` include:
22+
1823
- **Minimal dependencies**: Only C++17 standard library and [yaml-cpp](https://github.com/jbeder/yaml-cpp).
1924
- Declare **any struct a config**, also from external projects:
20-
```c++
21-
namespace external_project {
22-
void declare_config(ExternalObject& config); // that's all!
23-
} // namespace external_project
24-
```
25+
```c++
26+
namespace external_project {
27+
void declare_config(ExternalObject& config); // that's all!
28+
} // namespace external_project
29+
```
2530
- **Minimal** and **clear, human readable interfaces** for config definitions:
26-
```c++
27-
void declare_config(MyConfig& config) {
28-
using namespace config;
29-
name("MyConfig"); // Name for printing.
30-
field(config.distance, "distance", "m"); // Field definition.
31-
check(config.distance, GT, 0.0, "distance"); // Ensure 'distance > 0'.
32-
}
33-
```
31+
```c++
32+
void declare_config(MyConfig& config) {
33+
using namespace config;
34+
name("MyConfig"); // Name for printing.
35+
field(config.distance, "distance", "m"); // Field definition.
36+
check(config.distance, GT, 0.0, "distance"); // Ensure 'distance > 0'.
37+
}
38+
```
3439
- **Everything** related to a config is defined in a **single place**.
3540
```c++
3641
void declare_config(MyConfig& config) { /* ALL the information about a config is here */ }
3742
```
3843
- Parse any declared config from **various data sources**, **without pulling in dependencies** on data sources into core libraries:
39-
```c++
40-
core.cpp {
41-
struct MyConfig { ... };
42-
void declare_config(MyConfig& config) { ... };
43-
} // Depends only on C++ and yaml-cpp.
44-
45-
application.cpp {
46-
MyConfig config1 = fromYamlFile<MyConfig>(file_path);
47-
MyConfig config2 = fromRos<MyConfig>(node_handle);
48-
} // Pull in dependencies as needed.
49-
```
44+
45+
```c++
46+
core.cpp {
47+
struct MyConfig { ... };
48+
void declare_config(MyConfig& config) { ... };
49+
} // Depends only on C++ and yaml-cpp.
50+
51+
application.cpp {
52+
MyConfig config1 = fromYamlFile<MyConfig>(file_path);
53+
MyConfig config2 = fromRos<MyConfig>(node_handle);
54+
} // Pull in dependencies as needed.
55+
```
56+
5057
- **Automatically implements** frequently used functionalities, such as **checking** for valid parameters and **formatting** to string:
51-
```c++
52-
MyObject(const MyConfig& config) : config_(checkValid(config)) {
53-
// Do magic with config_, which has valid parameters only.
54-
std::cout << config_ << std::endl;
55-
}
56-
```
58+
```c++
59+
MyObject(const MyConfig& config) : config_(checkValid(config)) {
60+
// Do magic with config_, which has valid parameters only.
61+
std::cout << config_ << std::endl;
62+
}
63+
```
5764
- Automatic **run-time module creation** based on specified configurations:
58-
```cpp
59-
static auto registration = Registration<Base, MyDerived>("my_derived");
6065
61-
const std::string module_type = "my_derived";
62-
std::unique_ptr<Base> module = create(module_type);
63-
```
66+
```cpp
67+
static auto registration = Registration<Base, MyDerived>("my_derived");
68+
69+
const std::string module_type = "my_derived";
70+
std::unique_ptr<Base> module = create(module_type);
71+
```
72+
6473
- **Informative, clear**, and **human readable** printing of potentially complicated config structs:
65-
```
66-
=================================== MyConfig ===================================
67-
distance [m]: -0.9876
68-
A ridiculously long field name that will not be wrapped: Short Value (default)
69-
A ridiculously long field name that will also not be wrapped:
70-
A really really really ridiculously long string th
71-
at will be wrapped. (default)
72-
A really really really really really really ridiculously long field name that wi
73-
ll be wrapped: A really really really ridiculously long string th
74-
at will also be wrapped. (default)
75-
vec: [5, 4, 3, 2, 1]
76-
map: {b: 2, c: 3, d: 4, e: 5}
77-
mat: [[1, 2, 3],
78-
[4, 5, 6],
79-
[7, 8, 9]]
80-
my_enum: B
81-
sub_config [SubConfig] (default):
82-
f: 0.123 (default)
83-
s: test (default)
84-
sub_sub_config [SubSubConfig] (default):
85-
color: [255, 127, 0] (default)
86-
size: 5 (default)
87-
================================================================================
88-
```
74+
75+
```
76+
=================================== MyConfig ===================================
77+
distance [m]: -0.9876
78+
A ridiculously long field name that will not be wrapped: Short Value (default)
79+
A ridiculously long field name that will also not be wrapped:
80+
A really really really ridiculously long string th
81+
at will be wrapped. (default)
82+
A really really really really really really ridiculously long field name that wi
83+
ll be wrapped: A really really really ridiculously long string th
84+
at will also be wrapped. (default)
85+
vec: [5, 4, 3, 2, 1]
86+
map: {b: 2, c: 3, d: 4, e: 5}
87+
mat: [[1, 2, 3],
88+
[4, 5, 6],
89+
[7, 8, 9]]
90+
my_enum: B
91+
sub_config [SubConfig] (default):
92+
f: 0.123 (default)
93+
s: test (default)
94+
sub_sub_config [SubSubConfig] (default):
95+
color: [255, 127, 0] (default)
96+
size: 5 (default)
97+
================================================================================
98+
```
8999

90100
- **Verbose warnings and errors** if desired for clear and easy development and use:
91-
```
92-
=================================== MyConfig ===================================
93-
Warning: Check failed for 'distance': param > 0 (is: '-1').
94-
Warning: Check failed for 'subconfig.f': param within [0, 100) (is: '123').
95-
Warning: Failed to parse param 'vec': Data is not a sequence.
96-
Warning: Failed to parse param 'mat': Incompatible Matrix dimensions: Requested
97-
3x3 but got 3x4.
98-
Warning: Failed to parse param 'my_enum': Name 'D' is out of bounds for enum wit
99-
h names ['A', 'B', 'C'].
100-
Warning: Failed to parse param 'uint': Value '-1' underflows storage min of '0'.
101-
================================================================================
102-
103-
[ERROR] No module of type 'NotRegistered' registered to the factory for BaseT='d
104-
emo::Base' and ConstructorArguments={'int'}. Registered are: 'DerivedB', 'Derive
105-
dA'.
106-
107-
[ERROR] Cannot create a module of type 'DerivedA': No modules registered to the
108-
factory for BaseT='demo::Base' and ConstructorArguments={'int', 'float'}. Regist
109-
er modules using a static config::Registration<BaseT, DerivedT, ConstructorArgum
110-
ents...> struct.
111-
```
101+
102+
```
103+
=================================== MyConfig ===================================
104+
Warning: Check failed for 'distance': param > 0 (is: '-1').
105+
Warning: Check failed for 'subconfig.f': param within [0, 100) (is: '123').
106+
Warning: Failed to parse param 'vec': Data is not a sequence.
107+
Warning: Failed to parse param 'mat': Incompatible Matrix dimensions: Requested
108+
3x3 but got 3x4.
109+
Warning: Failed to parse param 'my_enum': Name 'D' is out of bounds for enum wit
110+
h names ['A', 'B', 'C'].
111+
Warning: Failed to parse param 'uint': Value '-1' underflows storage min of '0'.
112+
================================================================================
113+
114+
[ERROR] No module of type 'NotRegistered' registered to the factory for BaseT='d
115+
emo::Base' and ConstructorArguments={'int'}. Registered are: 'DerivedB', 'Derive
116+
dA'.
117+
118+
[ERROR] Cannot create a module of type 'DerivedA': No modules registered to the
119+
factory for BaseT='demo::Base' and ConstructorArguments={'int', 'float'}. Regist
120+
er modules using a static config::Registration<BaseT, DerivedT, ConstructorArgum
121+
ents...> struct.
122+
```
112123

113124
## Installation
114125

115126
This package is compatible with `ROS2`/`colcon`. Just clone it into your workspace and you should be all set!
127+
116128
```bash
117129
cd ~/my_ws/src
118130
git clone [email protected]:MIT-SPARK/config_utilities.git
119131
colcon build config_utilities_ros
120132
```
121133

122134
If you want to build and install without colcon/ROS, that is easy, too! Just clone this repository and build via CMake:
135+
123136
```bash
124137
git clone [email protected]:MIT-SPARK/config_utilities.git
125138
cd config_utilities/config_utilities
@@ -133,9 +146,11 @@ sudo make install
133146
```
134147

135148
## How to `config_utilities`
149+
136150
We provide detailed introductions about everything you need to know about `config_utilities` in the following [tutorials](docs/README.md#tutorials) and some verbose example [demos](docs/README.md#demos) that you can run.
137151

138152
The (non-ros) demos can be run via the `run_demo.py` utility in the scripts directory. If you are building this library via catkin, you can run one of the following to see the results of one of the corresponding demo files:
153+
139154
```
140155
python3 scripts/run_demo.py config
141156
python3 scripts/run_demo.py inheritance
@@ -146,20 +161,24 @@ python3 scripts/run_demo.py factory
146161
> If you're building via cmake, you can point `run_demo.py` to the build directory with `-b/--build_path`.
147162
148163
The ros demo for dynamic config updates can be run via:
164+
149165
```
150166
ros2 launch config_utilities_ros demo_ros_dynamic_config.yaml
151167
```
152168

153169
If you are looking for a specific use case that is not in the tutorials or demos, chances are you can find a good example in the `tests/` directory!
154170

155-
156171
# Example Projects using `config_utilities`
172+
157173
Many cool projects are already using `config_utilities`! If you are using `config_utilities` and would like to be featured, please open a pull request or reach out!
158174

159175
### Academic Open-source Projects
176+
160177
These projects are openly available, check them out to see what `config_utilities` can do!
161-
* [MIT-SPARK/Khronos](https://github.com/MIT-SPARK/Khronos)
162-
* [MIT-SPARK/Hydra](https://github.com/MIT-SPARK/Hydra)
178+
179+
- [MIT-SPARK/Khronos](https://github.com/MIT-SPARK/Khronos)
180+
- [MIT-SPARK/Hydra](https://github.com/MIT-SPARK/Hydra)
163181

164182
### Use in Industry
165-
Several leading companies and start-ups in robotics have let us know that they use `config_utilities`, but we cannot currently list them.
183+
184+
Several leading companies and start-ups in robotics have let us know that they use `config_utilities`, but we cannot currently list them.

‎config_utilities/CMakeLists.txt‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ find_package(yaml-cpp REQUIRED)
1616
find_package(Boost CONFIG REQUIRED COMPONENTS filesystem system)
1717
find_package(Eigen3 QUIET)
1818
find_package(glog QUIET)
19+
find_package(nlohmann_json QUIET)
1920

2021
add_library(
2122
${PROJECT_NAME}
@@ -29,6 +30,7 @@ add_library(
2930
src/factory.cpp
3031
src/formatter.cpp
3132
src/field_input_info.cpp
33+
src/introspection.cpp
3234
src/log_to_stdout.cpp
3335
src/log_to_glog.cpp
3436
src/logger.cpp
@@ -62,6 +64,10 @@ if(glog_FOUND)
6264
target_link_libraries(${PROJECT_NAME} PUBLIC glog::glog)
6365
target_compile_definitions(${PROJECT_NAME} PRIVATE CONFIG_UTILS_ENABLE_GLOG_LOGGING)
6466
endif()
67+
if(nlohmann_json_FOUND)
68+
target_link_libraries(${PROJECT_NAME} PRIVATE nlohmann_json::nlohmann_json)
69+
target_compile_definitions(${PROJECT_NAME} PRIVATE CONFIG_UTILS_ENABLE_JSON)
70+
endif()
6571

6672
add_executable(composite-configs app/composite_configs.cpp)
6773
target_link_libraries(composite-configs ${PROJECT_NAME})
@@ -84,6 +90,10 @@ install(
8490
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
8591
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
8692
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})
93+
install(DIRECTORY introspection_viewer/ DESTINATION ${CMAKE_INSTALL_BINDIR})
94+
install(PROGRAMS introspection_viewer/introspection_viewer.py
95+
RENAME config-utilities-viewer
96+
DESTINATION ${CMAKE_INSTALL_BINDIR})
8797
install(DIRECTORY include/ DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
8898
install(
8999
EXPORT config_utilities-targets
@@ -107,6 +117,6 @@ install(FILES ${CMAKE_CURRENT_BINARY_DIR}/config_utilitiesConfig.cmake
107117

108118
# Exposes this package to ament for ros2 run and other utilities to work
109119
find_package(ament_cmake_core QUIET)
110-
if (${ament_cmake_core_FOUND})
120+
if (${ament_cmake_core_FOUND})
111121
ament_package()
112122
endif()

‎config_utilities/app/composite_configs.cpp‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,9 @@ Invalid YAML or missing files get dropped during compositing.
2222
-d/--disable-substitutions: Turn off substitutions resolution
2323
--no-disable-substitutions: Turn substitution resolution on (currently on by default)
2424
--force-block-style: Force emitted YAML to only use block style
25+
-i/--config-utilities-introspect [OUTPUT_DIR]: Enable introspection and output results to OUTPUT_DIR.
26+
If OUTPUT_DIR is not specified, defaults to 'introspection_results'.
27+
Note that if the directory already exists, it will be overwritten.
2528
2629
Example:
2730
> echo "{a: 42, bar: hello}" > /tmp/test_in.yaml

‎config_utilities/include/config_utilities/factory.h‎

Lines changed: 22 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@
4141
#include <string>
4242
#include <vector>
4343

44+
#include "config_utilities/internal/introspection.h"
4445
#include "config_utilities/internal/logger.h"
4546
#include "config_utilities/internal/string_utils.h"
4647
#include "config_utilities/internal/visitor.h"
@@ -257,7 +258,6 @@ class ModuleRegistry {
257258

258259
template <typename BaseT, typename ConfigT>
259260
static void registerConfig(const std::string& type) {
260-
// NOTE(lschmid): This is not forbidden behavior, but is not recommended so for now simply warn the user.
261261
const auto key = ConfigPair::fromTypes<BaseT, ConfigT>();
262262
auto& registry = instance().config_registry;
263263
auto iter = registry.find(key);
@@ -266,6 +266,7 @@ class ModuleRegistry {
266266
return;
267267
}
268268

269+
// NOTE(lschmid): This is not forbidden behavior, but is not recommended so for now simply warn the user.
269270
if (iter != registry.end() && iter->second != type) {
270271
Logger::logInfo("Overwriting type name for config '" + typeName<ConfigT>() + "' for base module '" +
271272
typeName<BaseT>() + "' from '" + iter->second + "' to '" + type +
@@ -427,9 +428,21 @@ struct ObjectWithConfigFactory {
427428
// Add entries.
428429
template <typename DerivedT, typename DerivedConfigT>
429430
static void addEntry(const std::string& type) {
430-
const Constructor method = [](const YAML::Node& data, Args... args) -> BaseT* {
431+
const Constructor method = [type](const YAML::Node& data, Args... args) -> BaseT* {
431432
DerivedConfigT config;
432-
Visitor::setValues(config, data);
433+
auto set_data = Visitor::setValues(config, data);
434+
if (Settings::instance().introspection.enabled()) {
435+
auto get_info = Visitor::getInfo(config);
436+
// Also log the type parameter used for creation.
437+
auto& set_field = set_data.field_infos.emplace_back();
438+
set_field.name = Settings::instance().factory.type_param_name;
439+
set_field.value = type;
440+
set_field.is_meta_field = true;
441+
auto& get_field = get_info.field_infos.emplace_back();
442+
get_field = set_field;
443+
set_field.was_parsed = true;
444+
Introspection::logSetValue(set_data, get_info);
445+
}
433446
return new DerivedT(config, std::move(args)...);
434447
};
435448

@@ -443,6 +456,12 @@ struct ObjectWithConfigFactory {
443456
static std::unique_ptr<BaseT> create(const YAML::Node& data, Args... args) {
444457
std::string type;
445458
if (!getType(data, type)) {
459+
if (Settings::instance().introspection.enabled()) {
460+
// Log that the type could not be determined.
461+
Introspection::logSingleEvent(
462+
Introspection::Event(Introspection::Event::Type::GetError, Introspection::By::config("Factory::create()")),
463+
Settings::instance().factory.type_param_name);
464+
}
446465
return nullptr;
447466
}
448467

0 commit comments

Comments
 (0)