Feature Discussion: container configuration system

[katiewasnothere]

Problem statement

We're looking to improve the user experience and overall functionality of setting defaults for container to better enable future use cases.

Today container uses macOS's UserDefaults to configure settings needed at runtime. As mentioned in #608, UserDefaults may be idiomatic for macOS, but they apply globally to all sessions and do not handle representing complex, hierarchical data well. Additionally, we currently have two ways of setting these defaults, either directly with macOS's defaults command or through container system property.

#608 proposes moving to use environment variables in place of UserDefaults. However, we do not believe this is sufficient. Environment variables are not in a consistent location, are not sourced from data, and also do not handle representing complex, hierarchical data well.

What should be configurable?

There are two different components of the container project that need configurable defaults: core services and plugins.

Core services

There are various core services that are run by container in the APIServer, which is started on container system start. The APIServer itself and/or these core services it starts may have default configuration or setup that a user may wish to customize.

For example, when container is started, a default network is automatically created here by the APIServer. Today, the configuration for that default network is hardcoded into the code above. However, a user may wish to use a different network plugin for this default network or may want multiple default networks created when the APIServer is started.

Plugins

container uses a plugin architecture for extending the CLI surface and functionality. Users may wish to define default configurations for the various plugins they use to better suit their use cases.

For example, when a request is made to container to create a new network, the NetworksService, one of the core services on the APIServer, will start a new instance of the requested network plugin, such as the container-network-vmnet plugin. The container-network-vmnet plugin uses UserDefaults here to determine the default IP subnet to use if none is provided. While this is sufficient for the time being, if a user wanted to use multiple network plugins over the lifetime of container, they may wish to configure defaults specific to each plugin to set differing default IP subnets for each.

Design

We want to get feedback from the community on how we can design this to best fit users' needs. Right now we are looking into using TOML via TOMLDecoder for our configuration system.

Goals

• Default configurations should be sourced from data
• Default configurations should be configurable before starting container
• Default configurations should optionally be modifiable at runtime, as appropriate (TBD)
• Third party plugins should be able to define their own configurations independently of the container repo
• Use of UserDefaults should be removed entirely