Questions about setting the path to the config file path on the command line in Kong

[pauluzee]

I am using Kong for the first time after having used other Go CLI toolkits (so far: Cobra and Go-flags).

Our team has an idiom for optionally specifying a config file path on the command line (always with --configFile or -c ). The expectation is that if the user sets the config file path on the command line, the app will use that value to locate the file and not check all of the other "standard" locations (e.g., the home dir, in the same directory as the executable, etc.). Also, if the user sets configFile with an invalid path, the app will exit with an error message.

If the user doesn't set configFile on the command line, however, then the app checks all of the standard locations and loads the first file it finds. Most of the time, it is not considered an error if the app doesn't find a config file, but I can see times when we would want the app to exit immediately with an error message when it can't find a config file.

In all cases we want to know exactly which file was loaded; to avoid confusion, most of our apps log the full, absolute path of the file that initialized the application.

I would like to use the configuration loading mechanism built into Kong, but I'm having a hard time figuring out how to pass a user-supplied path to it. I've written and tested a method that returns the "first valid" path to a config file, but it expects the user-supplied path as a parameter. If that's empty, then it iterates over the list of standard locations and returns the first one it finds. While I see how to use a "BeforeApply" method to run this sort of init stuff, it's not clear to me where inside of Kong I can get the optional value from the command line. Is there a method inside of the Kong instance that I can get the value of a specific flag (such as *configFile). In other words, how do I read the value of configFile from the command line before Kong locates and loads the config file? Also, how do I get the Kong configuration file built-in to exit the app if it can't find a file?

[alecthomas]

The built in config loading isn't really designed with that in mind. It might be simpler to roll your own with this extra feature rather than try to wedge it in.

[pauluzee]

Thanks

[pauluzee]

I'll roll my own, but how do I get the string value of the configFile flag into the BeforeApply method? I still need that to read the user-supplied config file used to pre-populate the struct before Kong fills in the values from the command line

[pauluzee]

Something that works like this?

func (g *Global) BeforeApply(app *kong.Kong, vars kong.Vars) error {
       userSuppliedConfigFile := kong.GetFlagValue("configFile") // ???
       return g.InitApp(userSuppliedConfigFile) 
}

[alecthomas]

In the scenario you described, I would expect Global to be the flag containing the optional config file and you would check it for a value?

[pauluzee]

Global is a struct that needs to be pre-populated with corresponding values from a config file before it gets populated from the environment and then the command line (the Global struct is based on a similar struct--''Globals"--in your docker example).

Typically our apps init a single "global" configuration struct following this order of steps:

1. A new Global struct is created with hard-coded or calculated defaults (usually from const vars or from calculated values that don't require any user input, e.g. the executable file location, the full hostname, etc). I got this to work in Kong using a BeforeApply method attached to the Global struct
2. An optional root-level string flag named "configFile" is mapped to a corresponding string var defined "early" in main(); we got this to work in Cobra, but--so far--not in Go-flags. We would much rather use something a lot less opaque than Cobra, however, and we really like the way Kong (and Go-flags) uses struct tags to interact with the command line parser
3. The app then locates the config file, starting with the user-supplied configFile value and then iterating over a list of pre-defined standard locations. Depending on the needs of the app, not being able to find a config file may or may not cause the app to exit with an error
4. The config file--if found--is scanned and any values it contains are used to populate the corresponding fields in the global config struct. This could result in some of the hard-coded defaults being overwritten
5. If it's defined for the app, the environment is then checked for corresponding variables that--if found--are used to populate the corresponding fields in the global config struct. This could result in some of the hard-coded defaults as well as those supplied by the config file being overwritten
6. Finally, the command line is checked and is used to populate the corresponding fields in the global config struct. This could result in some of the hard-coded defaults, those supplied by the config file and from the environment being overwritten

This results in a CLI design that follows a consistent order of configuration variable precedence (going from highest to lowest):

1. Command line input
2. Environmental vars
3. Config file settings (with the optional user-supplied path taking precedence over any standard locations)
4. Hard-coded or calculated defaults

So, if the user wants to use his own config file (instead of the ones in the pre-defined standard locations), there's needs to be a way that we can access the configFile command line parameter before the rest of the command line is used to finish populating the global struct. In other words, we are looking for a way to ask Kong for the value of the configFile flag before Kong finally applies the command line values to the global struct.

[alecthomas]

I would like to use the configuration loading mechanism built into Kong, but I'm having a hard time figuring out how to pass a user-supplied path to it.

Have you seen this? That, combined with ConfigFlag should do what you describe.

[alecthomas]

This results in a CLI design that follows a consistent order of configuration variable precedence (going from highest to lowest):

This is the precedence supported by Kong, so that's good news :)

[pauluzee]

Loading a User-Supplied Configuration File in Kong

The ConfigFlag type worked as expected.

In the global struct I have this field for the the user-supplied config file:

type Global struct {
   ConfigFile kong.ConfigFlag `json:"configFile" name:"configFile" short:"c" help:"Full path to a user-supplied config file"`
   ...
}

I created a function, GetStdCfgPaths, which returns a []str populated with the standard config file paths defined for the application. Normally, this would also return an error, but I can work around this with a slightly different startup routine I put ahead of the command line parsing.

Here's what main looks like now:

func main() {
	stdCfgPaths, err := GetStdCfgPaths()
	if err != nil {
		fmt.Printf("get standard config file error: %s", err.Error())
		os.Exit(1)
	}
	// Process the command line
	cli := CLI{}
	ctx := kong.Parse(&cli,
		kong.Configuration(kong.JSON, stdCfgPaths...),
		kong.Name("testKong2"),
		kong.Description("A second app to test kong"),
		kong.ConfigureHelp(kong.HelpOptions{
			Compact: true,
		}),
	)
	err = ctx.Run(&cli.Globals)
	ctx.FatalIfErrorf(err)
}

When I run the program with -c set to a valid path to an actual config file, it loads the file as expected. If I run it with -c set to an invalid path (i.e., a path to a missing file), then the program exits with an error message, which is also as expected.

If I don't set -c, then the program uses the values from GetStdCfgPaths() and loads whatever it finds. If none of the paths returned from GetStdCfgPaths() point to an existing file, the program assumes that's not an error and continues without loading a config file. While this is probably OK most of the time, there are some applications that should exit if a config file is not found.

Ideally, we would like some way to get back the path of the file that was actually loaded, and have the opportunity to exit if the value is empty. We like to log the path of whatever config file actually gets used; we've had some issues in the past where the config a program loaded was not the one we thought it did, and it set us onto a wild goose chase.

Variable Precedence in Kong

For the most part, the variable loading precedence works as I thought, with a few exceptions.

Test Program

I noticed that when I set default values in a struct using a BeforeApply method, it appears to "block" the corresponding environmental variables. Values from the config file or from the command line come through as expected. Also, it looks like configuration file variables have a higher precedence than environmental variables (I would expect it to be the other way around).

I wrote a test program to investigate this; here are the important parts:

// Hard-coded defaults
const (
	DefaultDir        = "[DefDir]"
	DefaultLogDir     = DefaultDir + "/log"
	DefaultRootDir    = DefaultDir + "/root"
)

// Globals is the top-level container for variables that are visible to all
// sub-commands of the application.
type Globals struct {
	ConfigFile kong.ConfigFlag `json:"configFile" name:"configFile" short:"c"                 help:"Full path to a user-supplied config file"`
	RootDir    string          `json:"rootDir"    name:"rootDir"    env:"TEST_KONG2_ROOT_DIR" help:"Root directory" default:"[DefTagDir]/root"`
	LogDir     string          `json:"logDir"     name:"logDir"     env:"TEST_KONG2_LOG_DIR"  help:"Log directory"  default:"[DefTagDir]/log"`
}

// BeforeApply initializes the Globals struct with default values.
func (g *Globals) BeforeApply(app *kong.Kong, vars kong.Vars) error {
	g.RootDir = DefaultRootDir
	g.LogDir = DefaultLogDir
	return nil
}

Env Vars

The environment is set like this:

export TEST_KONG2_LOG_DIR="[EnvDir]/log"
export TEST_KONG2_ROOT_DIR="[EnvDir]/root"

Config Files

The "standard" config file is set like this:

 {
 	 "rootDir":    "[StdCfgDir]/root",
 	 "logDir":     "[StdCfgDir]/log",
 }

The "custom" config file is set like this:

{
	 "rootDir":    "[CustomCfgDir]/root",
	 "logDir":     "[CustomCfgDir]/log",
}

Test Results

Scenario 1

When I run the program and set logDir on the command line to "[CliDir]/log", it dumps the contents of Globals to the console, like this:

  {
    "rootDir": "[StdCfgDir]/root", // set in std cfg file; should be "[EnvDir]/root"
    "logDir": "[CliDir]/log" // set on command line; correct
  }

This was not as expected: rootDir should have been set by the environment, not by the config file (the environment should have a higher precedence than the config file); logDir, which was set on the command line, is correct (variables set on the command line have the highest precedence).

Scenario 2

When I renamed the standard config file to a non-standard name (which means that no config file will be loaded) and ran the program the same way, it produced this output:

  {
    "rootDir": "[DefDir]/root", // set in BeforeApply(); should be "[EnvDir]/root"
    "logDir": "[CliDir]/log" // set on command line; correct
  }

The program used the default set in the BeforeApply method instead of the TEST_KONG2_LOG_DIR variable set in the environment (and added to the logFile field in Globals with the tag env:"TEST_KONG2_ROOT_DIR").

Scenario 3

Next, I commented out the BeforeApply method, recompiled the program and ran it the same way (with no config file being loaded), which produced this output.

  {
    "rootDir": "[EnvDir]/root", // this is correct, but only because there's no config file and BeforeApply was removed
    "logDir": "[CliDir]/log"  // set on command line; correct
  }

Because Kong couldn't find a standard config file and BeforeApply was removed, it used the environment to set rootDir. This proves that Kong can read environmental vars when the Kong env:"" tag is used, but it's not applying it at the proper precedence level.

Scenario 4

When I unset all of the env vars and run the program (which still has BeforeApply() commented out), I get this output:

  {
    "rootDir": "[TagDir]/root", // set in the Kong default:"" tag; correct
    "logDir": "[CliDir]/log" // set on command line; correct
  }

Because there is no competing env var, or anything being set in BeforeApply(), the program is now using the value of the default:"" tag.

Scenario 5

Finally, when I unset all of the env vars, restore BeforeApply(), recompile and then run the program, I get this output:

  {
    "rootDir": "[DefDir]/root", // set in BeforeApply(); correct
    "logDir": "[CliDir]/log" // set on command line; correct
  }

This shows that variables set with BeforeApply() have a higher precedence than the ones set with the Kong env:"" tag.

Observed Order

So, based on the various scenarios in the test program, here's what what I've got so far for variable precedence in Kong:

1. Command line
2. Config files
3. Variables set in BeforeApply
4. Environmental vars
5. Variables set in Kong default:"" tags

Expected Order

Based on our best practices and well as some of the "12 Factor App" literature, we expected this order:

1. Command line
2. Environmental vars
3. Config files
4. Variables set in BeforeApply
5. Variables set in Kong default:"" tags

Questions

1. Is there anyway to get Kong to check environmental variables after it checks config files?
   Right now the apps we're working on don't use env vars, but there will be ones in the future that will, and these will have to comply with our team's expected ordering.
2. Is there a better way to initialize defaults than using a BeforeApply method?
   We could use a Kong default:"" tag, but this appears to be constrained to hard-coded values that can't be calculated. Many of our defaults are calculated: their values aren't known until runtime and therefore can't be placed in field tags
3. Is there anyway to get Kong to return the path to the actual config file that got loaded? (this would be blank if no file was loaded)
   This would help avoid confusion in the event that a program got its vars from the wrong config file

[pauluzee]

Correction in Scenario 2:
The program used the default set in the BeforeApply method instead of the TEST_KONG2_ROOT_DIR variable set in the environment (and added to the RootDir field in Globals with the tag env:"TEST_KONG2_ROOT_DIR").

[alecthomas]

Is there anyway to get Kong to check environmental variables after it checks config files?

No, and this is deliberate because in the following case I expect the config file passed on the command-line to take precedence.

export VALUE=env
go run ./_examples/order/main.go --config config.json
json

OTOH when there are default configuration files, I would expect envars to take precedence, which Kong doesn't currently.

Is there a better way to initialize defaults than using a BeforeApply method?

Yes, use default:"${var}" and pass kong.Vars{"var": "value"} as an option.

Is there anyway to get Kong to return the path to the actual config file that got loaded?

No, but that seems like a reasonable thing to the Flag model struct.