Merge pull request #209 from dssg/add_user_metadata_example

Describe an example of user_metadata use

Author: not-the-fish

example_experiment_config.yaml

@@ -128,21 +128,39 @@ state_config:
 
 # USER METADATA
 # These are arbitrary keys/values that you can have Triage apply to the
-# metadata for each matrix in the experiment. Any keys you include here can
-# be used in the 'model_group_keys' below.
+# metadata for every matrix in the experiment. Any keys you include here can
+# be used in the 'model_group_keys' below. For example, if you run experiments
+# that share a temporal configuration but that use different label definitions
+# (say, labeling building inspections with *any* violation as positive or
+# labeling only building inspections with severe health and safety violations
+# as positive), you can use the user metadata keys to indicate that the matrices
+# from these experiments have different labeling criteria. The matrices from the
+# two experiments will have different filenames (and not be overwritten or
+# inappropriately reused), and if you add the label_definition key to the model
+# group keys, models made on different label definition will have different
+# groups. In this way, user metadata can be used to expand Triage beyond its
+# explicitly supported functionality.
 user_metadata:
     label_definition: 'severe_violations'
 
 # MODEL GROUPING
 # Model groups are aimed at defining models which are equivalent across time splits.
+# In other words, you will probably want to define model groups by any  parameters
+# that distinguish models *other than the beginning end dates of their data.*
 # By default, the classifier module name, hyperparameters, and feature names are used.
+# 
 #
 # model_group_keys defines a list of *additional* matrix metadata keys that
-# should be considered when creating a model group
+# should be considered when creating a model group. For example, if the models are
+# built on matrices with different history lengths (train durations), different
+# in the next month, next year, or next two years), the frequency of rows for each
+# labeling windows (e.g., inspection violations entity (train example frequency), or
+# the definition of a positive label.
 model_group_keys:
     - 'train_duration'
     - 'train_label_window'
     - 'train_example_frequency'
+    - 'label_definition'
 
 # GRID CONFIGURATION
 # The classifier/hyperparameter combinations that should be trained