Preprocess needs a settings/configuration file

[raprasad]

Preprocess needs a settings/configuration file

• Purpose: set default values for preprocess assumptions
• Allows override by environment variables or possible another config file.

Examples of config settings:

• Date utility

  • Currently years must be between 1600 and 2100, unless column explicitly called year
  • Add YEAR_DETECTION_MIN, YEAR_DETECTION_MAX, or something similar

• Date recognition

  • Currently, when checking for dates, only the 1st 10 values are checked.
  • Add DATE_DETECT_NUM_CHECK_ROWS (or something similar)