Resources Folder Structure

The model repository resources folder (i.e., <root>/src/main/resources) folder contains all non-Java artefacts, such as xml files, json files, txt files and schema files. The resources folder structure has grown organically over time, and would benefit a reorganisation to group common artefacts together for ease and efficiency.

This page looks at the ingestion and reporting resources.

Legacy

Legacy folder structure refers to existing folder structure, before the implementation of any proposals.

Ingestion

The files associated with ingestion are spread of many folders, i.e., cdm-sample-files, config, ingestions, mapping-analytics, result-json-files, schemas.

Folder structure

• resources
  • cdm-sample-files
    • fpml-5-10/products/rates (folder for each schema, sub-folders for categories)
      • ingestion input files (xml or json)
      • ingestion expectation files
  • config
    • excluded mapping path files (txt)
  • ingestions
    • ingestion configuration files
  • mapping-analytics
    • ingestion configuration files
  • result-json-files
    • fpml-5-10/products/rates (folder structure mirrors cdm-sample-files)
      • ingestion output files (json)
  • schemas

Reporting

The files associated with regulatory reporting are all inside folder /regulatory-reporting, however there are a number of issues:

• /regulatory-reporting/data contains input files, expected output files and data-set descriptor files
• Key/value output expectations for a input are stored in the same file for all reports, making expectation management inefficient
• Report type expectations (e.g. serialised json from a report type object) are not stored, so cannot be asserted
• Report type validation failure expectations are not stored, so cannot be asserted

Folder structure

• resources
  • regulatory-reporting
    • data
      • expected
        • fpml-5-10/record-keeping/products (folder structure mirrors cdm-sample-files)
          • reporting output files (json, 1 file contains results from all reports)
      • fpml-5-10/record-keeping/products (folder structure mirrors cdm-sample-files)
        • enriched reporting input files (json)
      • test-pack descriptor files
    • lookup
      • look-up descriptor files
    • refdata
      • reference data descriptor files

Process orientated folder structure

This proposal groups artefacts together by their process, i.e. translate/ingestion, enrichment, reporting. Each process has an input and output folder, plus a config and analytics folders if necessary.

Ingestion

• Expectation files move from the input folder to the output folder.
• Alternative option for top-level translate folder, could be translate/ingestion as in the future there may be translate/projection.
• Input and output sub-folders are grouped by schema (e.g. fpml-5-10) and sub-category (e.g. product), same as previous structure.

Folder structure

• resources
  • translate
    • input
      • fpml-5-10/products/rates (folder for each schema, sub-folders for categories)
        • ingestion input files (xml or json)
    • output
      • fpml-5-10/products/rates (folder structure mirrors input folder)
        • ingestion output files (json)
        • ingestion expectation files (validation failures etc)
    • config
      • exclude
        • excluded mapping path files (txt)
      • ingestion configuration files
    • analytics
      • ingestion mapping analytics files
    • schema

Reporting

• Splits up the existing data folder into separate input, output and config folders.
• Output folder contains expectations for key/value format, report type format and validation failure expectations.
• Input and output sub-folders are grouped by test-packs (e.g. rates), not by ingestion schema/category as previously because reporting is separate from ingestion.

Folder structure

• resources
  • regulatory-reporting
    • input
      • rates (folder for each test-pack)
        • enriched reporting input files (json)
    • output
      • cftcpart45 (folder for each report)
        • rates (folder for each test-pack)
          • reporting output key-value files (json)
          • reporting output report files (json)
          • reporting expectation files (validation failures)
    • config
      • test-pack descriptor files
    • lookup
      • look-up descriptor files
    • refdata
      • reference data descriptor files

Enrichment

Should there be an enrichment folder? It is a separate process from ingestion and reporting that takes the ingestion output, and enriches it via functions (and possibly other enrichment such as LEIs).