WCL Transform #4
Description
I want to create a cli option that takes a transform.wcl file that has details for transforming data files from one layout to another.
These files don't have to be just wcl files either, they can be xml, csv/tsv, json, toml, yaml, hcl, binary or text.
The idea is you can define in the transfrom.wcl file how to parse the data from the source format into the destination format. It will then allow you to run the transformation over target files.
This will allow you to validate target files with the source schema too which will inform you if it will validate or not.
When doing the transform its important to not load the whole thing into memory in one go, we will stream it so we can support massive datasets.
For text and binary support we should have tokeniser support built into the language so we can tokenise and then build the data structure out of the tokens we collect and parse (do it off pattern matching).