Skip to content

VDW v.1.6.1
Compare
Choose a tag to compare
@RoelantVos RoelantVos released this 28 Apr 02:07
v.1.6.1
1177afa
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

The most recent release for Virtual Data Warehouse this is 1.6.1.

These updates have fully decoupled the management of the (source-to-target) mapping metadata with the code generation, making it easier to use other tools for some of the functions if desired and by virtue of this support a bigger ecosystem for open source Data Warehouse Automation.

TEAM now saves the design metadata as Json files that conform to the generic schema for Data Warehouse automation. The Virtual Data Warehouse tool can now be configured to read all (Json) files from a designated directory and apply the templates (patterns) to these files using the provided templating engine.

This means it is possible to use TEAM without VDW and vice-versa. It is also possible to incorporate the schema validation functionality available in the Data Warehouse Automation class library to make sure all files are conform the standard. Last, but not least it makes it easier to create your own patterns using the available metadata without software constraints.

The TEAM software is still geared towards Data Vault use-cases, but due to the ability to tweak the patterns without code changes it can now be better adapted for other applications also. For example, this release was used to generate an Persistent Staging Area for Azure Data Factory!

Because of the pattern engine, VDW is completely agnostic of design approaches.

This approach also makes it easier to integrate the metadata into CI/CD pipelines and version control, because the key design artefacts are now all text based – making it easier to commit changes (differentials).

The TEAM metadata remain the key artefacts to version control because these contain the true design Intellectual Property explaining which data elements go where – mapping and lineage. However, the Json files that are created as part of the ‘activation’ routine can be consumed in a DevOps pipeline for code generation.

Alternatively, complex output from the pattern code generation can be written back into the Json structure as (source) transformations.

Since the Json files conform to the generic schema for Data Warehouse Automation they can also be manually tweaked or even created. This may be useful in case there is no interest in using TEAM, but still intent to use the VDW code generation and patterns, or if custom complex transformation need to be added.

In addition to the above there is a long list of minor usability improvements including, but not limited to:

  • Easier interaction with the data grids for data entry. Various context menus have been created to simplify adding, exporting and removing rows.
  • Improved validation mechanisms that perform pre-checks before any generation work is started.
  • Masking of passwords in the software (note that passwords are still visible in the text files).
  • Better exception handling, logging and reporting.
  • Usage of external files for many operations, making the tools easier to configure and upgrade. For example – many SQL statements, patterns and lists are now stored in script and Json files and are installed as part of the software.
  • Better handling of schemas and tables, for example tables with the same name but different schemas do not cause errors anymore.
  • Usability improvements in the GUI, for example updating values when paths change etc.
Assets 3
Loading