Skip to content

Parse CSV schema in meta configuration #5

@gfissore-rai

Description

@gfissore-rai

In the current implementation, the schema of the input CSV data is specified implicitly by using the int_attr, float_attr, etc... relations inside the meta configuration module. However, Rel supports a native way to specify a schema through the config:schema relation used in load_csv[config].

It should be possible to get rid of the int_attr, float_attr, etc... relations by specifying the schema relation in the meta configuration, in the exact same way as it's done for load_csv. This comes with some advantages:

  1. In many cases, when you load CSV files, you might already specify the schema with config:schema, so it makes a lot of sense to just reuse this (I would also suppose that this is the right way to go from a performance standpoint);
  2. moving the type information from the hardcoded relation names (as in int_attr) to the input (as in config:schema = {(:quantity, "int");}) is more general and probably more future-proof; indeed, I think it would be easy to implement this in such a way that if new data types are supported in config:schema in the future, they will work out of the box with the CSV library (in the current implementation you would need to update the library by adding a new_type_attr relation).

@grigory93 would love to hear your thoughts on this!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions