Description
In the current implementation of the parsing of the context, the function _convert_paths_to_absolute_posix — which converts relative to absolute paths — restrict the conversion to three types of keys, namely
# only check a few conf keys that are known to specify a path string as value
conf_keys_with_filepath = ("filename", "filepath", "path")This is very restrictive and can lead to unexpected issues while implementing custom data set as explained in the example below.
Context
Let's imagine that I want to extend the SQLQueryDataSet by using a Jinja template and feed it with parameters. I would have done something like the following
class TemplatedSQLQueryDataSet(SQLQueryDataSet):
def __init__(
self,
templated_query_filepath: str,
parameters_path: str,
credentials: Optional[Dict[str, Any]] = None,
load_args: Optional[Dict[str, Any]] = None,
fs_args: Optional[Dict[str, Any]] = None,
) -> None:
if parameters_path is not None:
self._query_parameters = JSONDataSet(filepath=parameters_path).load()
super().__init__(
"",
credentials or {},
load_args or {},
fs_args or {},
templated_query_filepath,
)
def _load(self) -> pd.DataFrame:
load_args = copy.deepcopy(self._load_args)
engine = self.engines[self._connection_str] # type: ignore
if self._filepath:
load_path = get_filepath_str(PurePosixPath(self._filepath), self._protocol)
with self._fs.open(load_path, mode="r") as fs_file:
template = fs_file.read()
load_args["sql"] = Template(source=template).render(**self._query_parameters)
return pd.read_sql_query(con=engine, **load_args)While this works when running a pipeline or using kedro ipython, it potentially breaks when used in a notebook under notebooks with a catalog entry like
whatever:
type: my_project.extras.datasets.templated_sql_dataset.TemplatedSQLQueryDataSet
templated_query_filepath: data/01_raw/template.sql
parameters_path: data/03_primary/parameters.json
because the relative paths are not properly parsed when loading the context.
One can solve this particular example by replacing the two paths by filepath and path for example which belong to the list of parsed keys mentioned at the top of this issue. In terms of user experience, it is nevertheless pretty unintuitive. Another possibility is to have nested elements like "template": {"filepath": ...}.
Possible Implementation
Could we use something like a regex containing path instead of a rigid list?
Thanks in advance!
Description
In the current implementation of the parsing of the context, the function
_convert_paths_to_absolute_posix— which converts relative to absolute paths — restrict the conversion to three types of keys, namelyThis is very restrictive and can lead to unexpected issues while implementing custom data set as explained in the example below.
Context
Let's imagine that I want to extend the
SQLQueryDataSetby using a Jinja template and feed it with parameters. I would have done something like the followingWhile this works when running a pipeline or using
kedro ipython, it potentially breaks when used in a notebook undernotebookswith a catalog entry likebecause the relative paths are not properly parsed when loading the context.
One can solve this particular example by replacing the two paths by
filepathandpathfor example which belong to the list of parsed keys mentioned at the top of this issue. In terms of user experience, it is nevertheless pretty unintuitive. Another possibility is to have nested elements like"template": {"filepath": ...}.Possible Implementation
Could we use something like a regex containing
pathinstead of a rigid list?Thanks in advance!