diff --git a/docs/e2e-examples/movies.md b/docs/e2e-examples/movies.md index 394bb08..553bedf 100644 --- a/docs/e2e-examples/movies.md +++ b/docs/e2e-examples/movies.md @@ -88,10 +88,12 @@ Open the `pipelines/ratings.yaml` file and delete the default content. Then, add the following content: ```yaml -- implementation: nodestream.pipeline.extractors:FileExtractor +- implementation: nodestream.pipeline.extractors.files:FileExtractor arguments: - globs: - - data/ratings.csv + sources: + - type: local + globs: + - data/ratings.csv ``` This will tell the pipeline to extract the data from the `ratings.csv` file. @@ -137,10 +139,12 @@ Open the `pipelines/movies.yaml` file and delete the default content. Then, add the following content: ```yaml -- implementation: nodestream.pipeline.extractors:FileExtractor +- implementation: nodestream.pipeline.extractors.files:FileExtractor arguments: - globs: - - data/movies.csv + sources: + - type: local + globs: + - data/movies.csv ``` This will tell the pipeline to extract the data from the `movies.csv` file. @@ -185,10 +189,12 @@ Open the `pipelines/tags.yaml` file and delete the default content. Then, add the following content: ```yaml -- implementation: nodestream.pipeline.extractors:FileExtractor +- implementation: nodestream.pipeline.extractors.local:FileExtractor argumenets: - globs: - - data/tags.csv + sources: + - type: local + globs: + - data/tags.csv ``` This will tell the pipeline to extract the data from the `tags.csv` file. diff --git a/docs/official-plugins/shell.md b/docs/official-plugins/shell.md index d3e0405..47ddf62 100644 --- a/docs/official-plugins/shell.md +++ b/docs/official-plugins/shell.md @@ -74,10 +74,12 @@ curl https://example.com/api/data.json -X GET --header "Accept: application/json - O ignore_stdout: true #Do not provide output from command to next step in pipeline. -- implementation: nodestream.pipeline.extractors:FileExtractor +- implementation: nodestream.pipeline.extractors.files:FileExtractor arguments: - globs: - - data.json + sources: + - type: local + globs: + - data.json ``` This is roughly equivalent to the following shell command: diff --git a/docs/tutorial-basics/create-a-new-pipeline.md b/docs/tutorial-basics/create-a-new-pipeline.md index 8d79287..6a216b7 100644 --- a/docs/tutorial-basics/create-a-new-pipeline.md +++ b/docs/tutorial-basics/create-a-new-pipeline.md @@ -86,10 +86,12 @@ The second section is to interpret the data into nodes and relationships. For now, leave the second section as it is and replace the first section with the following: ```yaml -- implementation: nodestream.pipeline.extractors:FileExtractor +- implementation: nodestream.pipeline.extractors.files:FileExtractor arguments: - globs: - - data/*.csv + sources: + - type: local + globs: + - data/*.csv ``` This step uses the `FileExtractor` to load the data from the `data` directory in your project. diff --git a/docs/tutorials-advanced/extending-the-dsl.mdx b/docs/tutorials-advanced/extending-the-dsl.mdx index d5e7e03..0dd9d85 100644 --- a/docs/tutorials-advanced/extending-the-dsl.mdx +++ b/docs/tutorials-advanced/extending-the-dsl.mdx @@ -129,10 +129,12 @@ Once the new file type is registered, it can be used in the DSL as a file extrac The file extractors will automatically be aware of and use the new file type. ```yaml -- implementation: nodestream.pipeline.extractors:FileExtractor +- implementation: nodestream.pipeline.extractors.files:FileExtractor arguments: - globs: - - data/*.tsv + sources: + - type: local + globs: + - data/*.tsv ``` ## Creating an Interpretation diff --git a/docs/tutorials-intermediate/configuring-projects.md b/docs/tutorials-intermediate/configuring-projects.md index 5fc7b90..72ae416 100644 --- a/docs/tutorials-intermediate/configuring-projects.md +++ b/docs/tutorials-intermediate/configuring-projects.md @@ -102,8 +102,10 @@ For example, lets say in a live environment we want to pull data from an API, bu annotations: - live -- implementation: nodestream.pipeline.extractors:FileExtractor +- implementation: nodestream.pipeline.extractors.files:FileExtractor arguments: + sources: + - type: local globs: - !env FILE_PATH annotations: