Skip to content

Commit 2c21edc

Browse files
TobiasNxTobiasNx
committed
Uodate session
1 parent d6ad7cb commit 2c21edc

File tree

2 files changed

+188
-13
lines changed

2 files changed

+188
-13
lines changed

‎06_MetafactureCLI.md‎

Lines changed: 187 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,8 @@ To use Metafacture on the commandline we can download the latest distribution e.
2020

2121
Hint: If 7.0.0 is not published yet use the runner version of the [prerelease 7.0.0-rc1](https://github.com/metafacture/metafacture-core/releases/tag/metafacture-core-7.0.0-rc1).
2222

23-
Download `metafacture-core-$VERSION-dist.tar.gz` or the zip version and extract the archive.
23+
Download `metafacture-core-$VERSION-dist.tar.gz` or the zip version and extract the archive to your choosen folder.
24+
In the folder you find the `flux.bat` and `flux.sh`
2425

2526
The code below assumes you moved the resulting folder to your home directory and renamed it to `'metafacture'`
2627

@@ -49,17 +50,19 @@ The playground has a nice feature to export and import Metafacture Workflows.
4950

5051
[So lets go to the Playground.](https://metafacture.org/playground/?flux=%22https%3A//openlibrary.org/books/OL2838758M.json%22%0A%7C+open-http%0A%7C+as-lines%0A%7C+decode-json%0A%7C+encode-yaml%0A%7C+print%0A%3B)
5152

52-
Export the workflow with the Export Button and lets run the flux.
53+
Export the workflow with the Export Button:
5354

5455
![Shows Export Button in Playground.](images/Export.png)
5556

57+
Open your terminal and and lets run the flux with the following command:
58+
5659
Linux:
5760

5861
```bash
5962
~/metafacture/flux.sh downloads/playground.flux
6063
```
6164

62-
or Windows:
65+
Windows:
6366

6467
```bash
6568
~/metafacture/flux.bat downloads/playground.flux
@@ -68,30 +71,32 @@ or Windows:
6871
The result of running the Flux-Script via CLI should be the same as with the Playground.
6972

7073
The Metafacture CLI Tool expects a flux file for every workflow.
71-
Our runned workflow only has the following flux and no additional files since it i querring data from the web and it has no fix transformations.
74+
Our runned workflow only has the following flux and no additional files since it is fetching data from the web and it has no fix transformations.
75+
76+
The downloaded file should have the following content, defining the playground specific variables and the flux workflow that you also saw in the playground. You can delete the playground specific variables since they are not needed here.
7277

7378
```default
7479
"https://openlibrary.org/books/OL2838758M.json"
7580
| open-http
7681
| as-lines
7782
| decode-json
78-
| encode-yaml
83+
| encode-json(prettyPrinting="true")
7984
| print
8085
;
8186
```
8287

8388
## Use local files for transformation
8489

8590
If you want to load a local file instead of fetching data from the web we need to change the flux a little bit with an texteditor.
86-
Download the following file [11942150X.json](/home/tobias/git/metafacture-tutorial/sample-scripts/lesson_06/11942150X.json)
91+
Download the following file [11942150X.json](./sample-scripts/lesson_06/11942150X.json)
8792
and adjust the path to your file.
8893

89-
Adjust your `downloads/playground.flux` script:
94+
Adjust your `downloads/playground.flux` script, so that it does not load data from the web, but opens a local file with `open-file` and read it `as-recrods` since the json file is pretty printed:
9095

9196
```default
9297
"path/to/your/file/11942150X.json" // Adjust your path!
9398
| open-file
94-
| as-lines
99+
| as-records
95100
| decode-json
96101
| encode-yaml
97102
| print
@@ -100,7 +105,177 @@ Adjust your `downloads/playground.flux` script:
100105

101106
Run it again as shown above.
102107

103-
If we want to use fix we need to refrence the fix file that in the playground we only refrenced via `|fix`
108+
It should output:
109+
110+
```JSON
111+
{
112+
"professionOrOccupation" : [ {
113+
"id" : "https://d-nb.info/gnd/4629643-8",
114+
"label" : "Politologin"
115+
}, {
116+
"id" : "https://d-nb.info/gnd/4025243-7",
117+
"label" : "Hochschullehrer"
118+
} ],
119+
"placeOfBirth" : [ {
120+
"id" : "https://d-nb.info/gnd/4050042-1",
121+
"label" : "Riga"
122+
} ],
123+
"gender" : [ {
124+
"id" : "https://d-nb.info/standards/vocab/gnd/gender#female",
125+
"label" : "Weiblich"
126+
} ],
127+
"dateOfDeath" : [ "1992-09-17" ],
128+
"dateOfBirth" : [ "1928-09-24" ],
129+
"placeOfDeath" : [ {
130+
"id" : "https://d-nb.info/gnd/4009352-9",
131+
"label" : "Cambridge, Mass."
132+
} ],
133+
"variantNameEntityForThePerson" : [ {
134+
"forename" : [ "Judith Nisse" ],
135+
"surname" : [ "Shklar" ]
136+
}, {
137+
"forename" : [ "Judita" ],
138+
"surname" : [ "Nisse" ]
139+
}, {
140+
"forename" : [ "Judith" ],
141+
"surname" : [ "Shklar" ]
142+
} ],
143+
"type" : [ "Person", "AuthorityResource", "DifferentiatedPerson" ],
144+
"@context" : "https://lobid.org/gnd/context.jsonld",
145+
"gndSubjectCategory" : [ {
146+
"id" : "https://d-nb.info/standards/vocab/gnd/gnd-sc#8.1p",
147+
"label" : "Personen (Politologen, Staatstheoretiker)"
148+
} ],
149+
"oldAuthorityNumber" : [ "(DE-588a)11942150X", "(DE-588c)4439975-3", "(DE-588)158461525", "(DE-588a)158461525" ],
150+
"geographicAreaCode" : [ {
151+
"id" : "https://d-nb.info/standards/vocab/gnd/geographic-area-code#XD-US",
152+
"label" : "USA"
153+
} ],
154+
"deprecatedUri" : [ "https://d-nb.info/gnd/158461525" ],
155+
"affiliation" : [ {
156+
"id" : "https://d-nb.info/gnd/2012974-9",
157+
"label" : "Harvard University"
158+
} ],
159+
"describedBy" : {
160+
"id" : "https://d-nb.info/gnd/11942150X/about",
161+
"license" : {
162+
"id" : "http://creativecommons.org/publicdomain/zero/1.0/",
163+
"label" : "http://creativecommons.org/publicdomain/zero/1.0/"
164+
},
165+
"dateModified" : "2019-07-11T15:58:51.000",
166+
"descriptionLevel" : {
167+
"id" : "https://d-nb.info/standards/vocab/gnd/description-level#1",
168+
"label" : "Allgemeines, Interdisziplinäre Allgemeinwörter"
169+
}
170+
},
171+
"gndIdentifier" : "11942150X",
172+
"id" : "https://d-nb.info/gnd/11942150X",
173+
"preferredName" : "Shklar, Judith N.",
174+
"wikipedia" : [ {
175+
"id" : "https://de.wikipedia.org/wiki/Judith_N._Shklar",
176+
"label" : "https://de.wikipedia.org/wiki/Judith_N._Shklar"
177+
} ],
178+
"variantName" : [ "Shklar, Judith", "Nisse, Judita", "Shklar, Judith Nisse" ],
179+
"preferredNameEntityForThePerson" : {
180+
"forename" : [ "Judith N." ],
181+
"surname" : [ "Shklar" ]
182+
},
183+
"sameAs" : [ {
184+
"id" : "http://id.loc.gov/rwo/agents/n82231811",
185+
"collection" : {
186+
"id" : "http://www.wikidata.org/entity/Q13219454",
187+
"abbr" : "LC",
188+
"publisher" : "Library of Congress",
189+
"icon" : "http://www.loc.gov/favicon.ico",
190+
"name" : "NACO Authority File"
191+
}
192+
}, {
193+
"id" : "http://viaf.org/viaf/7412367",
194+
"collection" : {
195+
"id" : "http://www.wikidata.org/entity/Q54919",
196+
"abbr" : "VIAF",
197+
"publisher" : "OCLC",
198+
"icon" : "http://viaf.org/viaf/images/viaf.ico",
199+
"name" : "Virtual International Authority File (VIAF)"
200+
}
201+
}, {
202+
"id" : "http://www.wikidata.org/entity/Q455736",
203+
"collection" : {
204+
"id" : "http://www.wikidata.org/entity/Q2013",
205+
"abbr" : "WIKIDATA",
206+
"publisher" : "Wikimedia Foundation Inc.",
207+
"icon" : "https://www.wikidata.org/static/favicon/wikidata.ico",
208+
"name" : "Wikidata"
209+
}
210+
}, {
211+
"collection" : {
212+
"abbr" : "DNB",
213+
"name" : "Gemeinsame Normdatei (GND) im Katalog der Deutschen Nationalbibliothek",
214+
"publisher" : "Deutsche Nationalbibliothek",
215+
"icon" : "https://www.dnb.de/SiteGlobals/Frontend/DNBWeb/Images/favicon.png?__blob=normal&v=4",
216+
"id" : "http://www.wikidata.org/entity/Q36578"
217+
},
218+
"id" : "https://d-nb.info/gnd/11942150X/about"
219+
}, {
220+
"id" : "https://d-nb.info/gnd/158461525",
221+
"collection" : {
222+
"id" : "http://www.wikidata.org/entity/Q36578",
223+
"abbr" : "DNB",
224+
"publisher" : "Deutsche Nationalbibliothek",
225+
"icon" : "http://www.dnb.de/SiteGlobals/StyleBundles/Bilder/favicon.png?__blob=normal&v=1",
226+
"name" : "Gemeinsame Normdatei (GND) im Katalog der Deutschen Nationalbibliothek"
227+
}
228+
}, {
229+
"id" : "https://dbpedia.org/resource/Judith_N._Shklar",
230+
"collection" : {
231+
"id" : "https://dbpedia.org"
232+
}
233+
}, {
234+
"collection" : {
235+
"abbr" : "dewiki",
236+
"name" : "Wikipedia (Deutsch)",
237+
"publisher" : "Wikimedia Foundation Inc.",
238+
"icon" : "https://de.wikipedia.org/static/favicon/wikipedia.ico",
239+
"id" : "http://www.wikidata.org/entity/Q48183"
240+
},
241+
"id" : "https://de.wikipedia.org/wiki/Judith_N._Shklar"
242+
}, {
243+
"collection" : {
244+
"abbr" : "enwiki",
245+
"name" : "Wikipedia (English)",
246+
"publisher" : "Wikimedia Foundation Inc.",
247+
"icon" : "https://en.wikipedia.org/static/favicon/wikipedia.ico",
248+
"id" : "http://www.wikidata.org/entity/Q328"
249+
},
250+
"id" : "https://en.wikipedia.org/wiki/Judith_N._Shklar"
251+
}, {
252+
"id" : "https://isni.org/isni/0000000121193284",
253+
"collection" : {
254+
"id" : "https://isni.org"
255+
}
256+
}, {
257+
"collection" : {
258+
"abbr" : "DE-611",
259+
"name" : "Kalliope Verbundkatalog",
260+
"publisher" : "Staatsbibliothek zu Berlin - Preußischer Kulturbesitz",
261+
"icon" : "https://kalliope-verbund.info/img/favicon.ico",
262+
"id" : "https://kalliope-verbund.info"
263+
},
264+
"id" : "https://kalliope-verbund.info/gnd/11942150X"
265+
}, {
266+
"collection" : {
267+
"abbr" : "DDB",
268+
"name" : "Deutsche Digitale Bibliothek",
269+
"publisher" : "Deutsche Digitale Bibliothek",
270+
"icon" : "https://www.deutsche-digitale-bibliothek.de/favicon.ico",
271+
"id" : "http://www.wikidata.org/entity/Q621630"
272+
},
273+
"id" : "https://www.deutsche-digitale-bibliothek.de/person/gnd/11942150X"
274+
} ]
275+
}
276+
```
277+
278+
If we want to use fix we need to refrence the fix file that in the playground we only refrenced via `| fix`
104279

105280
```default
106281
"path/to/your/file/11942150X.json"
@@ -113,11 +288,11 @@ If we want to use fix we need to refrence the fix file that in the playground we
113288
;
114289
```
115290

116-
Create a new file with a `fixFile.fix`, files with fix scripts should have a `.fix` file suffix.
291+
Create a new file with the name `fixFile.fix`, files with fix scripts should have a `.fix` file suffix.
117292

118-
Add the follwoing line as content:
293+
Add the follwoing line as content to this file:
119294

120-
```PEARL
295+
```perl
121296
retain("preferredName","id","type[]")
122297

123298
```

‎08_Harvest_data_with_OAI-PMH.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ You can also transform incoming data and immediately store/index it with MongoDB
4747

4848
Add the following fixes to the file:
4949

50-
```PEARL
50+
```perl
5151
copy_field("001","_id")
5252
copy_field("245??.a","title")
5353
copy_field("100??.a","creator[].$append")

0 commit comments

Comments
 (0)