Document how to start zbMATH Open data import

[physikerwelt]

Could you document how the zbMATH Open data import can be started?

[LizzAlice]

Sure, in this issue, or somewhere else?

[physikerwelt]

It should be referenced from this issue if you can think of a better place. As long as it is documented somewhere we can move the documentation to another place later.

[LizzAlice]

I'll document it here for now, as it will change anyway as soon as we introduce Airflow or sth similar.

1. Enter mardi-importer container
2. Create data dir
3. Copy config file and data dump file from data/zbmath on mardi02 into data dir
4. add line in /usr/local/lib/python3.9/dist-packages/wikibaseintegrator/models/claims.py in from_json after if 'datatype' in claim['mainsnak']: --> if claim['mainsnak']['datatype'] == 'contentmath': continue
5. Get name of latest uploaded paper
6. In mardi/import/zbmath/ZBMathSource.py in push(): un-comment the code after and including if not found; substitute paper title with the latest uploaded paper (or a few papers before that, to be safe)
7. start import

[physikerwelt]

I suggest to add those steps to the docker file or the entry point of the container.

[physikerwelt]

I'll document it here for now, as it will change anyway as soon as we introduce Airflow or sth similar.

I think you should try the already existing job-runner first. It was good for creating DLMF pages.

1. Enter `mardi-importer` container

2. Create `data` dir

3. Copy config file and data dump file from `data/zbmath` on `mardi02` into `data` dir

This can be avoided by mounting the directory in the docker-compose file.

4. add line in `/usr/local/lib/python3.9/dist-packages/wikibaseintegrator/models/claims.py` in `from_json` after `if 'datatype' in claim['mainsnak']:`  --> `if claim['mainsnak']['datatype'] == 'contentmath': continue`

Can be changed in the container.

5. Get name of latest uploaded paper

This requires some work. The script should be changed to write the name of the latest successfully imported paper to a directory mounted to the host machine.

6. In `mardi/import/zbmath/ZBMathSource.py` in `push()`: un-comment the code after and including `if not found`; substitute paper title with the latest uploaded paper (or a few papers before that, to be safe)

It can also be included in the container, but read the file written in step 5 instead.

7. start import

How exactly?