Update ETL break due to missing lookup tables

[TobiasNx]

org.metafacture.metamorph.api.MorphExecutionException: File not found: /home/sol/git/lobid-resources/lookup-tables/data/opacLinks/isil2opac_almaMmsId.tsv
org.metafacture.metamorph.api.MorphExecutionException: File not found: /home/sol/git/lobid-resources/lookup-tables/data/opacLinks/isil2opac_zdbId.tsv

These are included via git submodule, not sure why this problem occurs. The problem also appears to affect other lookup maps that are included via git submodule e.g. lbz-vocabs: hbz/lbz-vocabs#4 (comment)

Current consequences:

• erros in ETL overcrowd logs
• broken ETL process due to overcroded logs that block storage
• missing labels which should which should have been mapped via the lookuptables that are provided via git submodule
• no updates due to broken transfromation

---

Context:
The documentation of our use of gitsubmodule: https://github.com/hbz/lobid-resources?tab=readme-ov-file#tables-as-gitsubmodules

PR that introduced the submodules: #1902 (comment)

[TobiasNx]

It seems that all git submodules are not loaded. Updates of lobid resources are missing labels: hbz/lbz-vocabs#4

[TobiasNx]

@Phu2 hinted:

That the prod system has the following:

~/git/lobid-resources/lookup-tables$ git status
HEAD detached from e6a273f
Untracked files:
  (use "git add <file>..." to include in what will be committed)
	data/rvk.tsv

The git submodules should be regularly updated in order to keep the transformation updated. I am not sure if th eHEAD detached is a result of this or if it is the result of a missing pull of the submodule?

[TobiasNx]

Due to the missing update transformation also nwbib is not updated.

[dr0i]

broken ETL process due to overcroded logs

It's the other way around: overcroded logs due to broken ETL.

[TobiasNx]

We also should improve the documentation how the git submodules should be pull in context of pulling the current master to the server, that is hidden in the README.

[dr0i]

e38b880 forces a clean installation through the monit restart script (this forcing was part of the cronjob but not the monit restart script).

• log lesser (not every document) if e.g. a map is missing to prevent HD to be full

[TobiasNx]

broken ETL process due to overcroded logs

It's the other way around: overcroded logs due to broken ETL.

I think that is not completly true but maybe this is semantics.

The scrictness level is set so that the ETL continues and even the resources is transfomed skipping the broken part but the error is logged. The ETL does not break, even when the transformation does not work correctly. Only the mapper parser exception breaks the transformation fully. This would also explain why there are missing lbz labels in lobid-resources.

Since there are multiple errors for each transformations due to the missing github submodules lookup files the logs seem to overcrowd because they are to big. @Phu2 reported that he stopped the updates because there was not enough room on the server because of the big logs.