Prod nextflow issues

[jamesamcl]

FTP paths

• All of the files published from the nextflow pipeline should be copied to FTP.
• We shouldn't use paths like "nextflow_latest" or "nextflow_pipeline" on the FTP, the fact that we use nextflow is an implementation detail that doesn't need to be reflected in our published data paths

Might be easier to do the copy to FTP as a separate Jenkins job and remove from the nextflow pipeline. We can copy the entire out folder.

Embeddings

Seems to be re-embedding a lot every time. I think the current embeddings aren't being swapped in as the prev embeddings for successive runs.

[haideriqbal]

This was done for testing to keep nextflow pipeline files separate while we are testing it. It will be reverted to the normal prod structure.

[jamesamcl]

Good, but there are still lots of files missing like the embeddings, separate ontology jsons for each ontology, etc. we should just copy the whole lot

[haideriqbal]

Embeddings one need to be copied... the separate ontology files are there... they are gziped together so when someone extracts it they will get separate ontology files.

[jamesamcl]

Would like to be able to download a single ontology after the refactor. We are also still missing other files published in Nextflow like the text tagger db, umaps, pca jsons, semsim tables. Hence why I would like us to copy the entire output folder.

[jamesamcl]

Here is an example of all of the files

https://ftp.ebi.ac.uk/pub/databases/spot/ols/dev2/

[haideriqbal]

closing this as now in production.