What exactly does the full pipeline look like?

[linneakirby]

I am still unclear on what exactly the full pipeline for this project looks like. I have been doing my best to reconcile databases, but it seems like there's a significant amount of entries not found in WikiData, and we are all making slightly different choices when reconciling. I think it would be useful to have an example of what the final output will be so we can all more easily standardize decision-making when reconciling.

[linneakirby]

@TTG3333 @Sbebastien @SCN-MNG do you have any thoughts/opinions?

[Sbebastien]

I think it would be useful to see the final output, maybe with one of the smaller databases reconciled and converted to rdf ?

[fujinaga]

Yes, I agree. Your should all try to convert smaller databases (or a subset) to RDF, load them to Virtuoso, and try some SPARQL queries.

[SCN-MNG]

I am still unclear on what exactly the full pipeline for this project looks like. I have been doing my best to reconcile databases, but it seems like there's a significant amount of entries not found in WikiData, and we are all making slightly different choices when reconciling. I think it would be useful to have an example of what the final output will be so we can all more easily standardize decision-making when reconciling.

My understanding is that our current pipeline can be summarized to:

1. Obtaining Raw Data
2. Reconcile Raw Data against Wikidata Entities (QID)
3. Find Wikidata Properties (PID) to Describe Relationship Between The Entities
4. Convert to RDF (With Turtle Serialization)
5. Import to Virtuoso

I do not know exactly what happens after we import to Virtuoso since our server isn't fully set up yet. But I expect it results in a SPARQL endpoint for all the data we have reconciled, similar to Wikidata Query Service. At least that's how I expect it to be before we implement Natural Language Query.

[linneakirby]

That's also my understanding. My problem is not knowing what comes after importing to Virtuoso. If we can make at least a small working example, I think it would make the decision-making during reconciliation more straightforward, as we would have a sense of how users may interact with the final product.

[fujinaga]

In Virtuoso, you will get a SPARQL query interface much like the Wikidata Query Service.