LinkedMusic Project Meeting - Data Processing Note Dump

[malajvan]

Technical Meeting

Conversation with Alaistair on Friday

• Explore two options for obtaining data from MusicBrainz:

  • JSON dump:
    • Manual parsing of links through JSON
    • Potential for increased error rates due to ignoring schema.org context
  • CSV dump:
    • Install MusicBrainz server components, such as musicbrainz-docker and db-only mirror docker-compose
    • Two ways to obtain data: Full database or JSON dump
  • JSON dump is a more straightforward option and doesn't require the MusicBrainz server
    • Use JSON data dump and reconcile with the reconciliation service API

• Discuss the flattening process:

  • Understand the role of OpenRefine and its specific functionalities
  • Consider bypassing OpenRefine for problem-solving
  • Evaluate the use of the W3C reconciliation service API
  • Downloading Wikidata and create a custom service for Wikidata? (faster traverse with graphdb)

Presentation Meeting

• Wikidata: can we apply canonical choosing from MusicBrainz (string text matching) (for VIM)

• Consider the need for data flattening

  • Evaluate the use of the Reconciliation Service API and its impact on data structure
  • Assess the necessity of maintaining nested JSON format, especially for databases without record-based data

• Do we need JSON-LD?

  • Explore the use of JSON-LD, and potentially set up RDF/SPARQL searching (consult Andrew Hankinson)

• Data Reconciliation:

  • If using Wikidata Reconciliation Service API: can the API utilize multiple columns for reconciliation, and maybe even rank them by priority (for example: if reconciling column "composer", prioritize viaf id, then rism id, then the text string name etc)

Conclusion

Plan for the near future:

1. Using the JSON dump of MusicBrainz, test out Reconciliation Service API, focus on reusability, ease of use, and scalability (can non-technical people use it? Can we replace OpenRefine with it? Can it handle changing of database models? ) handle cases where reconciliation already exists
2. Discuss with @homework36 and @dchiller about nested structure format - do they want nested json for searching? can we just keep original database relational model for search? How flat do we flatten/ do we flatten the data at all?
3. With a structure of the data for searching/indexing, test with Sparql

[maregold]

Some extra notes from both Dylan and Van's presentations:

Data Lake Demonstration (Dylan Hillerbrand)

• No Mode 1, need a way to add that (Anna did this at the meeting, and now shows up on Wikidata)
  o What do you show the user: if they searched for Dorian, do we show them Dorian or do we show them Mode 1 (which is from the original database)
• Tooltip energy: we’re showing you this because it is the same thing
• Google shows the original term and the substituted term
  o Did you mean this
• Has to be clear to the user is that what they are seeing is not everything in every database, just what databases had this specific tag
• Where is user testing planned into the project
• Is it possible to search melodic materials across different formats

Data export for the Data Lake (Hong Van Pham / Jacob deGroot-Maggetti)

• Once you use the API you won’t have to flatten the database
• Joining the communities to have more control on Wikidata
• What do you do when things change in the source database
  o Formats hopefully won’t change that often
• Different definitions for anonymous
  o Unspecified
  o Wikidata might combine everything together
• Sending results of the reconciliation back to the original database if possible
• Make our own Wikibase / static ontology file website for things of local interest
  o When we won't be able to get properties for really niche things, such as jSymbolic properties

[dchiller]

Thanks, Van + Marissa...I'll add mine here too.

• Can we do something similar to the MusicBrainz "canonical" item?
  TROMPA project lessons
  — Why would we populate something that already exists — the relationships might already be there…
  — What is the original content that we will create?
  — Why collect a bunch of publicly available information? Will you update this, and how will you update?
  - What does everyone in the project need?
  - ALSO: Let’s feed data back in for sustainability
• Can we provide the originating database with the reconciled data?
• Share the "why" something was returned
• Allow user to interact with the possible relationships between fields ("did you mean?")
• Don't worry about dead databases...we can just download them and reconcile once
• What will we do if a Wikidata item we rely on is deleted?
• AH: Store documents separately than index
• Look at projectblacklight.org (recommended by Jon Dunn) for possible Solr-enabled backend

[ahankinson]

I would avoid Blacklight... I used it extensively at the Bodleian, and we use it in Muscat. But we're in the process of removing it in Muscat, and if I had stayed longer at the Bod I would have phased it out.

• Hundreds of dependencies
• Adds unnecessary layers of abstraction on top of Solr
• requires a Ruby on Rails server
• template customization is EXTREMELY complicated if you're not a Rails dev....

[dchiller]

Ah...very good reasons, thanks!

[dchiller]

From your experience, is there anything valuable about the way it does things? In other words, not as a reason to use it, but any approach that we could learn from for what we are doing, if that makes sense?

[ahankinson]

It has a good facet interface for range searches… that’s really the only good thing I can think of.

[dchiller]

Great...thanks!