Deploy live SPARQL playground with datasets for projects & credit classes

[clevinson]

MVP: Initial hosted RDF Triple Store / Knowledge Graph Integration

This issue tracks initial work for getting our graph infrastructure up and running. The goal is to automate schema publishing to a queryable RDF graph (hosted in Neptune) and start loading real project data. Mostly backend and infra-focused, but with an eye toward future UX improvements.

✅ MVP Tasks

• Get Apache Neptune instance running, and explore how to integrate it @clevinson
  • Spin up the graph database and start poking around—make sure it’ll work for loading and querying RDF generated from our LinkML YAML schemas.
• Set up GitHub Action to convert YAML in schema/data/playground/* into RDF and post to Neptune @paul121
  • Automate the pipeline: convert LinkML → RDF → load to Neptune on push to main.
• Refactor schema/data folder to follow new pattern (schema/data/playground) and convert existing credit class JSON-LD to YAML @clevinson
  • Move everything into a consistent structure and rewrite the JSON-LD examples as YAML so they’re easier to maintain.
• Generate project datasets for C02, C06, BT01 @S4mmyb
  • Draft project instances for a few real projects to test loading and queries.

🌀 Possible Additional MVP Tasks

• Add root identifiers to credit class & project schemas (@id in JSON-LD / identifier: true in LinkML)
• Update slot_uri for projectStartDate and projectEndDate
• Add hosted SPARQL UI (e.g. http://yasgui.triply.cc) — ideally with LLM integration to help generate queries

[JeancarloBarrios]

As I understand it, the graph database will be queried directly, raw, at this point? Right?

[JeancarloBarrios]

I also came a bit late to the party, but it's worth considering in the future.
https://github.com/oxigraph/oxigraph as a database.
It's from the Linux Foundation, so it's open source, and we can better manage costs and vendor lock-in in the long term. Also they are already using for subtrace blockhain, https://github.com/relationlabs/Relation-Graph

[clevinson]

As I understand it, the graph database will be queried directly, raw, at this point? Right?

Yes, this is really just about getting some basic infrastructure up and running to store the different Regen Network related datasets all in a single graph database. More for exploration and to have that has a part of the pipeline, rather than thinking of this as the true "architecture" for an on-chain system or pinning service that indexes datasets anchored on-chain into a graph database.

Does using Amazon Neptune have more vendor lock-in than a managed Postgres database? In the Postgres case, I believe RND is using tools like Heroku's managed Postgres or Amazon RDS, as opposed to manually managing a Postgres server in an ec2 instance.

I'm looking at hte pricing page and it does seem like neputine actually isn't that cheap (cheapest instance looks like 254USD/mo, punching in some basic info into aws pricing calculator with db.r4.large instance type).

Personally I think vendor lockin isn't too much a concern as long as any solution is a valid SPARQL database using the same API, but maybe because of pricing it would be better to manually manage our own on EC2.

@JeancarloBarrios What do you think of oxigraph versus a more traditional option like Apache Jena, or Blazegraph? Another managed option actually might be using Stardog's Free tier, which I didn't know existed until researching it today.

[JeancarloBarrios]

I could do a comparison if you'd like. Honestly, I only researched it for a day, but I can provide a more comprehensive comparison by tomorrow. It's also reassuring that it's open source (I've also reviewed the code, and it appears to be of high quality). It feels more modern; for example, it has features that allow clients to use WASM binding, making it easier to integrate at some point and facilitating integration with various stacks (at least JavaScript, Python, and Rust), in the case that we wanted to build applications around it. It's being used for multiple projects that intersect with us, including web3, RDF, and knowledge graph use cases. However, anything that supports SPARQL is good enough for the use case.

Edit: I seem to have been wrong about LF,

[paul121]

Hey @clevinson @S4mmyb just doing some initial thinking on the process for GH actions.. something I think we need to align on is the directory/file structure within schema/data/playground. I assume we will have a couple scripts to:

1. validate all yaml files in these directories
2. convert all yaml files to RDF/JSONLD + send to Graph

For both the linkml-validate and linkml-convert commands we need to pass in a --target-class flag with value eg CreditProjectInfo. So we need to consider how we identify the target class for each yaml file...

Some options might be: (just using C01-001 as an example ID...)

1. Create directories for each target class within schema/data/playground/{target-class-name}?
   • So schema/data/playground/C01ProjectInfo/C01-001.yaml + schema/data/playground/CreditClassInfo/C01.yaml ?
   • This would work, but might start to be confusing for projects/files across many credit classes that use the same target class like CreditProjectInfo:
     • schema/data/playgound/CreditProjectInfo/C01-001.yaml - C01-010.yaml, also C06-001.yaml - C06-010.yaml ...
2. So maybe it would be nice to have a top-level credit class directory that contains all metadata files for that class (credit class + project)?
   • schema/data/playground/C01/CreditProjectInfo/C01-001.yaml + C01-002.yaml + etc (nice because it only contains this credit class's project data, especially if we consider additional future metadata (batches, etc))
   • schema/data/playground/C01/CreditClassInfo/C01.yaml (this CreditClassInfo sub-directory would only ever have 1 file, but results in better overall organization?)

I think the second option would be my preference. But maybe there is something more elegant? I pushed a WIP commit to #41 to capture some of my ideas on GH action process. Will continue working on this next week!

[S4mmyb]

@JeancarloBarrios What do you think of oxigraph versus a more traditional option like Apache Jena, or Blazegraph? Another managed option actually might be using Stardog's Free tier, which I didn't know existed until researching it today.

It's a bummer that Neptune is expensive, and given our goals I don't think it makes sense at this point. BlazeGraph isn't maintained anymore, but I'm in support of exploring oxigraph if it makes sense. The one thing with that though, when comparing it to something like Jena, is that it doesn't support any reasoning. Apache has a bunch of reasoning tools which support RDFS, Owl, and Rules reasoners which might be nice to have in an instance. I think for us most importantly RDFS (link here).

[S4mmyb]

I think the second option would be my preference. But maybe there is something more elegant? I pushed a WIP commit to #41 to capture some of my ideas on GH action process. Will continue working on this next week!

I support going with option 2! Let's do that.