Skip to content

README.md

Latest commit

 

History

History
166 lines (120 loc) · 4.54 KB

README.md

File metadata and controls

166 lines (120 loc) · 4.54 KB

sds-storage-writer-ts

Build and tests with Node.js
npm

TypeScript RDF-Connect processor for ingesting SDS streams and generating persistent Linked Data Event Stream (LDES) or in general TREE fragmentations.
The processor consumes an SDS stream and stores all data, metadata, and fragmentation structures into a supported storage backend.

Currently supported storage systems:

  • MongoDB
  • Redis

This repository exposes one processor:

rdfc:IngestSDS

Processor that ingests an SDS stream to generate a Linked Data Event Stream.
Uses sds:Bucket to describe fragmentation (see rdfc:Bucketize).

This processor stores:

  • SDS members (data)
  • SDS metadata
  • SDS buckets and relations
  • SDS fragmentation structure

in a persistent database backend, making the stream available for LDES servers to serve and traverse.

Import

@prefix owl: <http://www.w3.org/2002/07/owl#>.

<> owl:imports <./node_modules/@rdfc/sds-storage-writer-ts/processor.ttl>.

Example Configuration

@prefix rdfc: <https://w3id.org/rdf-connect#>.
@prefix owl:  <http://www.w3.org/2002/07/owl#>.

### Import processor definitions
<> owl:imports <./node_modules/@rdfc/sds-storage-writer-ts/processor.ttl>.

### Define readers
<dataInput> a rdfc:Reader.
<metadataInput> a rdfc:Reader.

### Configure the processor
<ingestor> a rdfc:IngestSDS;
    rdfc:dataInput <dataInput>;
    rdfc:metadataInput <metadataInput>;
    rdfc:database [
        rdfc:url "mongodb://localhost:27017/ldes";
        rdfc:metadata "meta";
        rdfc:data "data";
        rdfc:index "index";
    ].

Processor Semantics

Given an SDS stream with a predefined fragmentation:

# Member exists
ex:sample1 a ex:Object;
  ex:x "2";
  ex:y "5".

sds:DataDescription {
  [] sds:stream <bucketizedStream>;
    sds:payload ex:sample1; # Stream membership
    sds:bucket <bucket2>.

  # Bucket relations
  <bucket1> sds:relation [
    sds:relationType tree:GreaterThanRelation;
    sds:relationBucket <bucket2>;
    sds:relationValue 1;
    sds:relationPath ex:x 
  ].
}

The processor will:

  • Store the member data ex:sample1
  • Store SDS metadata sds:DataDescription and sds:RemoveDataDescription
  • Store bucket definitions sds:bucket and sds:relationBucket values
  • Store bucket relations sds:relation values
  • Persist the full fragmentation structure in the configured storage system.

Database Backends

MongoDB

Data is stored in collections defined by:

  • rdfc:metadata
  • rdfc:data
  • rdfc:index

Redis

Redis is supported as an alternative backend with the same logical structure, enabling:

  • fast ingestion
  • scalable storage

Configuration Reference

rdfc:IngestSDS

Property Type Required Description
rdfc:dataInput rdfc:Reader yes SDS member stream
rdfc:metadataInput rdfc:Reader yes SDS metadata stream
rdfc:database rdfc:DatabaseConfig yes Storage backend configuration

rdfc:DatabaseConfig

Property Type Required Description
rdfc:metadata string yes Metadata collection name
rdfc:data string yes Data collection name
rdfc:index string yes Index collection name
rdfc:url string no Database connection URL

Role in RDF-Connect Pipelines

Typical architecture:

[SDS Stream] + [Members Stream]
↓
rdfc:IngestSDS
↓
MongoDB / Redis
↓
LDES Server

This processor acts as the persistent storage layer between streaming RDF-Connect pipelines and TREE/LDES serving infrastructure.

© Ghent University – IMEC
MIT License

Authors:

  • Arthur Vercruysse
  • Julián Rojas
  • Ieben Smessaert