sds-storage-writer-ts

TypeScript RDF-Connect processor for ingesting SDS streams and generating persistent Linked Data Event Stream (LDES) or in general TREE fragmentations.
The processor consumes an SDS stream and stores all data, metadata, and fragmentation structures into a supported storage backend.

Currently supported storage systems:

• MongoDB
• Redis

This repository exposes one processor:

---

rdfc:IngestSDS

Processor that ingests an SDS stream to generate a Linked Data Event Stream.
Uses sds:Bucket to describe fragmentation (see rdfc:Bucketize).

This processor stores:

• SDS members (data)
• SDS metadata
• SDS buckets and relations
• SDS fragmentation structure

in a persistent database backend, making the stream available for LDES servers to serve and traverse.

---

Import

@prefix owl: <http://www.w3.org/2002/07/owl#>.

<> owl:imports <./node_modules/@rdfc/sds-storage-writer-ts/processor.ttl>.

---

Example Configuration

@prefix rdfc: <https://w3id.org/rdf-connect#>.
@prefix owl:  <http://www.w3.org/2002/07/owl#>.

### Import processor definitions
<> owl:imports <./node_modules/@rdfc/sds-storage-writer-ts/processor.ttl>.

### Define readers
<dataInput> a rdfc:Reader.
<metadataInput> a rdfc:Reader.

### Configure the processor
<ingestor> a rdfc:IngestSDS;
    rdfc:dataInput <dataInput>;
    rdfc:metadataInput <metadataInput>;
    rdfc:database [
        rdfc:url "mongodb://localhost:27017/ldes";
        rdfc:metadata "meta";
        rdfc:data "data";
        rdfc:index "index";
    ].

---

Processor Semantics

Given an SDS stream with a predefined fragmentation:

# Member exists
ex:sample1 a ex:Object;
  ex:x "2";
  ex:y "5".

sds:DataDescription {
  [] sds:stream <bucketizedStream>;
    sds:payload ex:sample1; # Stream membership
    sds:bucket <bucket2>.

  # Bucket relations
  <bucket1> sds:relation [
    sds:relationType tree:GreaterThanRelation;
    sds:relationBucket <bucket2>;
    sds:relationValue 1;
    sds:relationPath ex:x 
  ].
}

The processor will:

• Store the member data ex:sample1
• Store SDS metadata sds:DataDescription and sds:RemoveDataDescription
• Store bucket definitions sds:bucket and sds:relationBucket values
• Store bucket relations sds:relation values
• Persist the full fragmentation structure in the configured storage system.

---

Database Backends

MongoDB

Data is stored in collections defined by:

• rdfc:metadata
• rdfc:data
• rdfc:index

Redis

Redis is supported as an alternative backend with the same logical structure, enabling:

• fast ingestion
• scalable storage

---

Configuration Reference

rdfc:IngestSDS

Property	Type	Required	Description
rdfc:dataInput	rdfc:Reader	yes	SDS member stream
rdfc:metadataInput	rdfc:Reader	yes	SDS metadata stream
rdfc:database	rdfc:DatabaseConfig	yes	Storage backend configuration

---

rdfc:DatabaseConfig

Property	Type	Required	Description
rdfc:metadata	string	yes	Metadata collection name
rdfc:data	string	yes	Data collection name
rdfc:index	string	yes	Index collection name
rdfc:url	string	no	Database connection URL

---

Role in RDF-Connect Pipelines

Typical architecture:

[SDS Stream] + [Members Stream]
↓
rdfc:IngestSDS
↓
MongoDB / Redis
↓
LDES Server

This processor acts as the persistent storage layer between streaming RDF-Connect pipelines and TREE/LDES serving infrastructure.

---

© Ghent University – IMEC
MIT License

Authors:

• Arthur Vercruysse
• Julián Rojas
• Ieben Smessaert