You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/english/blog/post-intertwin-summary.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,11 +11,11 @@ description: "Digital Twin science made serverless with OSCAR within InterTwin.
11
11
draft: false
12
12
---
13
13
14
-
This post summarizes the work carried out in the InterTwin project to extend the OSCAR serverless platform across the cloud–HPC continuum. We developed DCNiOS, a Data Connector through Apache NiFi for OSCAR, that facilitates the creation of event-driven dataflows connecting storages system like dCache, S3 (via SQS), Kafka, and Rucio, enabled seamless offloading of OSCAR workloads to HPC via interLink, added interactive Jupyter notebooks as exposed services, and integrated Common Workflow Language (CWL) using `oscar-python` from them.
14
+
This post summarizes the work carried out in the [InterTwin](https://intertwin.eu) project to extend the OSCAR serverless platform across the cloud–HPC continuum. We developed [DCNiOS](https://github.com/interTwin-eu/dcnios), a Data Connector through [Apache NiFi](https://nifi.apache.org) for OSCAR, that facilitates the creation of event-driven dataflows connecting storages system like dCache, S3 (via SQS), Kafka, and Rucio, enabled seamless offloading of OSCAR workloads to HPC via interLink, added interactive Jupyter notebooks as exposed services, and integrated [Common Workflow Language (CWL)](https://www.commonwl.org) using [`oscar-python`](https://github.com/grycap/oscar_python) from them.
15
15
16
16
### Data-driven ingestion with DCNiOS
17
17
18
-
We built DCNiOS to connect dCache storage events with OSCAR through Apache NiFi. DCNiOS provides a YAML-based approach and a CLI tool to define and deploy the dataflow that listens to dCache Server-Sent Events (SSE) and triggers OSCAR services.
18
+
We built DCNiOS to connect [dCache](http://dcache.org) storage events with OSCAR through Apache NiFi. DCNiOS provides a YAML-based approach and a CLI tool to define and deploy the dataflow that listens to dCache Server-Sent Events (SSE) and triggers OSCAR services.
19
19
20
20
- What it solves: decouples ingestion rate (dCache) from processing rate (OSCAR) while keeping flows reconfigurable.
21
21
- How it’s delivered: a NiFi-based image with SSE client support and reusable process groups.
@@ -25,9 +25,9 @@ We built DCNiOS to connect dCache storage events with OSCAR through Apache NiFi.
25
25
26
26
We broadened the DCNiOS capabilities to support new event sources for triggering OSCAR services:
27
27
28
-
-S3 via SQS: process object-storage events at scale.
29
-
- Kafka: stream-processing triggers for high-throughput event flows.
30
-
- Rucio: scientific data management events feeding directly into OSCAR services.
28
+
-[Amazon S3](https://aws.amazon.com/s3) via [SQS](https://aws.amazon.com/sqs): process object-storage events at scale.
29
+
-[Kafka](https://kafka.apache.org/): stream-processing triggers for high-throughput event flows.
30
+
-[Rucio](https://rucio.cern.ch): scientific data management events feeding directly into OSCAR services.
31
31
32
32
These sources are orchestrated through Apache NiFi dataflows and set up with DCNiOS, complementing existing triggers and enabling flexible scientific pipelines.
0 commit comments