Add more metadata for pipelines by MSattrtand · Pull Request #399 · kbss-cvut/s-pipes

MSattrtand · 2025-07-03T09:29:56Z

Resolves partially kbss-cvut/s-pipes-editor-ui#195
Resolves kbss-cvut/s-pipes-editor-ui#206

blcham

see my comments

s-pipes-core/src/main/java/cz/cvut/spipes/constants/SPIPES.java

s-pipes-core/src/main/java/cz/cvut/spipes/logging/AdvancedLoggingProgressListener.java

s-pipes-core/src/main/java/cz/cvut/spipes/engine/ExecutionEngineImpl.java

s-pipes-core/src/main/java/cz/cvut/spipes/logging/AdvancedLoggingProgressListener.java

s-pipes-debug/src/main/java/cz/cvut/spipes/debug/dto/PipelineExecutionDto.java

s-pipes-core/src/main/java/cz/cvut/spipes/logging/AdvancedLoggingProgressListener.java

blcham · 2025-07-23T07:48:28Z

s-pipes-core/src/main/java/cz/cvut/spipes/logging/AdvancedLoggingProgressListener.java

+                addProperty(moduleExecution, SPIPES.has_script, module.getResource().toString().replaceAll("\\/[^.]*$", ""));
+                if(!metadataMap.containsKey(SPIPES.has_script.toString())){
+                    metadataMap.put(SPIPES.has_script.toString(), module.getResource().toString().replaceAll("\\/[^.]*$", ""));


Suggested change

addProperty(moduleExecution, SPIPES.has_script, module.getResource().toString().replaceAll("\\/[^.]*$", ""));

if(!metadataMap.containsKey(SPIPES.has_script.toString())){

metadataMap.put(SPIPES.has_script.toString(), module.getResource().toString().replaceAll("\\/[^.]*$", ""));

String script = module.getResource().toString().replaceAll("\\/[^.]*$", "");

addProperty(moduleExecution, SPIPES.has_script, script);

if(!metadataMap.containsKey(SPIPES.has_script.toString())){

metadataMap.put(SPIPES.has_script.toString(), script);

duplicate code but i still do not understand why we need to do this ... can you explain me whole this part what is doing, why we need it ?

Done. It saves the script's name as a property to be added to the database, then saves it to metadataMap if it's not there. We need to have it there so we'll have access to it from any function in the logger. E.g. right now I have a code like:
private void persistPipelineExecutionStarted(final EntityManager em, long pipelineExecutionId, Thing pipelineExecution, final String functionName, final String scriptPath)
Where functionName and scriptPath are parameters of the function I've added. To avoid using them as inputs of the function, I should use metadataMap and put these parameters there. It will also allow for easy access to these parameters from any other function if needed.

s-pipes-core/src/main/java/cz/cvut/spipes/constants/SPIPES.java

blcham

see my comments

s-pipes-model/src/main/resources/s-pipes-model.ttl

s-pipes-core/src/main/java/cz/cvut/spipes/logging/AdvancedLoggingProgressListener.java

s-pipes-core/src/main/java/cz/cvut/spipes/constants/SPIPES.java

s-pipes-core/src/main/java/cz/cvut/spipes/engine/ExecutionEngineImpl.java

s-pipes-core/src/main/java/cz/cvut/spipes/modules/AbstractModule.java

s-pipes-core/src/main/java/cz/cvut/spipes/engine/ExecutionEngineImpl.java

blcham · 2025-11-14T15:47:30Z

s-pipes-core/src/main/java/cz/cvut/spipes/engine/ExecutionEngineImpl.java

+        } else {
+            scriptPath = module.getScriptPath();
+        }
+        String script = module.getResource().toString().replaceAll("\\/[^.]*$", "");


I have no idea what following is doing ..

I refactored, so it would be more readable

s-pipes-model/src/main/resources/s-pipes-model.ttl

s-pipes-core/src/main/java/cz/cvut/spipes/logging/AdvancedLoggingProgressListener.java

blcham · 2025-11-19T15:33:45Z

s-pipes-core/src/main/java/cz/cvut/spipes/logging/AdvancedLoggingProgressListener.java

            addProperty(pipelineExecution, SPIPES.has_pipeline_execution_duration, computeDuration(startDate, finishDate));
-            addProperty(pipelineExecution, SPIPES.has_pipeline_name, pipelineName);
-//            addScript(pipelineExecution, scriptManager.getScriptByContextId(pipelineName));
+            pipelineExecution.setTypes(Collections.singleton(Vocabulary.s_c_finished_pipeline_execution));


why we replace? ... not sure if we do not want to addTypes instead of setTypes ... multiple types should coexist

Moreover, why we had to add em.getTransaction().commit() now? How about using addProperty(pipelineExecution, RDF_TYPE, Vocabulary.s_c_finished_pipeline_execution) instead

We didn't have to add em.getTransaction().commit(), as git shows it as a difference, even though it was already there.

We're setting type with pipelineExecution.setTypes in other places, so I decided to follow the same path. I've introduced the new function, addTypes, which allows us to add a new type to the pipeline without replacing it.

Add missing properties to Vocabulary

blcham · 2025-11-19T16:58:57Z

@blcham it is needed to review:

blcham · 2025-11-19T16:44:20Z

s-pipes-core/src/main/java/cz/cvut/spipes/manager/SPipesScriptManager.java

-        return getLocation(resourceContextUri);
+        String location = getLocation(resourceContextUri);
+
+        return Paths.get(location).toUri().toString();


This is weird, why do we need to do this? What is the difference between scriptPath and scriptURI?

I'm getting exceptions like this:

ERROR c.c.s.l.AdvancedLoggingProgressListener - Transaction handling error: Not a valid (absolute) IRI: /host_mnt/c/Users/MSattrtand/IntelliJProjects/s-pipes/doc/examples/hello-world/hello-world.sms.ttl [line 21]

The call chain is:

runService -> createFunctionContext -> getFunctionLocation -> getLocation -> OntDocumentManager.doAltURLMapping

According to the latest function Javadoc, it returns "The resolvable location of the alternative copy, if known, or uri otherwise". We know its local path, so we return it, not URI.

context.setScriptUri sets this local path as "_pScriptURI" parameter.

During the pipeline excecution we're getting to this call chain:

moduleExecutionFinished -> writeRawData.

Model, created from our context, contains this triple:

http://onto.fel.cvut.cz/ontologies/s-pipes/query_solution_1763649074917/_pScriptURI http://onto.fel.cvut.cz/ontologies/s-pipes/has_bound_value /host_mnt/c/Users/MSattrtand/IntelliJProjects/s-pipes-editor-ui/deploy/../../s-pipes/doc/examples/hello-world/hello-world.sms.ttl;, so the triple with the absolute path, not URI.

connection.add( new StringReader(w.getBuffer().toString()), "", RDFFormat.N3, connection.getValueFactory().createIRI(contextUri.toString()));

Will throw the exception I've described before.

"_pScriptPath" parameter is not used anywhere, since executionContext was introduced, so I'll delete it.

The best solution IMO would be to modify getLocation, so it would return URI, even if it doAltURLMapping returns just a path.

blcham · 2025-11-19T17:18:30Z

s-pipes-core/src/main/java/cz/cvut/spipes/engine/ExecutionEngineImpl.java

+        if (inputContext.getScriptUri() != null) {
+            scriptPath = inputContext.getScriptFile().toString();
+        } else {
+            scriptPath = "not defined";


does not make sense

Changed to null

MSattrtand requested a review from blcham July 3, 2025 09:36

This comment was marked as outdated.

Sign in to view

MSattrtand force-pushed the improve-pipeline-metadata branch from 12cec5c to 0f355a6 Compare July 18, 2025 13:12

blcham force-pushed the improve-pipeline-metadata branch from 0f355a6 to cdd3d2d Compare July 18, 2025 15:29

blcham requested changes Jul 19, 2025

View reviewed changes

blcham reviewed Jul 19, 2025

View reviewed changes

s-pipes-core/src/main/java/cz/cvut/spipes/engine/ExecutionEngineImpl.java Outdated Show resolved Hide resolved

blcham reviewed Jul 22, 2025

View reviewed changes

blcham self-requested a review July 23, 2025 07:45

blcham reviewed Jul 23, 2025

View reviewed changes

blcham self-requested a review July 23, 2025 08:11

blcham requested changes Jul 23, 2025

View reviewed changes

MSattrtand force-pushed the improve-pipeline-metadata branch from 3f011cc to adc41d3 Compare October 10, 2025 10:27

MSattrtand force-pushed the improve-pipeline-metadata branch from 0412d55 to 516ae21 Compare October 17, 2025 12:54

blcham force-pushed the improve-pipeline-metadata branch from 516ae21 to b7be03e Compare November 14, 2025 15:23

blcham reviewed Nov 14, 2025

View reviewed changes

s-pipes-model/src/main/resources/s-pipes-model.ttl Outdated Show resolved Hide resolved

blcham reviewed Nov 14, 2025

View reviewed changes

s-pipes-core/src/main/java/cz/cvut/spipes/logging/AdvancedLoggingProgressListener.java Outdated Show resolved Hide resolved

blcham reviewed Nov 19, 2025

View reviewed changes

Evgenii Grigorev added 12 commits November 19, 2025 16:49

Add more metadata for pipelines

d2efdb4

Add interface for pipelineExecutionFailed to ProgressListener.java

5048050

Removed workarounds that are no longer necessary

16eed12

Changes status from literal to IRI

f011065

Removed exessive logging

b6b52a2

Renamed parameters to match their usage

5a52d16

Change the script type to URI from string

f4b54d8

Move extraction of script URI to the separate function

bc66053

Fix incorrect .ttl file

2dda96f

Remove duplicate code

3e631b0

Change Strings to URIs

4a35153

Change IRI of pipeline_execution_statu

43ab268

Evgenii Grigorev added 8 commits November 19, 2025 16:49

Fix metadataMap not being cleared

dd764e2

Fix execution page not working

776cdac

Update ExecutionEngineImpl to utilise data from inputContext

65c9f7f

Fix the test failure

a570b1b

Remove redundant code

c831da5

Change pipeline_execution_status to rdf:type

98e5b80

Fix exception occurring when location is not a URI

b5b4a9c

Refactor variable names

644775c

blcham force-pushed the improve-pipeline-metadata branch from 0300743 to 644775c Compare November 19, 2025 15:49

blcham added 3 commits November 19, 2025 16:51

Remove unused method

67c37b4

Refactor

faa4314

Add missing properties to Vocabulary

Fix JsonLd mapping of SPipes Debug REST API

94c6ae3

blcham reviewed Nov 19, 2025

View reviewed changes

blcham and others added 3 commits November 20, 2025 10:00

Change DTO for pipeline execution

aaf5d65

Improve logging

b109b9b

Fix local path being returned causing exception

42ed655

Conversation

MSattrtand commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

blcham left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

blcham left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blcham commented Nov 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MSattrtand commented Jul 3, 2025 •

edited

Loading