Suggestions to make a proper tool out of ontodocker.py

[samwaseda]

I usually don't like people copying and pasting a response coming from ChatGPT, but I find it pretty valid, so I would like to do it anyway, and maybe continue the discussion:

To turn ontodocker.py from a useful helper module into a proper, reusable tool/library, I’d suggest changes along four axes: API design, robustness, usability, and operability. Below is a concrete, engineering-oriented checklist.

---

1. API & Architecture: from functions → client

1.1 Introduce a first-class client object

Right now, state is implicit (address, token, headers passed around). A proper tool should make state explicit.

class OntodockerClient:
    def __init__(self, base_url, token=None, timeout=30):
        ...

Benefits:

• Centralized auth handling
• Shared session (connection pooling)
• Cleaner public API

Example usage:

client = OntodockerClient("https://example.org", token=TOKEN)
client.datasets.list()
client.datasets.create("my_dataset")
client.sparql.query("my_dataset", QUERY)

---

1.2 Separate concerns into submodules

Currently, HTTP, SPARQL, parsing, and I/O are mixed.

Suggested structure:

courier/
  ontodocker/
    client.py
    datasets.py
    sparql.py
    models.py
    exceptions.py

This makes:

• Testing easier
• Responsibilities clearer
• Extensions safer

---

2. Error Handling & Contracts

2.1 Replace generic exceptions with domain exceptions

Instead of:

raise RuntimeError("Something went wrong")

Define:

class OntodockerError(Exception): ...
class DatasetNotFound(OntodockerError): ...
class AuthenticationError(OntodockerError): ...
class SparqlQueryError(OntodockerError): ...

Benefits:

• Callers can recover programmatically
• CLI tooling becomes feasible

---

2.2 Validate assumptions explicitly

Examples:

• Endpoint URL format
• Dataset existence before upload/delete
• SPARQL result shape vs expected columns

Fail early and clearly, not via downstream KeyError or IndexError.

---

3. Data Modeling: stop passing raw strings everywhere

3.1 Introduce lightweight data models

Instead of passing strings like:

"https://host/api/v1/jena/myds/sparql"

Use:

@dataclass(frozen=True)
class Dataset:
    name: str
    sparql_endpoint: str
    graph_endpoint: str

Benefits:

• Self-documenting
• Fewer parsing bugs
• IDE/static-typing friendly

---

3.2 Typed SPARQL results

Right now, send_query returns a DataFrame unconditionally.

Better:

• Return raw result object

• Provide adapters:

  • to_dataframe()
  • to_dicts()
  • to_rdf_graph()

This avoids forcing pandas on all users.

---

4. Authentication & Configuration

4.1 Multiple auth strategies

Support:

• Token
• Environment variable
• .netrc
• Explicit headers

Example:

OntodockerClient.from_env()

---

4.2 Centralized request handling

Wrap requests.Session with:

• Retry logic
• Timeouts
• Consistent headers
• Logging hooks

This prevents every function from re-implementing HTTP semantics.

---

5. CLI: the biggest step toward “proper tool”

A thin CLI layer immediately upgrades usefulness.

Example:

ontodocker datasets list
ontodocker datasets create myds
ontodocker datasets upload myds data.ttl
ontodocker sparql query myds query.rq --csv

Implementation:

• argparse or typer
• CLI calls client methods (no logic duplication)

This makes the tool usable without Python scripting.

---

6. Observability & Debuggability

6.1 Structured logging

Replace silent failures or print-style debugging with:

logger.debug("Uploading dataset", dataset=name, size=...)

Allow users to enable verbose mode.

---

6.2 Dry-run / preview modes

Especially useful for:

• Deletes
• Uploads
• Dataset creation

---

7. Testing & Guarantees

7.1 Contract tests against a mock server

Use:

• responses or httpx_mock
• Recorded Ontodocker API fixtures

This protects against:

• API drift
• URL shape changes

---

7.2 Type hints everywhere

Add full typing:

def get_datasets(self) -> list[Dataset]:

This improves:

• Readability
• Refactoring safety
• IDE support

---

8. Documentation: make intent explicit

Minimum viable docs:

• “What is Ontodocker?”
• Dataset lifecycle diagram
• Auth setup
• 3 end-to-end examples

Optional but powerful:

• Architecture section explaining REST vs SPARQL roles
• Failure modes and recovery patterns

---

Bottom line

Right now, ontodocker.py is:

A competent internal utility module

With the steps above, it becomes:

A stable client library + CLI for ontology-backed RDF infrastructure

If you want, I can:

• Sketch a public API design
• Refactor one function as an example
• Propose a minimal v1.0 roadmap (what to do now vs later)

Just tell me the intended audience: internal developers, power users, or external third-party users.

[samwaseda]

@mbruns91 What do you think? I guess if we manage to do 1.2 first, we can make a lot of changes independently to make the module truly useful. From my side it would be great if you could involve me with the implementation of some of the stuff above.

[mbruns91]

I mostly like the suggestions.

• Regarding 1: Given the fact we also want to support other platforms than ontodocker, the layout of submodules might not fit all cases, but let's just start in the suggested direction.
• Regarding 2: Exception handling is definitly better in the suggested fashion.
• Regarding 3: That#s also good. I like packing datasets in a dataclass. Regarding SPARQL, I agree, that forcing pandas upon users could be avoided. I still think it is a good default; providing more adapters might make sense, but it's not like pandas should be expected to be exotic for the users of courier. The queries themselves I want to model entirely different by wrapping them in a dataclass

@dataclass
class SparqlQuery:
    """
    query:   the SPARQL query text
    qvars:    expected variable order (without '?'), or None to infer from results
    headers: display headers (same length as vars); if None, fall back to qvars
    """
    query: str = ""
    qvars: list[str] | None = None
    headers: list[str] | None = None

    def resolved_headers(self) -> list[str]:
        if self.headers:
            return self.headers
        return self.qvars or []

to pride a way to integrate knowledge about the expected response from the very beginning and to give an option to differentiate between SPARQL variables and "names" for their associated quantities. This also makes the creation of "query collections" straight forward by simply defining a set functions along this direction:

def query_graph(
    limit: int | None = None,
    ) -> SparqlQuery:
    """
    Query all triples in the graph.
    - If explicit_qvars=True, use SELECT ?s ?p ?o (columns known: s,p,o).
    - If explicit_qvars=False, use SELECT * (columns inferred after execution).
    """
    limit_clause = "" if limit is None else f"LIMIT {limit}"
    query = f"""
    SELECT ?s ?p ?o
    WHERE {{ ?s ?p ?o }}
    {limit_clause}
    """.strip()
    qvars = ["s", "p", "o"]
    headers = ["subject", "predicate", "object"]
    return SparqlQuery(query=query, qvars=qvars, headers=headers)

• Regarding 4: I would not overengineer the authentication at this point. json webtokens are pretty standard and will cover almost all use-cases.
• Regarding 5: I don't think we need a CLI at this point, if ever.
• Regarding 6: I think "print-style" exception raising is totally fine here. Logging would be more meaningful in a higher-level tool.
• Regarding 7: I know, I have to write the tests asap...
• Regarding 8: I know, I have to write the docs...

[mbruns91]

@samwaseda what I would still like to work out is the semantikon use-case beyond a simple KNG upload, which is, of course, valid, yet very simple.

Could it be interesting to allow a collaborative work on workflow-recipies? Is the mapping "workflow -> KNG representation" truly bijective? in this case, multiple collaborators could update a hosted KNG representation via SPARQL. courier would be the interface to the hosted graph and maybe also collision handling could be handled by it.

[samwaseda]

Could it be interesting to allow a collaborative work on workflow-recipies? Is the mapping "workflow -> KNG representation" truly bijective? in this case, multiple collaborators could update a hosted KNG representation via SPARQL. courier would be the interface to the hosted graph and maybe also collision handling could be handled by it.

That depends at little bit on the situation. If you mean two people working in the same field (such as continuum), then probably it makes more sense for them to do it in the WfMS of their choice (such as pyiron). But if you mean people from different fields working together, then yes, and the interaction will be less the workflow recipe, but more via the actual physical objects in the knowledge graph, i.e. if an experimentalist needs an elastic tensor and a theoretician can provide it, then they should use the same knowledge graph with the common class for the elastic tensor. Ideally, all the recipes should be connected this way in the end.

[samwaseda]

I still think it is a good default; providing more adapters might make sense, but it's not like pandas should be expected to be exotic for the users of courier.

I guess it's not so much about the user experience, but it's more about whether it can be represented by pandas data frame or not. I'm not really sure what kind of data can be stored on the triple store, but there's a possibility that you get something like a numpy array or other objects. In that case the user basically loses the possibility of obtaining the data in the first place.