Merge pull request #108 from dice-group/develop

Code for WCOJ update paper (#107)

Author: bigerl

CMakeLists.txt

@@ -1,5 +1,5 @@
 cmake_minimum_required(VERSION 3.24)
-project(tentris VERSION 1.4.0
+project(tentris VERSION 1.5.0
         DESCRIPTION "Tentris - A tensor-based Triplestore.")
 
 include(cmake/boilerplate_init.cmake)

README.MD

@@ -1,86 +1,85 @@
-# Tᴇɴᴛʀɪs: A Tensor-based Triple Store
+# Tentris: A Tensor-based Triple Store
 
-<p><img src = "https://tentris.dice-research.org/iswc2020/assets/img/Tentris_logo.svg" alt = "Tᴇɴᴛʀɪs Logo" width = "30%" align = "center"></p>
+Tentris is a tensor-based triplestore that natively supports worst-case optimal joins.
 
-Tᴇɴᴛʀɪs is a tensor-based RDF triple store with SPARQL support. It is introduced and described in:
-> [Alexander Bigerl, Felix Conrads, Charlotte Behning, Mohamed Ahmed Sherif, Muhammad Saleem and Axel-Cyrille Ngonga Ngomo (2020)
-**Tentris – A Tensor-Based Triple Store.
-** In: The Semantic Web – ISWC 2020](https://tentris.dice-research.org/iswc2020/)
+This is the research version of Tentris. A commercial version is available at https://github.com/tentris/tentris.
 
-and
+## Features
 
-> [Alexander Bigerl, Lixi Conrads, Charlotte Behning, Muhammad Saleem and Axel-Cyrille Ngonga Ngomo (2022) Hashing the Hypertrie: Space- and Time-Efficient Indexing for SPARQL in Tensors. In: The Semantic Web – ISWC 2022 Hashing the Hypertrie: Space- and Time-Efficient Indexing for SPARQL in Tensors](https://tentris.dice-research.org/iswc2022/)
+- SPARQL endpoint supporting `SELECT`, `SELECT DISTINCT` and `ASK` queries at `/sparql` and `/stream`. The `WHERE` block
+  may contain basic graph patterns and OPTIONAL clauses.
+  Queries must be well-designed [Kaminski et al. (2018)](https://doi.org/10.1007/s00224-017-9802-9)
+- Supports `INSERT DATA`/`DELETE DATA` updates that are synchronized with a reader-writer lock at `/update`.
+- Supports streaming results through the `/stream` endpoint for large results.
+- Tentris is a persistent triplestore that uses [metall](https://github.com/LLNL/metall) to manage its disk-based index.
+  Note that previous versions were in-memory.
 
-## Get It
+## Download
 
-* download [static prebuilt binaries](https://github.com/dice-group/tentris/releases)
-  and [try them out](#running-tentris)
-* [build it with docker](#docker)
+We provide pre-built binaries in the [releases](https://github.com/dice-group/tentris-research-project/releases) that run on Linux machines with fairly recent (after 2015) x86_64 CPUs.
 
-## Running Tᴇɴᴛʀɪs
+## Running
 
-<details><summary> </summary>
+Initialize Tentris' index with an empty graph with:
 
-#### Bulk-load Data
+```shell
+./tentris_loader -f <(echo "")
+```
 
-Provide an NTRIPLE or TURTLE file to build the an index. By default, the index is stored in the current directory. The
-path can be changed with the option `--storage`.
+Or initialize it by loading a turtle or ttl file:
 
 ```shell
 tentris_loader --file my_nt_file.nt
 ```
 
-#### Start HTTP endpoint
-
-To start Tᴇɴᴛʀɪs as a HTTP endpoint on port 9080 run now:
+Start the Tentris endpoint at [127.0.0.1:9080/sparql](http://127.0.0.1:9080/sparql):
 
 ```
 tentris_server -p 9080
 ``` 
 
-#### Query
-
-The SPARQL endpoint may now be queried locally at: `127.0.0.1:9080/sparql?query=*your query*`. You can execute queries
-with the following curl command:
+Run a query with :
 
 ```shell
 curl -G \
 --data-urlencode 'query=SELECT * WHERE { ?s ?p ?o . }' \
 '127.0.0.1:9080/sparql'
 ```
 
-If you want to type the query in your browser, the query string must be URL encoded. You can use any online URL encoder
-like <https://meyerweb.com/eric/tools/dencoder>.
-
-The following endpoints are available:
-Available endpoints:
+Or update your data with:
 
-- HTTP GET `/sparql?query=` for normal queries
-- HTTP GET `/stream?query=` for queries with huge results
-- HTTP GET `/count?query=` as a workaround for count (consumes a select query)
-
-</details>
+```shell
 
-## Docker
+curl -X POST \
+-H 'Content-Type: application/sparql-update' \
+--data 'INSERT DATA { <http://example.org/subject> <http://example.org/predicate> "object" . }' \
+'127.0.0.1:9080/sparql'
+```
 
-<details><summary> </summary>
+The following endpoints are available:
 
-Use the [Dockerfile](./Dockerfile) to build tentris.
+- HTTP GET `/sparql` for normal SPARQL queries
+- HTTP GET `/stream` for SPARQL queries with huge results
+- HTTP POST `/update` for `INSERT DATA` and `DELETE DATA` updates
 
-* A docker image is available on [docker hub](https://hub.docker.com/r/dicegroup/tentris_server). Get it with
-    ```shell script
-    docker build -f Dockerfile .
-    docker pull dicegroup/tentris_server
-    ```
+## Build It
 
-</details>
+Tentris is known to build on Ubuntu 22.04 and newer.
+Building was tested with Clang 17 & 19. As C++ standard template library, only `libstdc++11` (v13) was tested. For
+details
+refer to the [Dockerfile](./Dockerfile) or GitHub actions.
 
-## Build It Yourself
+# Research
 
-<details><summary> </summary>
+- Alexander Bigerl, Liss Heidrich, Nikolaos Karalis and Axel-Cyrille Ngonga Ngomo (2025) **Efficient Updates for
+  Worst-Case Optimal Join Triple Stores.** In: The Semantic Web – ISWC
+  2025 | [website](https://github.com/dice-group/tentris-wcoj-with-updates/)
 
-Tᴇɴᴛʀɪs is known to build on Ubuntu 22.04 and newer.
-Building was tested with Clang 17 & 19. As standard library, only libstdc++11 (v13) was tested. For details
-refer to the [Dockerfile](./Dockerfile) or github actions.
+- Alexander Bigerl, Lixi Conrads, Charlotte Behning, Muhammad Saleem and Axel-Cyrille Ngonga Ngomo (2022) **Hashing the
+  Hypertrie: Space- and Time-Efficient Indexing for SPARQL in Tensors.** In: The Semantic Web – ISWC
+  2022 | [website](https://tentris.dice-research.org/iswc2022/)
 
-</details>
+- Alexander Bigerl, Felix Conrads, Charlotte Behning, Mohamed Ahmed Sherif, Muhammad Saleem and Axel-Cyrille Ngonga
+  Ngomo (2020)
+  **Tentris – A Tensor-Based Triple Store.**
+  In: The Semantic Web – ISWC 2020 | [website](https://tentris.dice-research.org/iswc2020/)

conanfile.py

@@ -24,7 +24,7 @@ class Recipe(ConanFile):
     generators = "CMakeDeps", "CMakeToolchain"
 
     def requirements(self):
-        self.requires("hypertrie/0.9.6", transitive_headers=True)
+        self.requires("hypertrie/0.10.0", transitive_headers=True)
         self.requires("rdf4cpp/0.0.27.1", transitive_headers=True)
         self.requires("sparql-parser-base/0.3.6")
         self.requires("unordered_dense/4.4.0", transitive_headers=True, force=True)

libs/endpoint/CMakeLists.txt

@@ -13,17 +13,17 @@ add_library(${lib}
         src/dice/endpoint/SparqlEndpoint.cpp
         src/dice/endpoint/CountEndpoint.cpp
         src/dice/endpoint/SparqlStreamingEndpoint.cpp
+        src/dice/endpoint/SparqlUpdateEndpoint.cpp
         src/dice/endpoint/SparqlQueryCache.cpp
         src/dice/endpoint/Endpoint.cpp
         )
 add_library(${PROJECT_NAME}::${lib_suffix} ALIAS ${lib})
 
-target_include_directories(${lib}
-        PUBLIC
+target_include_directories(${lib} PUBLIC
         $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/src>
         PRIVATE
         ${CMAKE_CURRENT_SOURCE_DIR}/private-include
-        )
+)
 
 target_link_libraries(${lib} PUBLIC
         ${PROJECT_NAME}::triple-store
@@ -34,7 +34,7 @@ target_link_libraries(${lib} PUBLIC
         spdlog::spdlog
         cppitertools::cppitertools
         rapidjson
-        )
+)
 
 include(${CMAKE_SOURCE_DIR}/cmake/install_components.cmake)
-install_component(PUBLIC ${lib_suffix} src)
+install_component(PUBLIC ${lib_suffix} src)

libs/endpoint/private-include/dice/endpoint/ParseSPARQLUpdateParam.hpp

@@ -0,0 +1,46 @@
+#ifndef TENTRIS_PARSESPARQLUPDATEPARAM_HPP
+#define TENTRIS_PARSESPARQLUPDATEPARAM_HPP
+
+#include <spdlog/spdlog.h>
+
+#include <restinio/helpers/http_field_parsers/content-type.hpp>
+#include <restinio/request_handler.hpp>
+#include <restinio/uri_helpers.hpp>
+
+#include <dice/sparql2tensor/UPDATEQuery.hpp>
+
+
+namespace dice::endpoint {
+
+	struct update_error : std::runtime_error {
+		explicit update_error(const std::string &message)
+			: std::runtime_error(message) {}
+	};
+
+	inline sparql2tensor::UPDATEDATAQueryData parse_sparql_update_param(restinio::request_handle_t &req) {
+		using namespace dice::sparql2tensor;
+		using namespace restinio;
+		auto content_type = req->header().opt_value_of(http_field::content_type);
+		auto content_type_value = http_field_parsers::content_type_value_t::try_parse(*content_type);
+		if (not content_type_value.has_value() or
+			content_type_value.value().media_type.type != "application" or
+			content_type_value.value().media_type.subtype != "sparql-update") {
+			throw update_error("Expected content-type: application/sparql-update");
+		}
+		std::string sparql_update_str{req->body()};
+		try {
+			auto update_query = UPDATEDATAQueryData::parse(sparql_update_str);
+			return update_query;
+		} catch (std::exception &ex) {
+			static constexpr auto message = "Value of parameter 'update' is not parsable: ";
+			throw update_error{std::string{message} + ex.what()};
+		} catch (...) {
+			static constexpr auto message = "Unknown error";
+			throw update_error{message};
+		}
+	}
+
+}// namespace dice::endpoint
+
+
+#endif//TENTRIS_PARSESPARQLUPDATEPARAM_HPP

libs/endpoint/private-include/dice/endpoint/SparqlStreamingEndpoint.hpp

@@ -5,14 +5,14 @@
 
 namespace dice::endpoint {
 
-    class SPARQLStreamingEndpoint final : public Endpoint {
+	class SPARQLStreamingEndpoint final : public Endpoint {
 
-    public:
-        SPARQLStreamingEndpoint(tf::Executor &executor, triple_store::TripleStore &triplestore, SparqlQueryCache &sparql_query_cache, EndpointCfg const &endpoint_cfg);
+	public:
+		SPARQLStreamingEndpoint(tf::Executor &executor, triple_store::TripleStore &triplestore, SparqlQueryCache &sparql_query_cache, EndpointCfg const &endpoint_cfg);
 
-    protected:
-        void handle_query(restinio::request_handle_t req, std::chrono::steady_clock::time_point timeout) override;
-    };
+	protected:
+		void handle_query(restinio::request_handle_t req, std::chrono::steady_clock::time_point timeout) override;
+	};
 }// namespace dice::endpoint
 
-#endif//TENTRIS_SPARQLSTREAMINGENDPOINT_HPP
+#endif//TENTRIS_SPARQLSTREAMINGENDPOINT_HPP

libs/endpoint/private-include/dice/endpoint/SparqlUpdateEndpoint.hpp

@@ -0,0 +1,19 @@
+#ifndef TENTRIS_SPARQLUPDATEENDPOINT_HPP
+#define TENTRIS_SPARQLUPDATEENDPOINT_HPP
+
+#include <dice/endpoint/Endpoint.hpp>
+#include <rapidjson/stringbuffer.h>
+
+namespace dice::endpoint {
+
+	class SPARQLUpdateEndpoint final : public Endpoint {
+	public:
+		SPARQLUpdateEndpoint(tf::Executor &executor, triple_store::TripleStore &triplestore, SparqlQueryCache &sparql_query_cache, EndpointCfg const &endpoint_cfg);
+
+	protected:
+		void handle_query(restinio::request_handle_t req, std::chrono::steady_clock::time_point timeout) override;
+	};
+
+}// namespace dice::endpoint
+
+#endif//TENTRIS_SPARQLUPDATEENDPOINT_HPP

libs/endpoint/src/dice/endpoint/CountEndpoint.cpp

@@ -5,25 +5,25 @@
 
 namespace dice::endpoint {
 
-    CountEndpoint::CountEndpoint(tf::Executor &executor,
-                                 triple_store::TripleStore &triplestore,
-                                 SparqlQueryCache &sparql_query_cache,
-                                 EndpointCfg const &endpoint_cfg)
-        : Endpoint(executor, triplestore, sparql_query_cache, endpoint_cfg) {}
+	CountEndpoint::CountEndpoint(tf::Executor &executor,
+								 triple_store::TripleStore &triplestore,
+								 SparqlQueryCache &sparql_query_cache,
+								 EndpointCfg const &endpoint_cfg)
+		: Endpoint(executor, triplestore, sparql_query_cache, endpoint_cfg) {}
 
-    void CountEndpoint::handle_query(restinio::request_handle_t req, std::chrono::steady_clock::time_point timeout) {
-        using namespace dice::sparql2tensor;
-        using namespace restinio;
+	void CountEndpoint::handle_query(restinio::request_handle_t req, std::chrono::steady_clock::time_point timeout) {
+		using namespace dice::sparql2tensor;
+		using namespace restinio;
 
-        auto sparql_query = parse_sparql_query_param(req, this->sparql_query_cache_);
-        if (not sparql_query)
-            return;
+		auto sparql_query = parse_sparql_query_param(req, this->sparql_query_cache_);
+		if (not sparql_query)
+			return;
 
-        auto const count = this->triplestore_.count(*sparql_query, timeout);
+		auto const count = this->triplestore_.count(*sparql_query, timeout);
 
-        req->create_response(status_ok())
-                .set_body(fmt::format("{}", count))
-                .done();
-        spdlog::info("HTTP response {}: counted {} results", status_ok(), count);
-    }
+		req->create_response(status_ok())
+				.set_body(fmt::format("{}", count))
+				.done();
+		spdlog::info("HTTP response {}: counted {} results", status_ok(), count);
+	}
 }// namespace dice::endpoint

libs/endpoint/src/dice/endpoint/HTTPServer.cpp

@@ -3,6 +3,7 @@
 #include <dice/endpoint/CountEndpoint.hpp>
 #include <dice/endpoint/SparqlEndpoint.hpp>
 #include <dice/endpoint/SparqlStreamingEndpoint.hpp>
+#include <dice/endpoint/SparqlUpdateEndpoint.hpp>
 
 #include <csignal>
 #include <cstring>
@@ -58,6 +59,10 @@ namespace dice::endpoint {
 						  SPARQLEndpoint{executor_, triplestore_, sparql_query_cache_, cfg_});
 		spdlog::info("  GET /sparql?query= for normal queries");
 
+		router_->http_post(R"(/sparql)",
+						   SPARQLUpdateEndpoint{executor_, triplestore_, sparql_query_cache_, cfg_});
+		spdlog::info("  POST  /sparql for update queries");
+
 		router_->http_get(R"(/stream)",
 						  SPARQLStreamingEndpoint{executor_, triplestore_, sparql_query_cache_, cfg_});
 		spdlog::info("  GET  /stream?query= for queries with huge results");