Skip to content

Commit a57464d

Browse files
mbroechelerferenc-csaky
mbroecheler
and
ferenc-csaky
authored
Docs update for 0.7.0 (#1391)
* updating documentation for 0.7 release * add interface definition * add search bar and engine * update front page and readme to clarify the idea behind DataSQRL. --------- Co-authored-by: Ferenc Csaky <[email protected]>
1 parent d109895 commit a57464d

23 files changed

+1389
-141
lines changed

‎README.md‎

Lines changed: 194 additions & 26 deletions
Large diffs are not rendered by default.

‎documentation/blog/2025-07-27-datasqrl-0.7.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
---
2+
slug: datasqrl-0.7-release
3+
title: "DataSQRL 0.7 Release: The Data Delivery Interface"
4+
authors: [matthias]
5+
tags: [release]
6+
---
7+
8+
<head>
9+
<meta property="og:image" content="/img/blog/release_0.7.0.png" />
10+
<meta name="twitter:image" content="/img/blog/release_0.7.0.png" />
11+
</head>
12+
13+
# DataSQRL 0.7 Release: The Data Delivery Interface
14+
15+
<img src="/img/blog/release_0.7.0.png" alt="DataSQRL 0.7.0 Release >|" width="40%"/>
16+
17+
DataSQRL 0.7 marks a major milestone in our journey to automate data pipelines, thanks to significant improvements to the serving layer:
18+
19+
* Support for the Model Context Protocol (MCP) for tooling and resource access
20+
* REST API support
21+
* JWT-based authentication and authorization
22+
23+
These features enable developers to build a wide range of production-ready data interfaces.
24+
This release also includes performance and configuration improvements to the serving layer of DataSQRL-generated pipelines.
25+
26+
You can find the full release notes and source code on our [GitHub release page](https://github.com/DataSQRL/sqrl/releases/tag/0.7.0).
27+
To update your local installation of DataSQRL, simply pull the latest Docker image:
28+
```bash
29+
docker pull datasqrl/cmd:0.7.0
30+
```
31+
32+
## The Last Mile: Data Delivery
33+
34+
Data delivery is the final and most visible stage of any data pipeline. It's how users, applications, and AI agents actually access and consume data. Most enterprise data interactions happen through APIs, making the delivery interface a critical component. At DataSQRL, we've invested heavily in automating the upstream parts of the pipeline: from Flink-powered data processing to Postgres-backed storage. With version 0.7, we turn our focus to the serving layer: introducing support for the Model Context Protocol (MCP) and REST APIs, as well as JWT-based authentication and authorization. These additions ensure seamless integration with most authentication providers and enable secure, token-based data access, with fine-grained authorization logic enforced directly in the SQRL script. This completes our vision of end-to-end pipeline automation, where consumption patterns inform data storage and processing—closing the loop between data production and usage.
35+
36+
Check out the [interface documentation](../docs/interface) for more information.
37+
38+
<!--truncate-->
39+
40+
## Major Contributions
41+
42+
In addition to the flagship features—MCP, REST, and JWT support—which we’ll discuss in more detail in future blog posts, the 0.7.0 release contains a number of additional features and improvements.
43+
44+
### Configuration & CLI Improvements
45+
- Refactored CLI and Flink configuration logic.
46+
- Improved error messages during package config validation.
47+
- Enforced predictable ordering and sorting of config keys.
48+
- Migrated deprecated config key naming (`-dir` → `-folder`), now with compile-time warnings.
49+
- Standardized configuration schema and structured logging to `/build/logs`.
50+
51+
### Testing Infrastructure Enhancements
52+
- Added new `sqrl-container-testing` module.
53+
- Converted tests to use AssertJ.
54+
- Increased test coverage for `SqrlConfig` and dependency mapping.
55+
- Fixed test runner error reporting and exit code handling.
56+
- Reworked dependent service startup to trigger post-compilation.
57+
58+
### Flink-SQL Runner Integration
59+
- Integrated `flink-sql-runner` into `DatasqrlRun`.
60+
- Temporarily merged `sqrl-test` module into `sqrl-run`.
61+
- Simplified Docker image setup (public `ghcr.io` images).
62+
- Updated submodule paths and version to `0.7.0`.
63+
64+
### Authentication & API Enhancements
65+
- Added initial JWT-based authentication support.
66+
- Published documentation for JWT and Swagger-based OpenAPI specs for REST endpoints.
67+
- Added batch GraphQL mutation support with transactional semantics.
68+
- Replaced `GraphQLBigInteger` with native `Long` handling.
69+
70+
### Kafka & Runtime Improvements
71+
- Kafka topic names now support templating.
72+
- Added an async OpenAI test use case and resolved snapshot issues.
73+
- Fixed intermittent WebSocket failures in `SubscriptionClient`.
74+
75+
### Project Structure and CI Pipeline
76+
- Simplified project structure and removed outdated dependency declarations.
77+
- Refactored CI pipeline and added automated GitHub package cleanup workflow.
78+
79+
### Updated Dependencies
80+
81+
Upgraded versions for the following dependencies and patched critical vulnerabilities:
82+
83+
- Apache Flink and Flink connectors (e.g., Postgres CDC)
84+
- Vert.x and related plugins
85+
- Apache Iceberg, DuckDB, PostgreSQL JDBC
86+
- AWS SDK BOM
87+
- JSON Schema Validator, Netty, OpenCSV
88+
- Micrometer, Log4j, Reactor, Immutables, Testcontainers
89+
- Maven plugins: Enforcer, GPG, Build-helper

‎documentation/docs/configuration.md‎

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,7 @@ In terms of security, we support JWT auth, that can be specified under the `conf
8585
|--------------|------------|-----------|---------------------------|
8686
| `config` | **object** | see below | Vert.x JWT configuration. |
8787

88-
```json
88+
```json5
8989
{
9090
"engines": {
9191
"vertx" : {
@@ -95,7 +95,7 @@ In terms of security, we support JWT auth, that can be specified under the `conf
9595
"pubSecKeys": [
9696
{
9797
"algorithm": "HS256",
98-
"buffer": "<auth-secret>"
98+
"buffer": "<signer-secret>" // Base64 encoded signer secret string
9999
}
100100
],
101101
"jwtOptions": {
@@ -166,6 +166,17 @@ Used as a *table-format* engine together with a query engine such as Flink or Sn
166166
"physical": false,
167167
"sorted": true, // deterministic ordering (mostly for tests)
168168
"visual": true
169+
},
170+
171+
"api": {
172+
"protocols": [ // protocols that are being exposed by the server
173+
"GRAPHQL",
174+
"REST",
175+
"MCP"
176+
],
177+
"endpoints": "FULL", // endpoint generation strategy (FULL, GRAPHQL, OPS_ONLY)
178+
"add-prefix": true, // add an operation-type prefix before function names
179+
"max-result-depth": 3 // maximum depth of graph traversal when generating operations from a schema
169180
}
170181
}
171182
}
@@ -307,6 +318,12 @@ The built-in fallback (excerpt - full version [here](https://raw.githubuserconte
307318
"physical": false,
308319
"sorted": true,
309320
"visual": true
321+
},
322+
"api": {
323+
"protocols": ["GRAPHQL", "REST", "MCP"],
324+
"endpoints": "FULL",
325+
"add-prefix": true,
326+
"max-result-depth": 3
310327
}
311328
},
312329
"connectors": {

‎documentation/docs/interface.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# Designing the Interface
2+
3+
Based on the SQRL script, DataSQRL generates the interface for the compiled data pipeline. DataSQRL supports the following interfaces:
4+
5+
* GraphQL (Mutations, Queries, and Subscriptions)
6+
* MCP (Tooling and Resources)
7+
* REST (GET and POST)
8+
* Data Product (Data Lake and Database Views)
9+
10+
The first three are APIs that can be invoked programmatically. These generated API interfaces are configured by the `protocols` [compiler configuration](configuration.md#compiler-compiler).
11+
12+
For data products, DataSQRL generates view definitions as deployment assets in `build/deploy/plan` which can be queried directly.
13+
14+
## Data Products
15+
16+
For data products, each visible table defined in the SQRL script is exposed as a view or physical table depending on the pipeline optimization. The mapping between visible tables in the SQRL script and exposed tables in the interface is 1-to-1.
17+
18+
We recommend generating unique table names for the physical tables by configuring a table-name suffix in the [connector configuration](configuration#connectors-connectors), e.g. by configuring the `table-name` for `postgres` or the `catalog-name` for `iceberg` to `${sqrl:table-name}_MY_SUFFIX` . This separates views from physical tables to provide modularity and support updates without impacting downstream consumers.
19+
20+
## APIs
21+
22+
### GraphQL
23+
24+
DataSQRL uses the GraphQL data model as the base model for all API access to data. The [SQRL language specification](sqrl-language.md#api-mapping) defines how the tables and relationships in the SQRL script map to GraphQL types and fields, as well as how tables map to queries, mutations, and subscriptions.
25+
26+
To generate the GraphQL schema from a SQRL script, add the `--api graphql` option to the [compile command](compiler.md#compile-command).
27+
28+
You can customize the GraphQL schema by:
29+
* Changing field cardinalities (e.g. `[Person]!` to `Person!`)
30+
* Changing scalar types (e.g. `Long` to `Int`)
31+
* Changing the argument name for mutations (e.g. `event` to `payload`)
32+
* Changing the type of fields to compatible types (e.g. `Person` to `SpecificPerson`)
33+
* Adding enums
34+
* Adding interfaces and structuring types with interfaces
35+
36+
:::warning
37+
Changes must be compatible with the underlying data as defined in the SQRL script and the [API mapping](sqrl-language.md#api-mapping).
38+
:::
39+
40+
To use a customized GraphQL schema, configure the schema explicitly in the [script configuration](configuration.md#script-script) or by providing it as a [command argument](compiler.md).
41+
42+
### MCP and REST
43+
44+
DataSQRL generates a list of operations from the GraphQL schema: one for each query and mutation endpoint.
45+
Queries are mapped to MCP tools with a `Get` prefix and REST endpoints under `rest/queries`. If the arguments are simple scalars, the REST endpoint is GET with URL parameters, otherwise POST with the arguments as payload.
46+
Mutations are mapped to MCP tools with an `Add` prefix and REST POST endpoints under `rest/mutations`.
47+
For the result set, DataSQRL follows relationship fields up to a configured depth (and without loops).
48+
49+
You can control the default operation generation with the [compiler configuration](configuration.md#compiler-compiler).
50+
51+
For complete control over the exposed MCP tools and resources as well as REST endpoints, you can define the operations explicitly in a separate GraphQL file (or multiple files).
52+
53+
The GraphQL file defining the operations contains one or multiple query or mutation queries.
54+
The name of the operation is the name of the MCP tool and REST endpoint.
55+
56+
The `@api` directive controls how the operation is exposed:
57+
* `rest`: `NONE`, `GET`, or `POST` to configure the HTTP method or not expose as REST endpoint.
58+
* `mcp`: `NONE`, `TOOL`, or `RESOURCE` to configure how the query is exposed in MCP.
59+
* `uri`: AN RFC 6570 template to configure the REST path and MCP resource path. Any operation arguments that are not defined in the uri template are considered part of the payload for REST (and the method must be POST).
60+
61+
```graphql
62+
query GetPersonByAge($age: Int!) @api(rest: GET, mcp: TOOL, uri: "/queries/personByAge/{age}") {
63+
Person(age: $age, limit: 10, offset: 0) {
64+
name
65+
email
66+
}
67+
}
68+
```
69+
70+
This defines an operation `GetPersonByAge` which is the name of the MCP tool and REST endpoint with the path `/queries/personByAge/{age}` using GET method.

‎documentation/docusaurus.config.ts‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,22 @@ const config: Config = {
7676
],
7777
],
7878

79+
themes: [
80+
[
81+
"@easyops-cn/docusaurus-search-local",
82+
/** @type {import("@easyops-cn/docusaurus-search-local").PluginOptions} */
83+
({
84+
hashed: true,
85+
language: ["en"],
86+
highlightSearchTermsOnTargetPage: true,
87+
explicitSearchResultPath: true,
88+
indexPages: true,
89+
searchResultLimits: 10,
90+
searchResultContextMaxLength: 50
91+
}),
92+
],
93+
],
94+
7995
themeConfig: {
8096
// Replace with your project's social card
8197
image: 'img/datasqrl-social-card.jpg',

0 commit comments

Comments
 (0)