diff --git a/docs/deploy-and-configure/configuration/.pages b/docs/deploy-and-configure/configuration/.pages index 221bca25d..191743b7b 100644 --- a/docs/deploy-and-configure/configuration/.pages +++ b/docs/deploy-and-configure/configuration/.pages @@ -3,6 +3,7 @@ nav: - Docker Orchestration: docker-orchestration - Build: dataintegration - Explore: explore + - Graph Insights: graphinsights - Keycloak: keycloak - Quad-Store: quad-store-configuration - Reverse Proxy: reverse-proxy diff --git a/docs/deploy-and-configure/configuration/graphinsights/explore-add-snapshot.png b/docs/deploy-and-configure/configuration/graphinsights/explore-add-snapshot.png new file mode 100644 index 000000000..f4c148e80 Binary files /dev/null and b/docs/deploy-and-configure/configuration/graphinsights/explore-add-snapshot.png differ diff --git a/docs/deploy-and-configure/configuration/graphinsights/explore-select-graphinsights-tab.png b/docs/deploy-and-configure/configuration/graphinsights/explore-select-graphinsights-tab.png new file mode 100644 index 000000000..80ece0d43 Binary files /dev/null and b/docs/deploy-and-configure/configuration/graphinsights/explore-select-graphinsights-tab.png differ diff --git a/docs/deploy-and-configure/configuration/graphinsights/explore-workspace-enable-graphinsights.png b/docs/deploy-and-configure/configuration/graphinsights/explore-workspace-enable-graphinsights.png new file mode 100644 index 000000000..a6805fe35 Binary files /dev/null and b/docs/deploy-and-configure/configuration/graphinsights/explore-workspace-enable-graphinsights.png differ diff --git a/docs/deploy-and-configure/configuration/graphinsights/index.md b/docs/deploy-and-configure/configuration/graphinsights/index.md new file mode 100644 index 000000000..48ad9c539 --- /dev/null +++ b/docs/deploy-and-configure/configuration/graphinsights/index.md @@ -0,0 +1,350 @@ +--- +tags: + - Configuration + - Graph-Insights + - GraphInsights +--- +# Graph Insights + +## Enable Graph Insights + +Follow the instructions related to your deployment type: a) docker-compose or b) helm / Kubernetes based. + +### (a) Enable in docker-compose deployment + +In our Corporate Memory docker-orchestration all main configurations can be directed by setting environment +variables in `environments/config.env`. +You can find the environments are set as usual in `environments/default.env` and `environments/config.env`. + +Add your license to ```licenses/graphinsights.lic``` then start the extension on a already running Corporate Memory +deployment. + +``` console +mkdir licenses +ln -s your-license-file.lic graphinsights.lic +make enable-extension EXTENSION=graphinsights +``` + +Please also have a look at `extensions/README.graphinsights.md`. + +### (b) Enable in helm deployment + +In helm based deployment you can enable Graph Insights by enable it in your value file. +It creates a new StatefulSet. +Preemptive you have to create a secret containing your license file. + +``` console +kubectl -n cmem create secret generic graphinsights-license --from-file your-graphinsights.lic +``` + +All needed configuration can be done in the Corporate Memory helm chart `value.yaml` file. +This enables the plugin. + +``` yaml +graphinsights: + enabled: true +``` + +Beside enabling the extension you also have to create a route/path in your Ingress or Route. +In the Charts `value.yaml` file is a configuration commented out. +You should enable this in your `value.yaml` file. + +``` yaml + # GraphInsights at /graphinsights path (if enabled). + - path: /graphinsights + pathType: Prefix + serviceName: graphinsights + servicePort: 8080 +``` + +Also make sure you have set the clients and client credentials. +See [Configure OAuth clients (helm)](../../../deploy-and-configure/configuration/graphinsights/index.md#configure-oauth-clients-helm) + +The configuration mentioned below is rendered with those files, but you usually don't have to touch those: + +- `configuration-files/explore-application.yml` for Explore +- `configuration-files/cmem.integration.config.yml` for Graph Insights + +For more details please have a look in the helm value file. +Every configuration is documented there. +Please refer to [Kubernetes deployments](../../../deploy-and-configure/installation/scenario-k8s-deployment/index.md) +for more information. + +## Activate and verify the installation + +Fist you have enable the Graph Insight in your Explore workspace. By default ist is disabled. + +![Explore workspace configuration](explore-workspace-enable-graphinsights.png) + +Then you are able to create a snapshot, send it to Graph Insights and select the Graph Insights tab in explore. + +![Explore add snapshot](explore-add-snapshot.png) + +![Explore select_graphinsights](explore-select-graphinsights-tab.png) + +## Configuration + +This section explains the basic configuration of Graph Insights. Since Graph Insights is an application integrated +into explore, we need to have to configure Explore for the integration and Graph Insights as application itself. + +The following chapters explain the configuration in case you want to change the default behavior. + +While the extension in the docker-orchestration it is enabled through Spring profiles, we just use the benefits of +templating in helm and only use one profile. + +This is why the configuration differs in both deployments, but we tried to use the same variable names and configuration +snippets. + +### Explore configuration + +In our **docker-orchestration** you find the file which is loaded as `graphinsight` profile +at `conf/explore/application-graphinsights.yml`. +Environment variables as usual can be found in `environments/default.env` and `environments/config.env`. +Sizing can be changed in the loaded memory profile, e.g. at `environments/config.mem.16g.env`. +The deployment definition for explore with the extension is defined `extensions/docker-compose.graphinsights.yml` +in the explore service. + +In **helm deployments** you find the needed section inside the `.Values.graphinsights.enabled` in file +at `configuration-files/explore-application.yml`, where most of the configuration is inserted with GO-templates. +Some environment variables are set in the `value.yaml` and rendered in a ConfigMap `templates/explore-configmap.yaml`. + +``` yaml +spring.security.oauth2.client.registration.explore-service: + client-id: cmem-service-account + client-secret: change-me + authorization-grant-type: client_credentials + provider: keycloak +``` + +``` yaml +semspect: + enabled: true + integration: + url: http://graphinsights:8080/graphinsights + externalUrl: ${DEPLOY_BASE_URL}/graphinsights + automaticResyncCronExpression: "0 0/30 8-10 * * *" + localDatasetStatePath: /graphinsights/infinispan + ### Configure either fileShareIntegrationSettings or graph-store-integration-settings + # fileShareIntegrationSettings: + localGraphStoragePathExplore: /graphinsights + # localGraphStoragePathSemspect: /explore-share/ + graph-store-integration-settings: + semspect-dataplatform-url: http://explore/dataplatform +``` + +### Graph Insights configuration + +Graph Insights, like Explore, is a Spring Boot application, so it gets configured with environment variables and a YAML +file. + +In our **docker-orchestration** you find the file at `conf/graphinsights/cmem.integration.config.yml`. +Environment variables as usual can be found in `environments/default.env` and `environments/config.env`. +Sizing can be changed in the loaded memory profile, e.g. at `environments/config.mem.16g.env`. +The deployment definition of the extension is defined `extensions/docker-compose.graphinsights.yml`. + +In **helm deployments** you find the file at `configuration-files/cmem.integration.config.yml`, which is rendered as +ConfigMap and then mounted into the Graph Insight StatefulSet. +Environment variables are set in the `value.yaml` and rendered in a ConfigMap `templates/graphinsights-configmap.yaml`. +Sizing regarding memory, CPU or disk usage are configured in the `value.yaml`. + +``` yaml +--- +# for descriptions of the properties s. the same file in https://gitlab.eccenca.com/devops/eccenca-graphinsights-docker +semspect: + extensions: cmem + extension.cmem: + baseUri: "${DEPLOY_BASE_URL}" + authorization: + cacheDuration: 60 + core: + enableCaptionPropertySelection: false + enableDescriptionPropertySelection: false + +frontend: + appNameOverride: "Graph Insights" + enableLogo: false + enableAboutWindow: false + singleDatabaseMode: true + enableIframeOnlyMode: true + enableCopyDeepLink: false + fontFamily: "Roboto, helvetica, arial, sans-serif" + fontFace: > + @font-face { + font-family: 'Roboto'; + font-style: normal; + font-weight: 400; + src: url('https://eccenca.com/layouts/default/fonts/roboto-v20-latin-regular.woff2') format('woff2'); + } + + trustedOrigins: + - "${DEPLOY_BASE_URL}" + - "${DEPLOY_BASE_URL}/*" + + backlink: + objectURLTemplate: ${DEPLOY_BASE_URL}/{workspaceID}/explore?resource={objectID}&graph={contextGraphIRI} + categoryURLTemplate: ${DEPLOY_BASE_URL}/{workspaceID}/explore?type={categoryID}&graph={contextGraphIRI} + objectBacklinkDestination: "CMEM" + categoryBacklinkDestination: "CMEM" + multiObjectBacklinkDestination: "CMEM" + debug: true + logLevel: DEBUG + +# secure /api/** via resourceserver +spring.security.oauth2: + resourceserver: + jwt: + jwk-set-uri: "${OAUTH_JWK_SET_URL}" + # semspect backend for frontent as oauth2client + # To enable SLO add this as back-channel logout uri to the client semspect uses: + # http://docker.localhost/graphinsights/logout/connect/back-channel/keycloak + client: + registration: + keycloak: + client-id: "${GRAPHINSIGHTS_OAUTH_CLIENT_ID}" + authorization-grant-type: "authorization_code" + client-authentication-method: "client_secret_basic" + # not needed here because we never login to semspect directly + redirectUri: "${DEPLOY_BASE_URL}/graphinsights/login/oauth2/code/{registrationId}" + # openid is mandatory as spring somehow does not add it to the userinfo request + scope: + - openid + - profile + - email + # service account used for datastore access + semspect-service: + client-id: "${GRAPHINSIGHTS_OAUTH_SERVICE_CLIENT_ID}" + client-secret: "${GRAPHINSIGHTS_OAUTH_SERVICE_CLIENT_SECRET}" + authorization-grant-type: client_credentials + provider: keycloak + provider: + keycloak: + issuer-uri: "${OAUTH_ISSUER_URL}" + user-name-attribute: "preferred_username" + +# same (fixed) session timeout as explore backend +server.servlet.session.timeout: "36000s" +server.frame.ancestors: "${DEPLOY_BASE_URL}" + +logging: + level: + root: DEBUG + org.springframework: WARN + org.springframework.security: WARN + + pattern: + console: "%d{yyyy-MM-dd'T'HH:mm:ss.SSSX,UTC} %-5level %logger{15} - %message%n%xException" + +``` + +### Resource consideration + +Please refer to [Graph Insights Sizing](../../../deploy-and-configure/requirements/graph-insights-sizing.md) for more +information. + +### OAuth configuration + +Graph Insights requires authentication similar to Explore and Build (DataIntegration). +There is the need for a client to authenticate in a browser and a second client to allow inter-component communication. + +For convenience, by default we use the same clients as for the rest of the application: + +- Client for browser: cmem +- Client for component communication: cmem-service-account + +In case you want to have separate clients for production deployments, this chapter is for you. + +#### Configure OAuth clients (docker-compose) + +In our **docker-orchestration** to enrol those other clients, please follow these steps: + +1.) take a look at `environments/default.env` and copy these variables to your `environments/config.env` file: + +``` Makefile +# This is the browser client: +GRAPHINSIGHTS_OAUTH_CLIENT_ID=graph-insights +# was this before: +# GRAPHINSIGHTS_OAUTH_CLIENT_ID=${OAUTH_CLIENT_ID} + +# This is the inter component client: +GRAPHINSIGHTS_OAUTH_SERVICE_CLIENT_ID=graph-insights-service-account +# change the credital depending on your client secret, default would be: 7Ctw7eZvTeFYdwwvwopy8OjuMUVPb8A9 +GRAPHINSIGHTS_OAUTH_SERVICE_CLIENT_SECRET=changeme +``` + +2.) Replace this in `docker-compose.yml`: + +``` yaml + - "EXPLORE_CMEM_SERVICE_CLIENT=${CMEM_SERVICE_ACCOUNT_CLIENT_ID}" + - "EXPLORE_CMEM_SERVICE_CLIENT_SECRET=${CMEM_SERVICE_ACCOUNT_CLIENT_SECRET}" + # with: + - "EXPLORE_CMEM_SERVICE_CLIENT=${GRAPHINSIGHTS_OAUTH_SERVICE_CLIENT_ID}" + - "EXPLORE_CMEM_SERVICE_CLIENT_SECRET=${GRAPHINSIGHTS_OAUTH_SERVICE_CLIENT_SECRET}" +``` + +3.) And also replace this in `extensions/docker-compose.graphinsights.yml` + +``` yaml + - "GRAPHINSIGHTS_OAUTH_SERVICE_CLIENT_ID=${CMEM_SERVICE_ACCOUNT_CLIENT_ID}" + - "GRAPHINSIGHTS_OAUTH_SERVICE_CLIENT_SECRET=${CMEM_SERVICE_ACCOUNT_CLIENT_SECRET}" + # with: + - "GRAPHINSIGHTS_OAUTH_SERVICE_CLIENT_ID=${GRAPHINSIGHTS_OAUTH_SERVICE_CLIENT_ID}" + - "GRAPHINSIGHTS_OAUTH_SERVICE_CLIENT_SECRET=${GRAPHINSIGHTS_OAUTH_SERVICE_CLIENT_SECRET}" +``` + +#### Configure OAuth clients (helm) + +In **helm deployments**, once you have the clients available all you have to do is change these lines in your value.yaml +accordingly: + +``` yaml +global: + # GraphInsights service account OAuth credentials (if using GraphInsights). + # Can use the same credentials as CMEM or separate ones. + graphinsightsClientId: graph-insights-service-account + graphinsightsClientSecret: changeme # IMPORTANT: Change this to a secure value! + + # OAuth client ID for GraphInsights frontend authentication. + # Default: "graph-insights" (must exist in your Keycloak realm if GraphInsights is enabled). + oauthClientIdGraphInsights: graph-insights +``` + +#### Creating separate OAuth clients for Graph Insights + +If you start Corporate Memory with the docker-orchestration (or use our keycloak helm chart) for the first time, the +separate clients are already created when starting the Postgresql container. + +However, regenerating a new client secret is advisable: + +- Select the `eccenca Corporate Memory`-realm +- Select `Clients` +- Select the client `graph-insights-service-account`. +- Select the tab `Credentials` +- Press `Regenerate` +- Then copy the new secret and fill in the values from above. + +![Keycloak client secret regenerate](keycloak-client-secret-regenerate.png) + +Also have a look below and check if your +[Backchannel logout URL](../../../deploy-and-configure/configuration/graphinsights/index.md#set-backchannel-logout-url-for-graph-insights) is set. + +In an already running deployment you would have to create those clients on your own. + +We assume the we name the clients as following: + +- Browser client: `graph-insights` +- Inter-component client: `graph-insights-service-account` + +#### Set Backchannel logout URL for Graph Insights + +You would follow the same steps as in +[Keycloak Client configuration](../../../deploy-and-configure/configuration/keycloak/index.md) but name them +differently, e.g. as above. +In addition, on last step is missing: To have the logout working properly you have to add a client Backchannel logout +URL for the client ment for browser authentication (`graph-insights`): + +- Select the `eccenca Corporate Memory`-realm +- Select `Clients` +- Select the client `graph-insights`. +- Scroll down to this section and add this: `https:///graphinsights/logout/connect/back-channel/keycloak` + +![Keycloak backchannel LogExplore select_graphinsights](keycloak-client-backchannel.png) diff --git a/docs/deploy-and-configure/configuration/graphinsights/keycloak-client-backchannel.png b/docs/deploy-and-configure/configuration/graphinsights/keycloak-client-backchannel.png new file mode 100644 index 000000000..7de1cfb2c Binary files /dev/null and b/docs/deploy-and-configure/configuration/graphinsights/keycloak-client-backchannel.png differ diff --git a/docs/deploy-and-configure/configuration/graphinsights/keycloak-client-secret-regenerate.png b/docs/deploy-and-configure/configuration/graphinsights/keycloak-client-secret-regenerate.png new file mode 100644 index 000000000..b451e820d Binary files /dev/null and b/docs/deploy-and-configure/configuration/graphinsights/keycloak-client-secret-regenerate.png differ diff --git a/docs/deploy-and-configure/configuration/keycloak/index.md b/docs/deploy-and-configure/configuration/keycloak/index.md index 59311dfe8..2eee49ecb 100644 --- a/docs/deploy-and-configure/configuration/keycloak/index.md +++ b/docs/deploy-and-configure/configuration/keycloak/index.md @@ -37,11 +37,15 @@ To create a realm, use the drop down menu for choosing a realm on the left side. ## Client configuration Clients are used to link users and groups managed in Keycloak to Corporate Memory. -There are two different clients used by Corporate Memory: +There are two different types of clients used by Corporate Memory: - The first client is used to authenticate a user for using the web interface (usually named `cmem`). -- The other client is used as a technical user with the command line interface (typically named `cmem-service-account`). - Depending on the environment, there might be other use cases, when running background schedules, then a third client, also as technical user, might be useful. +- The second client is used as a technical user with the command line interface (usually named `cmem-service-account`). +- (optional, when using Graph Insights) similar to the `cmem` client you can configure Graph Insights to use a separate client (usually named `graph-insights`) +- (optional, when using Graph Insights) similar to the `cmem-service-account` client you can configure Graph Insights to communicate with a separate client (usually named `graph-insights-service-account`) + +For Graph Insight please refer the [Graph Insights OAUTH documentation](../../../deploy-and-configure/configuration/graphinsights/index.md#oauth-configuration) +Depending on the environment, there might be other use cases, when running background schedules, then a third client, also as technical user, might be useful. ### Access conditions, roles and groups diff --git a/docs/deploy-and-configure/configuration/reverse-proxy/index.md b/docs/deploy-and-configure/configuration/reverse-proxy/index.md index 63c2f601a..527e520c4 100644 --- a/docs/deploy-and-configure/configuration/reverse-proxy/index.md +++ b/docs/deploy-and-configure/configuration/reverse-proxy/index.md @@ -29,28 +29,30 @@ apache configuration template ServerAlias www.corporate-memory.example.com ProxyPreserveHost On - ProxyPass /auth https://keycloak.host/auth retry=0 - ProxyPassReverse /auth https://keycloak.host/auth - - ProxyPass /dataplatform https://dataplatform.host/dataplatform retry=0 - ProxyPassReverse /dataplatform https://dataplatform.host/dataplatform + ProxyPass /auth http://keycloak:8080/auth retry=0 + ProxyPassReverse /auth http://keycloak:8080/auth RewriteEngine on RewriteRule "^/dataintegration$" "/dataintegration/" [R] RewriteCond %{HTTP:Upgrade} =websocket [NC] - RewriteRule "/dataintegration/(.*)" wss://dataintegration.host/dataintegration/$1 [P,L] + RewriteRule "^/dataintegration/(.*)" ws://dataintegration/dataintegration/$1 [P,L] RewriteCond %{HTTP:Upgrade} !=websocket [NC] - RewriteRule "/dataintegration/(.*)" https://dataintegration.host/dataintegration/$1 [P,L] + RewriteRule "^/dataintegration/(.*)" http://dataintegration:80/dataintegration/$1 [P,L] - ProxyPassReverse /dataintegration https://dataintegration.host/dataintegration + ProxyPassReverse /dataintegration http://dataintegration:80/dataintegration - ProxyPass / https://datamanager.host/ retry=0 - ProxyPassReverse / https://datamanager.host/ + ProxyPass /.well-known/acme-challenge ! + ProxyPass / http://explore:80/ retry=0 + ProxyPassReverse / http://explore:80 # https://github.com/gitlabhq/gitlabhq/issues/8924 AllowEncodedSlashes NoDecode + # Allow for compression + # https://httpd.apache.org/docs/2.4/mod/mod_deflate.html + SetOutputFilter DEFLATE + # Network timeout in seconds for proxied requests (default 300) # http://serverfault.com/questions/500467/apache2-proxy-timeout/583266 ProxyTimeout 1200 @@ -72,6 +74,8 @@ Information about the runtime environment which is used to run Explore and Build ## Linked Data delivery mode +TODO: What is this section? Still relevant? Also rename to explore? + The Linked Data delivery mode is able to serve data that uses the same namespace as the configured domain name as resolvable URIs including content negotiation. Therefore you can use the following template (e.g.: ): diff --git a/docs/deploy-and-configure/installation/index.md b/docs/deploy-and-configure/installation/index.md index 7b5652fe6..6cb2e9c72 100644 --- a/docs/deploy-and-configure/installation/index.md +++ b/docs/deploy-and-configure/installation/index.md @@ -5,11 +5,13 @@ icon: material/download-circle-outline This page describes proven deployment scenarios for eccenca Corporate Memory. -All Corporate Memory components are distributed as Docker images and can be obtained from eccenca's Artifactory service. To run them you need a Docker enabled Linux server. In addition to that, eccenca provides distribution archives for all components which contain configuration examples (YAML) as well as JAR/WAR artifacts. +All Corporate Memory components are distributed as Docker images and can be obtained from eccenca's container repository service. +To run them you need a Docker enabled Linux server. + ## Operating Systems (OS) -Corporate Memory is tested on Ubuntu 18.04 (backward compatible with 16.04 and 14.04) and RHEL 7.7. +Corporate Memory is tested on Ubuntu 24.04 (backward compatible with LTS versions older than that) as Debian 11, 12 and 13. Special note on RHEL SELinux Support: there is no limitation for RedHat SELinux. We recommend to keep the SELinux in *enforced* mode. You can keep the default setting of the `/etc/selinux/config` file. @@ -31,56 +33,17 @@ Special note on RHEL SELinux Support: there is no limitation for RedHat SELinux. ## Docker compose based Orchestration deployment -[Docker Compose](https://docs.docker.com/compose/) is a convenient way to provision several Docker containers locally for development setups or on remote servers for single node setups. - -eccenca is heavily using `docker compose` for all kinds of internal and customer deployments. For more details on how to use `docker compose` based orchestration refer to [Scenario: Local Installation](../installation/scenario-local-installation/index.md) and [Scenario: Single Node Cloud Installation](../installation/scenario-single-node-cloud-installation/index.md). - -## Explore - -### Scaling - -Run multiple Explore instances with the same configuration to enable high-availability and/or high-performance setups. - -#### Prerequisites - -For running multiple Explore instances the following prerequisites apply: - -- The same application configuration properties must be used by all scaled instances. -- If access control for any SPARQL endpoint is active, a shared Redis cache used by all Explore instances is required. - -#### Limitations - -When running multiple Explore instances it is not possible to use a shared Virtuoso backend with provisioned access control active. - -### Troubleshooting - -In case Explore failed to start, check the logs for error messages pointing to faulty parameters in the configuration. Since not every faulty behavior is apparent from reading the logs, the following checks can help you to verify the configuration: - -- Check the `http(s):///actuator/health/` endpoint to verify if the SPARQL proxy service endpoints are configured properly. - -Note: Refer to the [Spring documentation](https://docs.spring.io/spring-boot/docs/2.1.8.RELEASE/reference/htmlsingle/#boot-features-profiles) on how to set active profiles. - -### Plugins - -In some cases Explore needs to be extended with plugins. Extensions are necessary when drivers cannot be included due to licensing restrictions or when plugins are delivered separately. - -In this case, you have to update the .war file of Explore by placing the plugin .jar files in the same directory, or by stating the path via the configuration option. - -To include plugins that are located in the same directory as the `eccenca-DataPlatform.war` file, execute the .war file with the option `-u` or `--update-war`: - -```bash linenums="1" -# with plugins located in the same folder as the WAR file -java -jar ${JAVA_TOOL_OPTIONS} eccenca-DataPlatform.war --update-war -``` - -If the plugins to be included are not located in the same folder as the .war file, you can specify a directory containing the plugins as the argument of the `-u` or `--update-war` option. - -```bash linenums="1" -java -jar ${JAVA_TOOL_OPTIONS} eccenca-DataPlatform.war -u /data/plugins -``` +[Docker Compose](https://docs.docker.com/compose/) is a convenient way to provision several Docker containers locally for development +setups or on remote servers for single node setups. -The last command repackages the `eccenca-DataPlatform.war` by including all plugins (.jar) located in the specified directory. +eccenca is heavily using `docker compose` for all kinds of internal and customer deployments. +For more details on how to use `docker compose` based orchestration refer +to [Scenario: Local Installation](../installation/scenario-local-installation/index.md) and [Scenario: Single Node Cloud Installation](../installation/scenario-single-node-cloud-installation/index.md). -Note: Make sure that only the `eccenca-DataPlatform.war` file is in the directory since multiple .war files can cause problems. +## Helm based Kubernetes deployment -Note: During the update procedure, the directory `WEB-INF` is created. Due to security concerns the update mechanism does not delete this directory. You can delete it after the update process is finished. +Most production deployments are Kubernetes based. +We have deployments in AWS (EKS), Azure (AKS), Red Hat Openshift and self-hosted clusters. +We provide Charts for Corporate Memory and Keycloak at our [Helm Repository](https://helm.eccenca.com). +For more details on how to use `helm` based deployments refer +to [Scenario: Kubernetes Deployment](../installation/scenario-k8s-deployment/index.md). diff --git a/docs/deploy-and-configure/installation/scenario-k8s-deployment/images/cmem-helm-architecture.png b/docs/deploy-and-configure/installation/scenario-k8s-deployment/images/cmem-helm-architecture.png new file mode 100644 index 000000000..b57cc3213 Binary files /dev/null and b/docs/deploy-and-configure/installation/scenario-k8s-deployment/images/cmem-helm-architecture.png differ diff --git a/docs/deploy-and-configure/installation/scenario-k8s-deployment/images/cmem-helm-architecture.svg b/docs/deploy-and-configure/installation/scenario-k8s-deployment/images/cmem-helm-architecture.svg index c205b60ad..6b3c8f876 100644 --- a/docs/deploy-and-configure/installation/scenario-k8s-deployment/images/cmem-helm-architecture.svg +++ b/docs/deploy-and-configure/installation/scenario-k8s-deployment/images/cmem-helm-architecture.svg @@ -1,2 +1,4 @@ + + -
Bitnami Keycloak helm chart deployment
Bitnami Keycloak helm chart deployment<div></div>
Keycloak
 
 
[Not supported by viewer]
Ingress / Route
Ingress / Route
HTTP
HTTP
eccenca Corporate Memory helm chart deployment
eccenca Corporate Memory helm chart deployment<div></div>
Explore
Explore
Build
Build
Optional Services
Optional Services
HTTP
HTTP
HTTP
HTTP
LDAP
Directory Information
Service
[Not supported by viewer]
LDAPS
LDAPS

external
OpenID Connect or SAML zone
[Not supported by viewer]
Brokering 
<i>Brokering </i>
HTTPS
[Not supported by viewer]
cmemc


cmemc<br><br><br>
Ingress / Route
Ingress / Route
Ontotext GraphDB helm chart deployment
Ontotext GraphDB helm chart deployment<div></div>
GraphDB
[Not supported by viewer]
Data Sources
Data Sources<br>

Files
(csv, json,
xml, ...)
[Not supported by viewer]
JDBC
JDBC
HDFS
HDFS<br>
Python Plugins
Python Plugins
salesforce
salesforce
GraphQL
GraphQL
kafka
kafka
Browser


[Not supported by viewer]
HTTPS
[Not supported by viewer]
HTTPS
[Not supported by viewer]
HTTPS
[Not supported by viewer]
PostgreSQL
PostgreSQL
JDBC
[Not supported by viewer]
HTTP/HTTPS
<span>HTTP/HTTPS</span>
HTTP
HTTP
Bootstrap Job
cmemc
Bootstrap Job<br>cmemc<br>
HTTPS
[Not supported by viewer]
HTTPS
[Not supported by viewer]
Build Service
Build Service
Explore Service
Explore Service
Port 8080
Port 8080
Port 8080
Port 8080
Port 8080
Port 8080
Port 8080
Port 8080 \ No newline at end of file +
                                                                                                                                                                                      eccenca Corporate Memory helm chart deployment
...
Data Sources
Data Sources
External Services
External Services
Ontotext GraphDB helm chart deployment
Ontotext GraphDB helm chart deployment
Keycloak helm chart deployment
Keycloak helm chart deployment
Ingress / Route
Ingress / Route
HTTP
HTTP
Explore
Explore
Build
Build
HTTP
HTTP
LDAP
Directory Information
Service
LDAP...
LDAPS
LDAPS

external
OpenID Connect or SAML zone
external...
Brokering 
Brokering 
HTTPS
HTTPS
cmemc


cmemc...
Ingress / Route
Ingress / Route
GraphDB
GraphDB

Files
(csv, json,
xml, ...)
Files...
JDBC
JDBC
HDFS
HDFS
Python Plugins
Python Plugins
salesforce
salesforce
GraphQL
GraphQL
kafka
kafka
Browser


Browser...
HTTPS
HTTPS
HTTPS
HTTPS
HTTPS
HTTPS
Bootstrap Job
cmemc
Bootstrap Job...
HTTPS
HTTPS
HTTPS
HTTPS
Build Service
Build Service
Explore Service
Explore Service
Port 8080
Port 8080
Port 8080
Port 8080
Port 8080
Port 8080
Port 8080
Port 8080
Graph Insights
Graph Insights
Graph Insights Service
Graph Insights Service
Port 8080
Port 8080
Port 8080
Port 8080
HTTP 
HTTP 
HTTP 
HTTP 
HTTPS
HTTPS
Cached
volume
Cached...
Cached
volume
Cached...
Cached
volume
Cached...
Volume
Volume
Volume
Volume
Volume
Volume
HTTP 
HTTP 
HTTP
HTTP
HTTP
HTTP
GraphDB Service
GraphDB Service
Port 7200
Port 7200
HTTPS
HTTPS
Backup Job
Backup Job
HTTP
HTTP
LLM Provider
LLM Provider
HTTP
HTTP
S3
S3
S3
S3
Keycloak
 
 
Keycloak...
PostgreSQL
PostgreSQL
JDBC
JDBC
Keycloak Service
Keycloak Service
Port 8080
Port 8080
Port 8080
Port 8080
HTTP
HTTP
HTTP 
HTTP Text is not SVG - cannot display \ No newline at end of file diff --git a/docs/deploy-and-configure/installation/scenario-k8s-deployment/index.md b/docs/deploy-and-configure/installation/scenario-k8s-deployment/index.md index a7e0af5f1..362c6803d 100644 --- a/docs/deploy-and-configure/installation/scenario-k8s-deployment/index.md +++ b/docs/deploy-and-configure/installation/scenario-k8s-deployment/index.md @@ -24,7 +24,7 @@ provisioned cluster. ## Architecture - ![CMEM Helm Chart Architecture](images/cmem-helm-architecture.svg) + ![CMEM Helm Chart Architecture](images/cmem-helm-architecture.png) ## General Notice @@ -360,6 +360,11 @@ explore: ``` +#### (Optional) Graph Insights + +Setting up Graph Insights in Kubernetes deployment done with helm is described in our +[documentation for Graph Insights](../../../deploy-and-configure/configuration/graphinsights/index.md#b-enable-in-helm-deployment). + ### 3. Install the Corporate Memory chart To pull the Corporate Memory images, you need to provide credentials for the eccenca diff --git a/docs/deploy-and-configure/installation/scenario-local-installation/index.md b/docs/deploy-and-configure/installation/scenario-local-installation/index.md index 50e4ccde5..c49ecf0d2 100644 --- a/docs/deploy-and-configure/installation/scenario-local-installation/index.md +++ b/docs/deploy-and-configure/installation/scenario-local-installation/index.md @@ -15,6 +15,8 @@ The code examples in this section assumes that you have POSIX-compliant shell (l - [docker](https://www.docker.com/) and [docker compose](https://docs.docker.com/compose/install/) (v2) installed locally - [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) installed locally - [jq](https://jqlang.github.io/jq/download/) installed locally +- A GraphDB license (free or commercial) +- (optional) A Graph Insights license - make - build tools (apt-get install make) installed locally - At least 4 CPUs and 12GB of RAM (recommended: 16GB) dedicated to docker @@ -227,6 +229,20 @@ Please open http://docker.localhost:80 for validation. Run make logs to see log output ``` +### (Optional) Enable Graph Insights Extension + +If you want to have Graph Insights enabled as well, first you need a license. +Then you can simiply run this: + + +``` shell +mkdir licenses +ln -s your-license-file.lic graphinsights.lic +make enable-extension EXTENSION=graphinsights +``` + +Please also visit our extended [documentation for Graph Insights](../../../deploy-and-configure/configuration/graphinsights/index.md#a-enable-in-docker-compose-deployment) + ## Initial Login / Test Open your browser and navigate to  diff --git a/docs/deploy-and-configure/installation/scenario-single-node-cloud-installation/index.md b/docs/deploy-and-configure/installation/scenario-single-node-cloud-installation/index.md index 25c3abf69..967025fbe 100644 --- a/docs/deploy-and-configure/installation/scenario-single-node-cloud-installation/index.md +++ b/docs/deploy-and-configure/installation/scenario-single-node-cloud-installation/index.md @@ -27,7 +27,7 @@ sudo timedatectl set-timezone Europe/Berlin # install needed packages sudo apt-get install -y apt-transport-https ca-certificates curl gnupg2 \ - software-properties-common gnupg lsb-release gettext zip unzip git \ + software-properties-common gnupg lsb-release gettext zip unzip git \ make vim jq # install docker and docker-compose @@ -141,6 +141,20 @@ systemctl enable cmem-orchestration systemctl start cmem-orchestration ``` +### (Optional) Enable Graph Insights Extension + +If you want to have Graph Insights enabled as well, first you need a license. +Then you can simiply run this: + + +``` shell +mkdir licenses +ln -s your-license-file.lic graphinsights.lic +make enable-extension EXTENSION=graphinsights +``` + +Please also visit our extended [documentation for Graph Insights](../../../deploy-and-configure/configuration/graphinsights/index.md#a-enable-in-docker-compose-deployment) + ## Validation and Finalisation Open your browser and navigate to the host you have created in DNS server, e.g. `https://corporate-memory.eccenca.dev` diff --git a/docs/deploy-and-configure/requirements/index.md b/docs/deploy-and-configure/requirements/index.md index 882163fbc..f9257285d 100644 --- a/docs/deploy-and-configure/requirements/index.md +++ b/docs/deploy-and-configure/requirements/index.md @@ -38,11 +38,19 @@ The following numbers are based on existing customer deployments running Knowled | ------------------------------: | :-------------- | :--------------- | | eccenca Explore | \>= 2 cores[^u] | \>= 4 GB RAM | | eccenca Build (DataIntegration) | \>= 4 cores[^w] | \>= 8 GB RAM[^w] | +| eccenca Graph Insights | \>= 4 cores | \>= 20 GB RAM | | Triple / Quad Store | \>= 8 cores[^u] | \>= 16 GB RAM[^t] | | Keycloak incl. PostgreSQL[^c] | 2 cores | \>= 2 GB RAM | For GraphDB always also have a look at [GraphDB recommendations](https://graphdb.ontotext.com/documentation/11.0/requirements.html#hardware-sizing). +!!! note + + For Graph Insights, the calculation of the triples is not based on the amount of triples in the graph store + but on the sum of triples in all Graph Insights Snapshots. Have a look at the cmemc command group + [graph insights](../../automate/cmemc-command-line-interface/command-reference/graph/insights/index.md) to manage existing snapshots. + + ## Clients ### Browser / Web Client diff --git a/docs/deploy-and-configure/system-architecture/Corporate-Memory-System-Architecture_25.3.drawio.png b/docs/deploy-and-configure/system-architecture/Corporate-Memory-System-Architecture_25.3.drawio.png new file mode 100644 index 000000000..af0008fcf Binary files /dev/null and b/docs/deploy-and-configure/system-architecture/Corporate-Memory-System-Architecture_25.3.drawio.png differ diff --git a/docs/deploy-and-configure/system-architecture/index.md b/docs/deploy-and-configure/system-architecture/index.md index fb2ab3228..ecbb656e7 100644 --- a/docs/deploy-and-configure/system-architecture/index.md +++ b/docs/deploy-and-configure/system-architecture/index.md @@ -11,13 +11,17 @@ hide: This page describes the overall system architecture of eccenca Corporate Memory and its components. -![cmem-System-Architecture](Corporate-Memory-System-Architecture_v2.jpg) +![cmem-System-Architecture](Corporate-Memory-System-Architecture_25.3.drawio.png) eccenca Corporate Memory consists of three core components: -- [eccenca Build](../configuration/dataintegration/index.md) -- [eccenca Explore](../configuration/explore/index.md), and -- [cmemc (Corporate Memory Control)](../../automate/cmemc-command-line-interface/index.md) +- [eccenca Build](../configuration/dataintegration/index.md) +- [eccenca Explore](../configuration/explore/index.md), +- [cmemc (Corporate Memory Control)](../../automate/cmemc-command-line-interface/index.md), + +and one optional component: + +- [eccenca Graph Insights](../../explore-and-author/graph-exploration/graph-insights/index.md). _Build_ is the Corporate Memory component which enables integration of datasets into a single consistent knowledge graph. Datasets in their original format are mapped to RDF schemata and then linked to and persisted into a knowledge graph. @@ -34,6 +38,8 @@ Additionally, _Explore_ manages authorization of the users according to the acce The knowledge graph is stored in a quad store connected to _Explore_. This can either be a physical store like [GraphDB](http://graphdb.ontotext.com/), [Virtuoso](https://virtuoso.openlinksw.com/) or a remotely accessible SPARQL 1.1 compliant HTTP endpoint. +_Graph Insights_ … + Keycloak provides authentication. Keycloak can act as an authentication broker for already existing, external OpenId Connect or SAML infrastructures. In addition to that, Keycloak supports a wide variety of internal user management configuration scenarios and the option to connect to an external LDAP server for user and group synchronization. diff --git a/docs/release-notes/corporate-memory-25-2/index.md b/docs/release-notes/corporate-memory-25-2/index.md index 2e2fe6a86..a7b377a74 100644 --- a/docs/release-notes/corporate-memory-25-2/index.md +++ b/docs/release-notes/corporate-memory-25-2/index.md @@ -33,7 +33,7 @@ The highlights of this release are: This release delivers the following component versions: - eccenca DataIntegration v25.2.2 -- eccenca Explore v25.2.5 +- eccenca Explore v25.2.6 - eccenca Corporate Memory Control (cmemc) v25.4.0 We tested this release with the following dependency components: @@ -93,9 +93,14 @@ We are excited to announce the release of DataIntegration v25.2.2, which introdu - XML datasets now return empty values for empty tags when string values are expected - Project variable updates now properly use the triggering user's credentials -## eccenca Explore v25.2.5 +## eccenca Explore v25.2.6 -We are pleased to announce Explore v25.2.4, which brings significant enhancements to SHACL shape management, improved graph handling, and a modernized knowledge graph exploration experience. +We are pleased to announce Explore v25.2.6, which brings significant enhancements to SHACL shape management, improved graph handling, and a modernized knowledge graph exploration experience. + +**v25.2.6 of Explore ships the following fix:** + +- Removal of sub dependency redis +- Update Spring Boot to 3.5.7 **v25.2.5 of Explore ships the following fix:** diff --git a/docs/release-notes/corporate-memory-25-3/index.md b/docs/release-notes/corporate-memory-25-3/index.md index c2e43c7ec..0686c4b93 100644 --- a/docs/release-notes/corporate-memory-25-3/index.md +++ b/docs/release-notes/corporate-memory-25-3/index.md @@ -266,3 +266,5 @@ We are excited to announce cmemc v25.5.0, which introduces new features, improve ### eccenca Explore - With the deprecation of the assistance feature, `assist.*` properties can be removed from your `application.yml`, they are not used any more and have been replaced by `spring.ai` configuration. +- If you are using (the deprecated) `shui:valueQuery` with any `sh:PropertyShape`, make sure you change the query signature to return the variable bindings `?resource` and `?graph`. + The UI will display an error message if this is overlooked.