INTERSECT Registry Service

IMPORTANT: This repository will remain unstable until at least the release of INTERSECT-SDK v0.9.0 .

The INTERSECT Registry Service is a core service associated with the INTERSECT Ecosystem. The goal of this microservice is to centralize credential management to, and configuration of, the INTERSECT control plane and data plane for domain science microservices. It is NOT concerned with scientific campaigns themselves.

How users interact with this service

This service will generally only need to be interacted with by users who are developing/deploying domain science microservices. If you are attempting to write an INTERSECT Campaign, utilize iHub instead.

Workflow

The basic workflow is this:

1. Users authenticate themselves against our auth server via the UI.
2. Once authenticated, users are able to obtain the system name, the client API key, and reserve a Service namespace. If writing a Client, all they will need is the Client API key and the system name of any associated Services.
3. After reserving a service namespace, the service and the API key will appear in the user's services table. Both the service and the API key will need to be included in the SDK's Intersect configuration.
4. The SDK developer includes the registry service URL, the service namespace, and the API key in their configuration. The SDK calls the appropriate API endpoint, and receives the appropriate credentials needed to integrate into INTERSECT. If the SDK loses connection, it will call the registry service API again until it regains connection; the idea is that the SDK will only give up its current workflow if the registry service goes down or if the credentials provided don't differ or remain invalid.

Local end-to-end development workflows

Since the goal is to move a considerable amount of configuration and credential logic from the SDK to a more centralized service, this will necessarily complicate localized end-to-end tests for INTERSECT developers, as this microservice will now be required as part of a minimal setup. To facilitate local workflows, this microservice can be configured in a "development" mode - the authentication server and the database will not need to be set up, the UI workflow can be skipped, and the SDK will be able to automatically retrieve a "root" user from the registry service.

TODO: add a complete docker-compose configuration for setting up backing services.

UI

The UI is meant to be extremely minimalistic, and should be functional without Javascript or CSS (but should use these technologies as part of Progressive Enhancement).

SDK design patterns

This approach is designed around the client-side service discovery pattern. SDK microservices, after initial configuration, should be able to exchange messages over a message broker with minimal network hops. The registry service is currently focused around authentication mechanisms, and does not directly keep track of whether or not microservices are actually connected or if they are working (however, you can quickly de-authenticate an SDK microservice through this core service).

The Registry Service will NOT attempt to take on a schema validation role, or will at most perform an extremely minimal schema validation role. This is better suited for iHub, as creating Campaigns relies heavily on the user-generated schemas, and schema version updates will be critical for iHub to track. Note, however, that in the SDK code, Services will need to be able to produce a valid schema before attempting to authenticate against the Registry Service.

How credentials work

• To reserve a Service namespace and generate an API key associated with it, or to get the Client API key, you have to authenticate against a central service. This will generally be an institutional auth server, not one maintained or controlled by the team deploying INTERSECT core services.
• Clients and Services use control/data plane users with limited permissions. Anyone can subscribe to events or publish requests/responses/commands, but only the appropriate applications should be able to publish events and subscribe to requests/responses/commands. If utilizing a data plane outside of a message broker, a data plane should have a similar restriction space (i.e. presigned URLs). Only INTERSECT core services have permissions beyond this.
• There is a universal Client user (keep in mind that Clients are mostly testing tools, and are meant to be temporal) and a user for each Service.
• The credentials provided to the Client/Service via the Registry Service API are generated by the registry service (users cannot create these credentials themselves) expected to be regularly rotated, SDK users can call the Registry Service API again with the same API key to get new credentials.
• The Client API key may also be rotated periodically.
• Users cannot determine their own Service API keys, but may choose when these keys are rotated. These will not be routinely rotated.

Development setup

Installing

Make sure you have UV installed (Instructions)

• uv sync
• uv run pre-commit install
• cp .env.example .env - will need to do this each time .env.example updates from remote

Running

• docker compose up -d - spins up the database and brokers
• uv run python -m intersect_registry_service*

Application runs on port 8000 unless you set SERVER_PORT

Conventions distinct to this project (READ THIS if developing)

• Instead of using url_for in Jinja/HTML templates, use url_abspath_for. The API is exactly the same, this allows us to use absolute paths.
• Instead of using request.url_for(name, **path_params), use intersect_registry_service.app.utils.urls.url_abspath_for(request, name, **path_params) when redirecting with RedirectResponse to another Registry Service URL.
  • To provide a callback URL to external services i.e. Keycloak, use intersect_registry_service.app.utils.urls.absolute_url_for(request, name) . If setting settings.PRODUCTION to True, this relies on you running behind a proxy.

Important configuration variables

All environment variables can be checked in intersect_registry_service/app/core/environment.py, any class value of Settings in SCREAMING_SNAKE_CASE is an environment variable.

• AUTH_IMPLEMENTATION can change between rudimentary (uses hardcoded users and roles, obviously not suited for production but makes local development a lot easier) or keycloak (authenticate against a Keycloak database).
• DEVELOPMENT_API_KEY can be set in testing environments to quickly allow for people to test their end-to-end Service/Client logic without having to fiddle with the registry service UI/database/auth-server. Note that this value should NOT be set in ANY instance outside of testing things locally.
• SYSTEM_NAME is the common namespace that all clients/services/core-services share with the message broker. This is important from a "system-of-system" perspective; the eventual intention is that if you wish to make an INTERSECT system-of-system union, you will have to ensure that the SYSTEM_NAME variable of each registry service is unique.
• Client credentials are currently set via environment variables (and are not stored in the database) but this is subject to change.
• BROKER_PROTOCOL and BROKER_APPLICATION - management of broker resources sometimes depends on both the broker protocol and the exact broker implementation.

Setting up Keycloak Auth

In order to set up a Keycloak instance for this project, Dockerfile.keycloak can be used to build one (or used as part of the compose project). After starting the Keycloak container, navigate to the port (http://localhost:8080 by default) and log in to the admin portal by using the default admin credentials: admin for the username, and admin for the password. After logging in, in the dropdown at the top left of the screen (that should have "master" selected by default), select "Create Realm". Use the resource file, keycloak/realm-export.json from this repository in the form and make sure that the name of the realm is DevRegistryKeycloak (if choosing a different name, make sure that the appropriate config options from .env.example are updated).

If all you want to do is test out Keycloak, this should be sufficent. A default user with the credentials username and password is already setup; this does not utilize any third-party providers. However, if you want to test out a third-party provider, read on.

After importing the realm, you can edit the registry client by clicking the the Clients page in the left hand navigation panel, and selecting registry-service-dev. Here you can change the redirect URL, regenerate the client secret if necessary, or manage the tokens that Keycloak provides. In order to create a new user not tied to a third party identity provider, click the Users page in the left hand navigation panel. Click the "Add user" button, and fill out the Username, email, first and last name fields. Although not all of these fields are marked as necessary, you will actually get errors after logging in if you do not fill all of them out. After creating the user, click on that user in the Users page, go to the credentials tab and set a password for that user. I would recommend making it permanent via the option in the create password dialogue if using this user for testing purposes.

In order to manage third party providers (Google should be provided by default), click the Identity Providers page in the left hand navigation panel under the Configure heading. Here, you can add third party identity providers. You'll need a client ID and secret from said provider to put as part of the configuration in Keycloak. The redirect URL will actually be generated here in Keycloak, and you will use the value you see in Keycloak in your identity provider's client configuration as the redirect URL there. So basically: Identity Provider provides Client ID and Secret -> You set Client ID and Secret in Keycloak. Keycloak provides Redirect URL -> You set Redirect URL in Identity Provider.

Things to remember when configuring a Keycloak Realm

• Be sure to set the realm client's "redirectUris" to the following (replace ${BASE_URL} with the actual URL, it's not a template):
  • ${BASE_URL} (root URL, normally requires auth on the Registry Service side)
  • ${BASE_URL}/login (login page, redirect after logging out)
  • ${BASE_URL}/login/callback (redirect after logging in, )

If the login endpoints are ever changed, you will need to adjust them accordingly in the realm config file.

Tech stack

• HTMX for the frontend, this website should be able to function without Javascript but should be able to optionally use Javascript to make using the website smoother
• FastAPI for the HTTP server and OpenAPI generation
• SQLModel and PostgreSQL for the data layer
• Alembic for handling database migrations
• Keycloak as an authentication server
• any technologies used for the control plane and the data plane on the INTERSECT-SDK .

Running DB migrations

(note: if not running commands in this directory, you should also include -c <path/to/alembic.ini> in your command)

It's useful to read the official Alembic documentation, but this section will attempt to give a condensed summary.

Whenever you change a file in intersect_registry_service/models which is meant to be representative of a table class (i.e. extends SQLModel and has the metaclass attribute table=True), you must do the following:

1. import the file in migrations/env.py (or just export it in models/__init__.py and let the * import take care of it)
2. run uv run alembic revision --autogenerate -m "<YOUR_MESSAGE_HERE>"
3. Make sure you read https://alembic.sqlalchemy.org/en/latest/autogenerate.html#what-does-autogenerate-detect-and-what-does-it-not-detect and fix things it can't detect.

From there, you can do one of two things:

• restart the server, because it will attempt to run the migrations on startup
• run uv run alembic upgrade head on the command line

Downgrades should always be manual. In production, you should generally run commands like uv run alembic downgrade -1 (the -1 signifies that you should downgrade one migration version), OR you can manually scan the migration version files for the revision field and run uv alembic downgrade <revision> . NOTE: if you downgrade a revision, it appears that you can NEVER reapply it to the same database...

To generate SQL output (instead of directly applying changes to a database), add the --sql flag to any alembic command. It is possible that for production deployments we may not be able to directly modify the DB, in which case submitting the SQL output to a DBA to execute is the correct approach. With this usecase, ALEMBIC_RUN_MIGRATIONS should be set to False in the configuration.

Production setup

All environment variables can be checked in intersect_registry_service/app/core/environment.py, any class value of Settings in SCREAMING_SNAKE_CASE is an environment variable.

Some notes:

• make sure you set AUTH_IMPLEMENTATION to keycloak in any serious deployment setup
• do NOT set DEVELOPMENT_API_KEY, leave it blank.

If you are running this behind a reverse proxy, make sure you do the following:

1. Set BASE_URL to your actual proxy path
2. Make sure the proxy forwards the original host in X-Forwarded-Host and X-Forwarded-Proto. An example nginx configuration is shown below:

server { 
 listen 80;
 listen  [::]:80;
 server_name _;

 location = /registry {
   return 302 /registry/;
 }

 location /registry/ {
   proxy_pass http://localhost:8000/;
   proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header X-Forwarded-Proto $scheme;
   proxy_set_header X-Forwarded-Host  $host;
 }
}

These are needed to provide Keycloak with the URLs which redirect back to the Registry Service.