Skip to content

Commit 6f9d6b0

Browse files
maxim-lixakovmaxim-lixakov
authored
[DOP-21268] - integration with SSO (Keycloak) (#123)
* [DOP-21268] - refactor auth configuration settings (add providers) * [DOP-21268] - implement KeycloakAuthProvider * [DOP-21268] - add docs * [DOP-21268] - update User model * [DOP-21268] - split Settings to SyncmasterSettings, WorkerSettings(SyncmasterSettings), BackendSettings(SyncmasterSettings) * [DOP-21268] - update RTD config * [DOP-21268] - make unified logging settings for worker and backend * [DOP-21268] - minor fixes * [DOP-21268] - add KeycloakAuthProvider interaction schema * [DOP-21268] - add SessionSettings
1 parent af13e94 commit 6f9d6b0

File tree

109 files changed

+1563
-438
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

109 files changed

+1563
-438
lines changed

‎.env.docker‎

Lines changed: 23 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,17 +4,34 @@ ENV=LOCAL
44
# Debug
55
SYNCMASTER__SERVER__DEBUG=true
66

7-
# Logging Backend
8-
SYNCMASTER__SERVER__LOGGING__SETUP=True
9-
SYNCMASTER__SERVER__LOGGING__PRESET=colored
7+
# Logging
8+
SYNCMASTER__LOGGING__SETUP=True
9+
SYNCMASTER__LOGGING__PRESET=colored
10+
SYNCMASTER__LOG_URL_TEMPLATE=https://grafana.example.com?correlation_id={{ correlation_id }}&run_id={{ run.id }}
1011

11-
# Logging Worker
12-
SYNCMASTER__WORKER__LOGGING__SETUP=True
13-
SYNCMASTER__WORKER__LOGGING__PRESET=json
12+
# Session
13+
SYNCMASTER__SERVER__SESSION__SECRET_KEY=session_secret_key
14+
15+
# Encrypt / Decrypt credentials data
16+
SYNCMASTER__CRYPTO_KEY=UBgPTioFrtH2unlC4XFDiGf5sYfzbdSf_VgiUSaQc94=
1417

1518
# Postgres
1619
SYNCMASTER__DATABASE__URL=postgresql+asyncpg://syncmaster:changeme@db:5432/syncmaster
1720

21+
# TODO: add to KeycloakAuthProvider documentation about creating new realms, add users, etc.
22+
# KEYCLOAK Auth
23+
SYNCMASTER__AUTH__SERVER_URL=http://keycloak:8080/
24+
SYNCMASTER__AUTH__REALM_NAME=manually_created
25+
SYNCMASTER__AUTH__CLIENT_ID=manually_created
26+
SYNCMASTER__AUTH__CLIENT_SECRET=generated_by_keycloak
27+
SYNCMASTER__AUTH__REDIRECT_URI=http://localhost:8000/v1/auth/callback
28+
SYNCMASTER__AUTH__SCOPE=email
29+
SYNCMASTER__AUTH__PROVIDER=syncmaster.backend.providers.auth.keycloak_provider.KeycloakAuthProvider
30+
31+
# Dummy Auth
32+
SYNCMASTER__AUTH__PROVIDER=syncmaster.backend.providers.auth.dummy_provider.DummyAuthProvider
33+
SYNCMASTER__AUTH__ACCESS_TOKEN__SECRET_KEY=secret
34+
1835
# RabbitMQ
1936
SYNCMASTER__BROKER__URL=amqp://guest:guest@rabbitmq:5672/
2037

‎.env.local‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5,16 +5,23 @@ export ENV=LOCAL
55
export SYNCMASTER__SERVER__DEBUG=true
66

77
# Logging
8-
export SYNCMASTER__SERVER__LOGGING__SETUP=True
9-
export SYNCMASTER__SERVER__LOGGING__PRESET=colored
8+
export SYNCMASTER__LOGGING__SETUP=True
9+
export SYNCMASTER__LOGGING__PRESET=colored
10+
export SYNCMASTER__LOG_URL_TEMPLATE="https://grafana.example.com?correlation_id={{ correlation_id }}&run_id={{ run.id }}"
1011

11-
# Logging Worker
12-
export SYNCMASTER__WORKER__LOGGING__SETUP=True
13-
export SYNCMASTER__WORKER__LOGGING__PRESET=json
12+
# Session
13+
export SYNCMASTER__SERVER__SESSION__SECRET_KEY=session_secret_key
14+
15+
# Encrypt / Decrypt credentials data
16+
export SYNCMASTER__CRYPTO_KEY=UBgPTioFrtH2unlC4XFDiGf5sYfzbdSf_VgiUSaQc94=
1417

1518
# Postgres
1619
export SYNCMASTER__DATABASE__URL=postgresql+asyncpg://syncmaster:changeme@localhost:5432/syncmaster
1720

21+
# Auth
22+
export SYNCMASTER__AUTH__PROVIDER=syncmaster.backend.providers.auth.dummy_provider.DummyAuthProvider
23+
export SYNCMASTER__AUTH__ACCESS_TOKEN__SECRET_KEY=secret
24+
1825
# RabbitMQ
1926
export SYNCMASTER__BROKER__URL=amqp://guest:guest@localhost:5672/
2027

‎.readthedocs.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ build:
1717
- VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH python -m poetry install --no-root --all-extras --with docs --without dev,test
1818
- VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH python -m poetry show -v
1919
- python -m pip list -v
20-
- SYNCMASTER__DATABASE__URL=postgresql+psycopg://fake:[email protected]:5432/fake SYNCMASTER__BROKER__URL=amqp://fake:faket@fake:5672/ python -m syncmaster.backend.export_openapi_schema docs/_static/openapi.json
20+
- SYNCMASTER__DATABASE__URL=postgresql+psycopg://fake:[email protected]:5432/fake SYNCMASTER__SERVER__SESSION__SECRET_KEY=session_secret_key SYNCMASTER__BROKER__URL=amqp://fake:faket@fake:5672/ SYNCMASTER__CRYPTO_KEY=crypto_key SYNCMASTER__AUTH__ACCESS_TOKEN__SECRET_KEY=fakepython python -m syncmaster.backend.export_openapi_schema docs/_static/openapi.json
2121

2222
sphinx:
2323
configuration: docs/conf.py

‎docker-compose.test.yml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ services:
4040
- 8000:8000
4141
volumes:
4242
- ./syncmaster:/app/syncmaster
43+
- ./docs/_static:/app/docs/_static
4344
- ./cached_jars:/root/.ivy2
4445
- ./reports:/app/reports
4546
- ./tests:/app/tests
@@ -156,6 +157,19 @@ services:
156157
retries: 3
157158
profiles: [hive, hdfs, all]
158159

160+
keycloak:
161+
image: quay.io/keycloak/keycloak:latest
162+
command: start-dev
163+
restart: unless-stopped
164+
environment:
165+
KEYCLOAK_ADMIN: admin
166+
KEYCLOAK_ADMIN_PASSWORD: admin
167+
ports:
168+
- 8080:8080
169+
volumes:
170+
- keycloak_data:/opt/keycloak/data
171+
profiles: [keycloak, all]
172+
159173
test-hive:
160174
image: mtsrus/hadoop:hadoop2.7.3-hive2.3.9
161175
restart: unless-stopped
@@ -184,3 +198,4 @@ services:
184198
volumes:
185199
postgres_test_data:
186200
rabbitmq_test_data:
201+
keycloak_data:

‎docs/backend/auth/custom.rst‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
.. _backend-auth-custom:
2+
3+
Custom Auth provider
4+
====================
5+
6+
You can implement custom auth provider by inheriting from class below and implementing necessary methods.
7+
8+
.. autoclass:: syncmaster.backend.providers.auth.AuthProvider
9+
:members:
10+
:member-order: bysource

‎docs/backend/auth/dummy.rst‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
.. _backend-auth-dummy:
2+
3+
Dummy Auth provider
4+
===================
5+
6+
Description
7+
-----------
8+
9+
This auth provider allows to sign-in with any username and password, and and then issues an access token.
10+
11+
After successful auth, username is saved to backend database. It is then used for creating audit records for any object change, see ``changed_by`` field.
12+
13+
Interaction schema
14+
------------------
15+
16+
.. dropdown:: Interaction schema
17+
18+
.. plantuml::
19+
20+
@startuml
21+
title DummyAuthProvider
22+
participant "Client"
23+
participant "Backend"
24+
25+
== POST v1/auth/token ==
26+
27+
activate "Client"
28+
alt Successful case
29+
"Client" -> "Backend" ++ : login + password
30+
"Backend" --> "Backend" : Password is completely ignored
31+
"Backend" --> "Backend" : Check user in internal backend database
32+
"Backend" -> "Backend" : Create user if not exist
33+
"Backend" -[#green]> "Client" -- : Generate and return access_token
34+
35+
else User is blocked
36+
"Client" -> "Backend" ++ : login + password
37+
"Backend" --> "Backend" : Password is completely ignored
38+
"Backend" --> "Backend" : Check user in internal backend database
39+
"Backend" x-[#red]> "Client" -- : 401 Unauthorized
40+
41+
else User is deleted
42+
"Client" -> "Backend" ++ : login + password
43+
"Backend" --> "Backend" : Password is completely ignored
44+
"Backend" --> "Backend" : Check user in internal backend database
45+
"Backend" x-[#red]> "Client" -- : 404 Not found
46+
end
47+
48+
== GET v1/namespaces ==
49+
50+
alt Successful case
51+
"Client" -> "Backend" ++ : access_token
52+
"Backend" --> "Backend" : Validate token
53+
"Backend" --> "Backend" : Check user in internal backend database
54+
"Backend" -> "Backend" : Get data
55+
"Backend" -[#green]> "Client" -- : Return data
56+
57+
else Token is expired
58+
"Client" -> "Backend" ++ : access_token
59+
"Backend" --> "Backend" : Validate token
60+
"Backend" x-[#red]> "Client" -- : 401 Unauthorized
61+
62+
else User is blocked
63+
"Client" -> "Backend" ++ : access_token
64+
"Backend" --> "Backend" : Validate token
65+
"Backend" --> "Backend" : Check user in internal backend database
66+
"Backend" x-[#red]> "Client" -- : 401 Unauthorized
67+
68+
else User is deleted
69+
"Client" -> "Backend" ++ : access_token
70+
"Backend" --> "Backend" : Validate token
71+
"Backend" --> "Backend" : Check user in internal backend database
72+
"Backend" x-[#red]> "Client" -- : 404 Not found
73+
end
74+
75+
deactivate "Client"
76+
@enduml
77+
78+
Configuration
79+
-------------
80+
81+
.. autopydantic_model:: syncmaster.backend.settings.auth.dummy.DummyAuthProviderSettings
82+
.. autopydantic_model:: syncmaster.backend.settings.auth.jwt.JWTSettings

‎docs/backend/auth/index.rst‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
.. _backend-auth-providers:
2+
3+
Auth Providers
4+
==============
5+
6+
Syncmaster supports different auth provider implementations. You can change implementation via settings:
7+
8+
.. autopydantic_model:: keycloak.backend.settings.auth.AuthSettings
9+
10+
.. toctree::
11+
:maxdepth: 2
12+
:caption: Auth providers
13+
14+
dummy
15+
keycloak
16+
17+
.. toctree::
18+
:maxdepth: 2
19+
:caption: For developers
20+
21+
custom

‎docs/backend/auth/keycloak.rst‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
.. _backend-auth-ldap:
2+
3+
KeyCloak Auth provider
4+
==================
5+
6+
Description
7+
-----------
8+
9+
TODO:
10+
11+
Strategies
12+
----------
13+
14+
TODO:
15+
16+
Interaction schema
17+
------------------
18+
19+
.. dropdown:: Interaction schema
20+
21+
.. plantuml::
22+
23+
@startuml
24+
title Keycloak Authorization Flow
25+
participant "Client (User from Browser)" as Client
26+
participant "Syncmaster"
27+
participant "Keycloak"
28+
29+
== Client Authentication at Keycloak ==
30+
Client -> Syncmaster : Request endpoint that requires authentication (/v1/users)
31+
32+
Syncmaster x-[#red]> Client : Redirect to Keycloak login URL (if no access token)
33+
34+
Client -> Keycloak : Callback redirect to Keycloak login page
35+
36+
alt Successful login
37+
Client --> Keycloak : Log in with login and password
38+
else Login failed
39+
Keycloak x-[#red]> Client -- : Display error (401 Unauthorized)
40+
end
41+
42+
Keycloak -> Client : Redirect to Syncmaster to callback endpoint with code
43+
Client -> Syncmaster : Callback request to /v1/auth/callback with code
44+
Syncmaster-> Keycloak : Exchange code for access token
45+
Keycloak --> Syncmaster : Return JWT token
46+
Syncmaster --> Client : Set JWT token in user's browser in cookies and redirect /v1/users
47+
48+
Client --> Syncmaster : Redirect to /v1/users
49+
Syncmaster -> Syncmaster : Get user info from JWT token and check user in internal backend database
50+
Syncmaster -> Syncmaster : Create user in internal backend database if not exist
51+
Syncmaster -[#green]> Client -- : Return requested data
52+
53+
54+
55+
== GET v1/users ==
56+
alt Successful case
57+
Client -> Syncmaster : Request data with JWT token
58+
Syncmaster --> Syncmaster : Get user info from JWT token and check user in internal backend database
59+
Syncmaster -> Syncmaster : Create user in internal backend database if not exist
60+
Syncmaster -[#green]> Client -- : Return requested data
61+
62+
else Access token is expired
63+
Syncmaster -> Keycloak : Get new JWT token via refresh token
64+
Keycloak --> Syncmaster : Return new JWT token
65+
Syncmaster --> Syncmaster : Get user info from JWT token and check user in internal backend database
66+
Syncmaster -> Syncmaster : Create user in internal backend database if not exist
67+
Syncmaster -[#green]> Client -- : Return requested data and set new JWT token in user's browser in cookies
68+
69+
else Refresh token is expired
70+
Syncmaster x-[#red]> Client -- : Redirect to Keycloak login URL
71+
end
72+
73+
deactivate Client
74+
@enduml
75+
76+
Basic configuration
77+
-------------------
78+
79+
.. autopydantic_model:: syncmaster.settings.auth.keycloak.KeycloakProviderSettings
80+
.. autopydantic_model:: syncmaster.settings.auth.jwt.JWTSettings
81+

‎docs/backend/configuration/cors.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,4 +5,4 @@ CORS settings
55

66
These settings used to control `CORS <https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS>`_ options.
77

8-
.. autopydantic_model:: syncmaster.settings.server.cors.CORSSettings
8+
.. autopydantic_model:: syncmaster.backend.settings.server.cors.CORSSettings

‎docs/backend/configuration/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ Configuration
1111
database
1212
broker
1313
logging
14+
session
1415
cors
1516
debug
1617
monitoring

0 commit comments

Comments
 (0)