Skip to content

Commit 1ab285a

Browse files
mkmuralimkmurali
committed
First version for release
0 parents  commit 1ab285a

File tree

9 files changed

+2352
-0
lines changed

9 files changed

+2352
-0
lines changed

‎.dockerignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
node_modules
2+
npm-debug.log

‎.gitignore‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# Logs
2+
logs
3+
*.log
4+
npm-debug.log*
5+
yarn-debug.log*
6+
yarn-error.log*
7+
lerna-debug.log*
8+
9+
# Diagnostic reports (https://nodejs.org/api/report.html)
10+
report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
11+
12+
# Runtime data
13+
pids
14+
*.pid
15+
*.seed
16+
*.pid.lock
17+
18+
# Directory for instrumented libs generated by jscoverage/JSCover
19+
lib-cov
20+
21+
# Coverage directory used by tools like istanbul
22+
coverage
23+
*.lcov
24+
25+
# nyc test coverage
26+
.nyc_output
27+
28+
# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
29+
.grunt
30+
31+
# Bower dependency directory (https://bower.io/)
32+
bower_components
33+
34+
# node-waf configuration
35+
.lock-wscript
36+
37+
# Compiled binary addons (https://nodejs.org/api/addons.html)
38+
build/Release
39+
40+
# Dependency directories
41+
node_modules/
42+
jspm_packages/
43+
44+
# TypeScript v1 declaration files
45+
typings/
46+
47+
# TypeScript cache
48+
*.tsbuildinfo
49+
50+
# Optional npm cache directory
51+
.npm
52+
53+
# Optional eslint cache
54+
.eslintcache
55+
56+
# Microbundle cache
57+
.rpt2_cache/
58+
.rts2_cache_cjs/
59+
.rts2_cache_es/
60+
.rts2_cache_umd/
61+
62+
# Optional REPL history
63+
.node_repl_history
64+
65+
# Output of 'npm pack'
66+
*.tgz
67+
68+
# Yarn Integrity file
69+
.yarn-integrity
70+
71+
# dotenv environment variables file
72+
.env
73+
.env.test
74+
75+
# parcel-bundler cache (https://parceljs.org/)
76+
.cache
77+
78+
# Next.js build output
79+
.next
80+
81+
# Nuxt.js build / generate output
82+
.nuxt
83+
dist
84+
85+
# Gatsby files
86+
.cache/
87+
# Comment in the public line in if your project uses Gatsby and *not* Next.js
88+
# https://nextjs.org/blog/next-9-1#public-directory-support
89+
# public
90+
91+
# vuepress build output
92+
.vuepress/dist
93+
94+
# Serverless directories
95+
.serverless/
96+
97+
# FuseBox cache
98+
.fusebox/
99+
100+
# DynamoDB Local files
101+
.dynamodb/
102+
103+
# TernJS port file
104+
.tern-port

‎Dockerfile‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
FROM node:16
2+
3+
# Create app directory
4+
WORKDIR /usr/src/app
5+
6+
# Install app dependencies
7+
COPY package*.json ./
8+
RUN npm ci --only=production
9+
RUN npm i -g pm2
10+
11+
# Bundle app source
12+
COPY . .
13+
14+
EXPOSE 3000
15+
CMD [ "pm2-runtime", "start", "index.js" ]

‎README.md‎

Lines changed: 257 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,257 @@
1+
# Proxy Authorization Service for Prefect UI and Prefect CLI
2+
3+
Prefect currently do not offer Authentication/Authorization options for API when run on-premises. This functionality is only available with Cloud subscription. So this Proxy service is a workaround for Prefect UI and CLI to enable authentication/authorization for API. All the requests to Prefect API will be proxied through this service.
4+
5+
## Setup
6+
7+
This initial implementation uses API Key and MySQL database backend to authenticate and authorize requests.
8+
9+
### Setup Database
10+
11+
This service looks for prefect_api_keys table in MYSQL Database. Please refer to data/db_init.sql for the initial schema and sample data.
12+
13+
```sql
14+
-- create table prefect_api_keys
15+
CREATE TABLE `prefect_api_keys` (
16+
`user_id` varchar(128) NOT NULL,
17+
`scopes` varchar(2000) NOT NULL,
18+
`api_key` varchar(128) NOT NULL,
19+
`key_issu_dt` datetime NOT NULL,
20+
`key_expr_dt` datetime NOT NULL,
21+
`creatd_dt` datetime NOT NULL,
22+
`last_updatd_dt` datetime NOT NULL,
23+
`creatd_by` varchar(11) NOT NULL,
24+
`last_updatd_by` varchar(11) NOT NULL,
25+
PRIMARY KEY (`user_id`)
26+
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
27+
28+
-- Sample Test Data
29+
insert into prefect_api_keys (user_id, scopes, api_key, key_issu_dt, key_expr_dt, creatd_dt, last_updatd_dt, creatd_by, last_updatd_by)
30+
values('USERID', 'mutation/*, query/*', 'TestKey001', '2021-12-27 00:00:00', '2022-12-31 00:00:00', '2021-12-30 00:00:00', '2021-12-30 00:00:00', 'ADMINUSERID', 'ADMINUSERID')
31+
```
32+
33+
Pass database credentials as environment variables to the service. Please see section below for specific environment variables.
34+
35+
### Environment Variables
36+
37+
The following Environment Variables are available to configure the Proxy service:
38+
39+
| Option | Default | Description |
40+
| ----------------------------- | ------------------- | ----------------------------------------- |
41+
| `ALLOW_PUBLIC_ACCESS` | `false` | Allow public access to the service |
42+
| `TENANT_ID` | `Default` | Tenant ID |
43+
| `TENANT_NAME` | `Default` | Tenant name |
44+
| `TENANT_SLUG` | `default` | Tenant slug |
45+
| `LOG_LEVEL` | `warn` | Default Log level (`error`,`warn`,`info`) |
46+
| `API_SERVICE_URL` | `<Prefect API URL>` | Prefect API Service URL |
47+
| `HOST` | `0.0.0.0` | Host |
48+
| `PORT` | `3000` | Port |
49+
| `LOG_LEVEL_OVERRIDE_DURATION` | `300` | Log level override duration (seconds) |
50+
| `ENV` | `NA` | Environment Name |
51+
| `DB_HOST` | `NULL` | MySQL Database Server Host |
52+
| `DB_USER` | `NULL` | MySQL Database Server User |
53+
| `DB_PASSWORD` | `NULL` | MySQL Database Server Password |
54+
| `DB_DATABASE` | `NULL` | MySQL Database Name |
55+
56+
### Docker Build
57+
58+
```bash
59+
# Build docker image
60+
docker build . -t iddoc/prefect-auth-proxy
61+
62+
# Run docker image
63+
# Prepare environment variables in .env file
64+
docker run -d -p 3000:3000 --env-file ./.env --name auth-proxy iddoc/prefect-auth-proxy
65+
```
66+
67+
### Running the service
68+
69+
This is a typical node.js application. So you can simply clone this repo and run.
70+
71+
```bash
72+
# Clone the repo
73+
git clone https://github.com/softrams/prefect-auth-proxy.git prefect-auth-proxy
74+
75+
# install dependencies
76+
cd prefect-auth-proxy
77+
npm install
78+
79+
# Run the service
80+
node index.js
81+
```
82+
83+
#### Production Environments
84+
85+
Depending on the environment and how you are deploying Prefect, there are multiple ways to run the service.
86+
87+
- This is simply a Nodejs application. So you can also run it as a service using `pm2` or `nodemon` for production deployments.
88+
- If you are running the service on a Kubernetes cluster, use the docker container and deploy as the ingress proxy service.
89+
90+
## Sample Initial Access Control Policy
91+
92+
Here is a sample initial access control policy for the Proxy service. You can use this policy to configure the access control policy for your Prefect UI and CLI.
93+
94+
### All Team Members
95+
96+
All team members will receive Read Only access to all aspects
97+
98+
```bash
99+
query/*
100+
```
101+
102+
### Model Operations, Development and QA Teams
103+
104+
Only select mutations are allowed for Model Operations Team and Development Team Members
105+
106+
```bash
107+
# Run a workflow
108+
mutation/create_flow_run
109+
110+
# Cancel a flow run
111+
mutation/cancel_flow_run
112+
113+
# Restart a run
114+
mutation/set_flow_run_states
115+
116+
```
117+
118+
### DevOps Team
119+
120+
For DevOps Team, all mutations are allowed.
121+
122+
```bash
123+
mutation/*
124+
```
125+
126+
## How to use
127+
128+
### Create API Keys and distribute to users/services
129+
130+
API key will be used to authenticate and authorize the requests. Create an API key by inserting an entry to the prefect_api_keys table. Please refer to data/db_init.sql for the initial schema and sample data.
131+
132+
### Using API Key via Browser
133+
134+
Run the following command from Browser Developer Tools console:
135+
136+
```bash
137+
let auth = (document.getElementsByTagName('a')[0].__vue__.$store.state).user
138+
auth.user.id='<api key>'
139+
```
140+
141+
Once this is done, you can access the Prefect UI just like you would access the Prefect UI regularly.
142+
143+
> **Note:** Please note that, if you are NOT authorized to do certain things from UI, you will NOT see any error messages, but the request will silently fail.
144+
145+
### Using API Key with CLI
146+
147+
> **Note:** Using API Key with CLI is restricted to DevOps team and automated CI/CD pipelines currently, as it requires wildcard permissions to support all operations. All other teams can still use CLI to query the data, but will not be permitted to create/run workflows or administer Prefect environments. Use Prefect UI to review logs and restart workflows.
148+
149+
```bash
150+
# Point to appropriate Prefect cloud server for the target environment
151+
# These are the default values for Prefect in DEV environment
152+
export PREFECT__CLOUD__ENDPOINT="<UI URL>"
153+
export PREFECT__CLOUD__GRAPHQL="<API URL>/graphql"
154+
155+
# Set server to cloud
156+
prefect backend cloud
157+
158+
# Authenticate with API Key
159+
prefect auth login -k <api key>
160+
161+
# Alternately you can also set API Key to use by
162+
# configuring this info in your .prefect.yml as follows
163+
[API_HOST]
164+
api_key = "<api key>"
165+
tenant_id = "<tenant id>"
166+
167+
# Now you are all set. Run any Prefect command from CLI
168+
169+
# List projects on the cloud server
170+
prefect get projects
171+
172+
```
173+
174+
## Enable Verbose Logging
175+
176+
### Set Log Level permanently
177+
178+
To enable verbose logging, set the `LOG_LEVEL` to `info`.
179+
This will log all requests and responses along with headers and other information.
180+
181+
### Set Log Level temporarily
182+
183+
It is possible to dynamically set log level to `info` for 5 minutes by invoking the `/logs` endpoint. This 5 minute duration can be overridden by setting the `LOG_LEVEL_OVERRIDE_DURATION` to a different value.
184+
185+
```bash
186+
# sets log level to info for 5 minutes
187+
curl --location --request POST 'http://localhost:3000/logs/info'
188+
```
189+
190+
## Cache Management
191+
192+
For better performance, API Key lookups are cached for 30 minutes. Cache expiration is checked every 5 minutes
193+
and any expired keys are removed from the cache.
194+
195+
For testing purposes, you may clear the
196+
entire cache or delete a specific api key from cache.
197+
198+
> **Note:** When multiple instances of this proxy are run (behing a load balancer), the cache is only cleared
199+
> from the instance that process the request. There is no guarantee that the cache will be cleared from all instances.
200+
> You may try multiple requests to make sure the cache is cleared in all instances.
201+
202+
### Clear Cache
203+
204+
Clear cache removes all cached API Key lookups.
205+
206+
```bash
207+
curl --location --request POST 'http://localhost:3000/cache/reset'
208+
```
209+
210+
### Delete Specific API Key from Cache
211+
212+
Use this operation for testing any changes made to API Key by deleting
213+
just a specific key from the cache.
214+
215+
```bash
216+
curl --location --request DELETE 'http://localhost:3000/cache/<api key>'
217+
```
218+
219+
### Audit Trail
220+
221+
Audit trail logs are enabled specifically for all allowed and blocked mutations. Search for `PREFECT_AUDIT_TRAIL` in the logs to see the audit trail logs.
222+
Here is an example of the audit trail log:
223+
224+
```
225+
PREFECT_AUDIT_TRAIL: BLOCKED Mutation create_flow_run for <user_id> {
226+
operationName: 'CreateFlowRun',
227+
variables: { id: '3e216621-8e22-4f7f-af69-284c644cba05' },
228+
query: 'mutation CreateFlowRun($context: JSON, $id: UUID!, $flowRunName: String, $parameters: JSON, $scheduledStartTime: DateTime, $runConfig: JSON, $labels: [String!]) {\n' +
229+
' create_flow_run(\n' +
230+
' input: {context: $context, flow_id: $id, flow_run_name: $flowRunName, parameters: $parameters, scheduled_start_time: $scheduledStartTime, run_config: $runConfig, labels: $labels}\n' +
231+
' ) {\n' +
232+
' id\n' +
233+
' __typename\n' +
234+
' }\n' +
235+
'}\n'
236+
}
237+
```
238+
239+
## Roadmap
240+
241+
This is the first version of the Prefect API Proxy. This version is ready for use with the required setup as explained above.
242+
You may fork this repository and make your own changes to the code to expand to fit your usecases.
243+
If its generic enough, please consider contributing to the project.
244+
245+
Here is a summary of the planned features:
246+
247+
| Feature | Description |
248+
| --------------------------------------- | --------------------------------------------------------------- |
249+
| Options for other Databases | Add additional options for Databases and make this configurable |
250+
| Automated Tests | Add automated tests for the Proxy Service |
251+
| Automated CI/CD pipelines | Add automated CI/CD pipelines and publish Docker Image |
252+
| Documentation for different deployments | Add documentation for different deployments |
253+
254+
## Credits
255+
256+
Our teams found Prefect to be an amazing framework, developed by [Prefect Technologies Inc.](https://prefect.io) and is licensed under the Apache 2.0 License.
257+
Thanks a lot for the great work!

0 commit comments

Comments
 (0)