Skip to content

Commit 9d23d6b

Browse files
danyi1212danyi1212
authored
Added Rust PDP (#255)
* Added Rust PDP * Update build script to pull latest changes and add logs Ensure the script pulls the latest changes from the `permit-opa` repository before creating the custom OPA tarball. Added a success message to confirm tarball creation. * Fixed linting errors * Renamed pdp crate to pdp-server * Removed duplicate authn middleware for tests * Refactor the PDP engine structure * Added watchdog crate * Added restart channel * Added graceful shutdown and term timeout * Add ServiceWatchdog with health checking capabilities * Changed app state to use ServiceWatchdog instead of PDP Engine * Remove example code * Changed Dockerfile to use new Rust PDP * Improve path handling in horizon fallback and add test for unmatched 404 response
1 parent 5057985 commit 9d23d6b

40 files changed

+5249
-57
lines changed

‎.dockerignore‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,16 @@
22
helm/
33
.venv/
44
.github/
5+
target/
6+
**/*.rs.bk
7+
**/*.pdb
8+
Cargo.lock
9+
.cargo/
10+
.dockerignore
11+
.gitignore
12+
README.md
13+
docs/
14+
**/tests/
15+
**/__pycache__/
16+
**/*.pyc
17+
**/.DS_Store

‎.editorconfig‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
1-
root=true
1+
root = true
22

33
[*]
4-
charset=utf-8
5-
end_of_line=lf
6-
insert_final_newline=false
7-
indent_style=space
8-
indent_size=2
9-
trim_trailing_whitespace=true
4+
charset = utf-8
5+
end_of_line = lf
6+
insert_final_newline = false
7+
indent_style = space
8+
indent_size = 4
9+
trim_trailing_whitespace = true
1010

11-
[*.py]
12-
indent_size=4
11+
[*.{yml,yaml}]
12+
indent_size = 2

‎.github/workflows/tests.yml‎

Lines changed: 47 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ name: PDP CI Tests
33
on:
44
pull_request:
55
push:
6-
branches: [master, main, v*]
6+
branches: [ master, main, v* ]
77
workflow_call:
88
secrets:
99
PDP_TESTER_API_KEY:
@@ -15,26 +15,25 @@ jobs:
1515
pre-commit:
1616
runs-on: ubuntu-latest
1717
steps:
18-
- uses: actions/checkout@v3
19-
- uses: actions/setup-python@v3
20-
- uses: pre-commit/action@v2.0.3
21-
18+
- uses: actions/checkout@v4
19+
- uses: actions/setup-python@v5
20+
- uses: pre-commit/action@v3.0.1
2221
pytests:
2322
runs-on: ubuntu-latest
2423
steps:
25-
- name: Python setup
26-
uses: actions/setup-python@v5
27-
with:
28-
python-version: '3.11.8'
24+
- name: Python setup
25+
uses: actions/setup-python@v5
26+
with:
27+
python-version: '3.11.8'
2928

30-
- name: Checkout code
31-
uses: actions/checkout@v4
29+
- name: Checkout code
30+
uses: actions/checkout@v4
3231

33-
- name: Run Pytests
34-
run: |
35-
python -m pip install --upgrade pip
36-
pip install ".[dev]"
37-
pytest -s --cache-clear horizon/tests/
32+
- name: Run Pytests
33+
run: |
34+
python -m pip install --upgrade pip
35+
pip install ".[dev]"
36+
pytest -s --cache-clear horizon/tests/
3837
3938
pdp-tester:
4039
runs-on: ubuntu-latest
@@ -77,15 +76,15 @@ jobs:
7776

7877
# Checkout the pdp-tester repository
7978
- name: Checkout pdp-tester repository
80-
uses: actions/checkout@v3
79+
uses: actions/checkout@v4
8180
with:
8281
repository: "permitio/pdp-tester"
8382
token: ${{ secrets.CLONE_REPO_TOKEN }}
8483
path: './pdp-tester'
8584

8685
# Setup Python environment
8786
- name: Setup Python
88-
uses: actions/setup-python@v3
87+
uses: actions/setup-python@v5
8988
with:
9089
python-version: "3.12"
9190

@@ -120,3 +119,33 @@ jobs:
120119
echo "========================================"
121120
echo ""
122121
done
122+
123+
rust-ci:
124+
name: Rust CI
125+
runs-on: ubuntu-latest
126+
steps:
127+
- name: Checkout code
128+
uses: actions/checkout@v4
129+
130+
- name: Install Rust toolchain
131+
uses: dtolnay/rust-toolchain@v1
132+
with:
133+
toolchain: stable
134+
components: rustfmt, clippy
135+
136+
- name: Rust cache
137+
uses: swatinem/rust-cache@v2
138+
with:
139+
shared-key: "rust-cache"
140+
141+
- name: Check formatting
142+
run: cargo fmt --all -- --check
143+
144+
- name: Run cargo check
145+
run: cargo check --all --verbose
146+
147+
- name: Run clippy
148+
run: cargo clippy --all -- -D warnings
149+
150+
- name: Run tests
151+
run: cargo test --all --verbose

‎.gitignore‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,3 +138,26 @@ dmypy.json
138138

139139
# MacOS specific
140140
.DS_Store
141+
142+
### Rust template
143+
# Generated by Cargo
144+
# will have compiled files and executables
145+
debug/
146+
target/
147+
148+
# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
149+
# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
150+
Cargo.lock
151+
152+
# These are backup files generated by rustfmt
153+
**/*.rs.bk
154+
155+
# MSVC Windows builds of rustc generate these, which store debugging information
156+
*.pdb
157+
158+
# RustRover
159+
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
160+
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
161+
# and can be added to the global gitignore or merged into this file. For a more nuclear
162+
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
163+
#.idea/

‎.pre-commit-config.yaml‎

Lines changed: 23 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -17,23 +17,34 @@ repos:
1717
args: [ --fix=lf ]
1818

1919
- repo: https://github.com/astral-sh/ruff-pre-commit
20-
rev: v0.11.1
20+
rev: v0.11.6
2121
hooks:
2222
- id: ruff
23-
args: [--fix]
23+
args: [ --fix ]
2424
files: \.py$
2525
types: [ file ]
2626
- id: ruff-format
2727
files: \.py$
2828
types: [ file ]
2929

30-
# - repo: https://github.com/pre-commit/mirrors-mypy
31-
# rev: v1.13.0
32-
# hooks:
33-
# - id: mypy
34-
# pass_filenames: false
35-
# additional_dependencies:
36-
# - pydantic
37-
# - types-requests
38-
# files: \.py$
39-
# types: [ file ]
30+
# - repo: https://github.com/pre-commit/mirrors-mypy
31+
# rev: v1.13.0
32+
# hooks:
33+
# - id: mypy
34+
# pass_filenames: false
35+
# additional_dependencies:
36+
# - pydantic
37+
# - types-requests
38+
# files: \.py$
39+
# types: [ file ]
40+
41+
- repo: https://github.com/doublify/pre-commit-rust
42+
rev: v1.0
43+
hooks:
44+
- id: cargo-check
45+
args: [ --all-targets ]
46+
files: \.rs$
47+
- id: fmt
48+
files: \.(rs|toml)$
49+
args: [ '--verbose' ]
50+
pass_filenames: false

‎Cargo.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[workspace]
2+
resolver = "2"
3+
members = ["pdp-server", "watchdog"]

‎Dockerfile‎

Lines changed: 61 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,46 @@
11
ARG OPA_BUILD=permit
2+
ARG TARGETPLATFORM
3+
ARG TARGETARCH
4+
5+
# RUST BUILD STAGE -----------------------------------
6+
# Build the Rust PDP binary for all targets
7+
# ----------------------------------------------------
8+
# BIG thanks to
9+
# - https://medium.com/@vladkens/fast-multi-arch-docker-build-for-rust-projects-a7db42f3adde
10+
# - https://stackoverflow.com/questions/70561544/rust-openssl-could-not-find-directory-of-openssl-installation
11+
# couldn't get this to work without the help of those two sources
12+
# (1) this stage will be run always on current arch
13+
# zigbuild & Cargo targets added
14+
FROM --platform=$BUILDPLATFORM rust:1.85-alpine AS rust_chef
15+
WORKDIR /app
16+
ENV PKGCONFIG_SYSROOTDIR=/
17+
RUN apk add --no-cache musl-dev openssl-dev zig pkgconf perl make
18+
19+
RUN cargo install --locked cargo-zigbuild cargo-chef
20+
RUN rustup target add x86_64-unknown-linux-musl aarch64-unknown-linux-musl
21+
22+
# (2) nothing changed
23+
FROM rust_chef AS rust_planner
24+
COPY . .
25+
RUN cargo chef prepare --recipe-path recipe.json
26+
27+
# (3) building project deps: need to specify all targets; zigbuild used
28+
FROM rust_chef AS rust_builder
29+
COPY --from=rust_planner /app/recipe.json recipe.json
30+
ENV OPENSSL_DIR=/usr
31+
RUN cargo chef cook --recipe-path recipe.json --release --zigbuild \
32+
--target x86_64-unknown-linux-musl --target aarch64-unknown-linux-musl
33+
34+
# (4) actuall project build for all targets
35+
# binary renamed to easier copy in runtime stage
36+
COPY . .
37+
RUN cargo zigbuild -r --target x86_64-unknown-linux-musl --target aarch64-unknown-linux-musl && \
38+
mkdir -p /app/linux/arm64/ && \
39+
mkdir -p /app/linux/amd64/ && \
40+
cp target/aarch64-unknown-linux-musl/release/pdp-server /app/linux/arm64/pdp && \
41+
cp target/x86_64-unknown-linux-musl/release/pdp-server /app/linux/amd64/pdp
42+
43+
244
# OPA BUILD STAGE -----------------------------------
345
# Build OPA from source or download precompiled binary
446
# ---------------------------------------------------
@@ -37,21 +79,20 @@ RUN mkdir -p /app/backup && chmod -R 777 /app/backup
3779

3880
# Install necessary libraries in a single RUN command
3981
RUN apk update && \
40-
apk add --no-cache bash build-base libffi-dev libressl-dev musl-dev zlib-dev gcompat
82+
apk add --no-cache bash build-base libffi-dev libressl-dev musl-dev zlib-dev gcompat wget
4183

4284
# Copy OPA binary from the build stage
4385
COPY --from=opa_build --chmod=755 /opa /app/bin/opa
4486

87+
# Copy the Rust PDP binary from the builder stage
88+
ARG TARGETPLATFORM
89+
COPY --from=rust_builder --chmod=755 /app/${TARGETPLATFORM}/pdp /app/pdp
90+
4591
# Environment variables for OPA
4692
ENV OPAL_INLINE_OPA_EXEC_PATH="/app/bin/opa"
4793

48-
# Copy required scripts
49-
COPY scripts /app/scripts
50-
5194
# Set permissions and ownership for the application
5295
RUN mkdir -p /config && chown -R permit:permit /config
53-
RUN chmod +x /app/scripts/wait-for-it.sh && \
54-
chmod +x /app/scripts/start.sh
5596

5697
# Ensure the `permit` user has the correct permissions for home directory and binaries
5798
RUN chown -R permit:permit /home/permit /app /usr/local/bin
@@ -61,7 +102,6 @@ USER permit
61102

62103
# Copy Kong routes and Gunicorn config
63104
COPY kong_routes.json /config/kong_routes.json
64-
COPY ./scripts/gunicorn_conf.py ./gunicorn_conf.py
65105

66106
USER root
67107

@@ -77,17 +117,14 @@ USER permit
77117
# Copy the application code
78118
COPY ./horizon /app/horizon
79119

120+
USER permit
121+
80122
# Version file for the application
81123
COPY ./permit_pdp_version /app/permit_pdp_version
82124

83125
# Set the PATH to ensure the local binary paths are used
84126
ENV PATH="/app/bin:/home/permit/.local/bin:$PATH"
85127

86-
# Uvicorn configuration
87-
ENV UVICORN_NUM_WORKERS=1
88-
ENV UVICORN_ASGI_APP="horizon.main:app"
89-
ENV UVICORN_PORT=7000
90-
91128
# opal configuration --------------------------------
92129
ENV OPAL_SERVER_URL="https://opal.permit.io"
93130
ENV OPAL_LOG_DIAGNOSE="false"
@@ -108,11 +145,9 @@ ENV PDP_VERSION_FILE_PATH="/app/permit_pdp_version"
108145
# and it is here as a safety measure on purpose.
109146
ENV OPAL_AUTH_PUBLIC_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDe2iQ+/E01P2W5/EZwD5NpRiSQ8/r/k18pFnym+vWCSNMWpd9UVpgOUWfA9CAX4oEo5G6RfVVId/epPH/qVSL87uh5PakkLZ3E+PWVnYtbzuFPs/lHZ9HhSqNtOQ3WcPDTcY/ST2jyib2z0sURYDMInSc1jnYKqPQ6YuREdoaNdPHwaTFN1tEKhQ1GyyhL5EDK97qU1ejvcYjpGm+EeE2sjauHYn2iVXa2UA9fC+FAKUwKqNcwRTf3VBLQTE6EHGWbxVzXv1Feo8lPZgL7Yu/UPgp7ivCZhZCROGDdagAfK9sveYjkKiWCLNUSpado/E5Vb+/1EVdAYj6fCzk45AdQzA9vwZefP0sVg7EuZ8VQvlz7cU9m+XYIeWqduN4Qodu87rtBYtSEAsru/8YDCXBDWlLJfuZb0p/klbte3TayKnQNSWD+tNYSJHrtA/3ZewP+tGDmtgLeB38NLy1xEsgd31v6ISOSCTHNS8ku9yWQXttv0/xRnuITr8a3TCLuqtUrNOhCx+nKLmYF2cyjYeQjOWWpn/Z6VkZvOa35jhG1ETI8IwE+t5zXqrf2s505mh18LwA1DhC8L/wHk8ZG7bnUe56QwxEo32myUBN8nHdu7XmPCVP8MWQNLh406QRAysishWhXVs/+0PbgfBJ/FxKP8BXW9zqzeIG+7b/yk8tRHQ=="
110147

111-
112-
113-
114148
# We ignore this callback because we are sunsetting this feature in favor of the new inline OPA data updater
115149
ENV PDP_IGNORE_DEFAULT_DATA_UPDATE_CALLBACKS_URLS='["http://localhost:8181/v1/data/permit/rebac/cache_rebuild"]'
150+
116151
# if we are using the custom OPA binary, we need to load the permit plugin,
117152
# if we don't then we MUST not add a non existing plugin
118153
FROM main AS main-vanilla
@@ -125,8 +160,16 @@ ENV PDP_OPA_PLUGINS='{"permit_graph":{}}'
125160

126161
FROM main-${OPA_BUILD} AS application
127162

128-
# 7000 sidecar port
163+
# Environment variables with defaults
164+
ENV PDP_HORIZON_HOST=0.0.0.0
165+
ENV PDP_HORIZON_PORT=7001
166+
ENV PDP_PORT=7000
167+
ENV PDP_PYTHON_PATH=python3
168+
169+
# 7000 pdp port
170+
# 7001 horizon port
129171
# 8181 opa port
130-
EXPOSE 7000 8181
172+
EXPOSE 7000 7001 8181
173+
131174
# Run the application using the startup script
132-
CMD ["/app/scripts/start.sh"]
175+
CMD ["/app/pdp"]

‎build_opal_bundle.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ if [ "$PDP_VANILLA" != "true" ]; then
2424
cd "../permit-opa"
2525
find * \( -name '*go*' -o -name 'LICENSE.md' \) -print0 | xargs -0 tar -czf "$build_root"/custom/custom_opa.tar.gz --exclude '.*'
2626
cd "$build_root"
27+
echo "Custom OPA tarball created successfully."
2728
else
2829
echo "Skipping custom OPA tarball creation for pdp-vanilla environment."
2930
fi

‎pdp-server/.dockerignore‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Git
2+
.git/
3+
.gitignore
4+
5+
# Rust build artifacts
6+
target/
7+
**/*.rs.bk
8+
9+
# RocksDB data
10+
data/
11+
12+
# Editor directories
13+
.idea/
14+
.vscode/
15+
.cursor/
16+
*.iml
17+
18+
# Docker
19+
Dockerfile
20+
.dockerignore
21+
22+
# Temporary files
23+
**/*.tmp
24+
**/*.log
25+
26+
# Documentation
27+
*.md
28+
!README.md

0 commit comments

Comments
 (0)