Skip to content

Improving the Dockerfile #160

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tmikula-dev merged 2 commits into from
May 26, 2026
Merged

Improving the Dockerfile #160

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
459e1e0
Improving the Dockerfile for project
tmikula-dev May 25, 2026
9fd16fc
Reactions on the review comments.
tmikula-dev May 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions .github/workflows/check_python.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,9 @@ jobs:
if [[ "${{ github.event_name }}" == "pull_request" ]]; then
CHANGED_FILES=$(gh api \
"repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/files" \
--jq '.[].filename | select(endswith(".py") or . == "requirements.txt")')
--jq '.[].filename | select(endswith(".py") or (startswith("requirements") and endswith(".txt")))')
Comment thread
tmikula-dev marked this conversation as resolved.
else
CHANGED_FILES=$(git diff --name-only "${{ github.sha }}~1" "${{ github.sha }}" -- '*.py' 'requirements.txt')
CHANGED_FILES=$(git diff --name-only "${{ github.sha }}~1" "${{ github.sha }}" -- '*.py' 'requirements*.txt')
fi

if [[ -n "$CHANGED_FILES" ]]; then
Expand Down Expand Up @@ -69,7 +69,7 @@ jobs:
cache: 'pip'

- name: Install dependencies
run: pip install -r requirements.txt
run: pip install -r requirements-dev.txt

- name: Analyze code with Pylint
id: analyze-code
Expand Down Expand Up @@ -105,7 +105,7 @@ jobs:
cache: 'pip'

- name: Install dependencies
Comment thread
lsulak marked this conversation as resolved.
Outdated
run: pip install -r requirements.txt
run: pip install -r requirements-dev.txt

- name: Check code format with Black
id: check-format
Expand All @@ -130,7 +130,7 @@ jobs:
cache: 'pip'

- name: Install dependencies
run: pip install -r requirements.txt
run: pip install -r requirements-dev.txt

- name: Check types with Mypy
id: check-types
Expand All @@ -155,7 +155,7 @@ jobs:
cache: 'pip'

- name: Install Python dependencies
run: pip install -r requirements.txt
run: pip install -r requirements-dev.txt

- name: Check code coverage with Pytest
run: pytest --cov=. -v tests/unit/ --cov-fail-under=80
Expand All @@ -180,7 +180,7 @@ jobs:
cache: 'pip'

- name: Install dependencies
run: pip install -r requirements.txt
run: pip install -r requirements-dev.txt

- name: Run integration tests
run: pytest tests/integration/ -v --tb=short --log-cli-level=INFO
Expand Down
5 changes: 2 additions & 3 deletions DEVELOPER.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,13 @@ EventGate ships two Lambda functions:

## Prerequisites
- Python 3.13 (current required runtime)
- PostgreSQL client dev package
- For local development, you may temporarily switch to the commented `psycopg2-binary==2.9.10` in `requirements.txt`.
- Docker (for local integration tests using testcontainers)

## Set Up Python Environment
```shell
python3 -m venv .venv
source .venv/bin/activate
pip3 install -r requirements.txt
pip3 install -r requirements-dev.txt
```

## Run Pylint Tool Locally
Expand Down
23 changes: 9 additions & 14 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ ARG SASL_SSL_ARTIFACTS=./sasl_ssl_artifacts
# Trusted certs
COPY $TRUSTED_SSL_CERTS /opt/certs/

# Production dependencies
COPY requirements.txt ${LAMBDA_TASK_ROOT}/requirements.txt

RUN \
echo "######################################################" && \
echo "### Import trusted certs before doing anything else ###" && \
Expand All @@ -36,33 +39,25 @@ RUN \
echo "### -> Basics ###" && \
echo "### -> GCC (some makefiles require cmd which)###" && \
echo "### -> dependencies for kerberos SASL_SSL ###" && \
echo "### -> PostgreSQL dev headers (psycopg2) ###" && \
echo "##############################################" && \
dnf install -y \
wget tar xz bzip2-devel zlib-devel \
which make gcc gcc-c++ \
libffi-devel cyrus-sasl-devel cyrus-sasl-gssapi openssl-devel krb5-workstation && \
libffi-devel cyrus-sasl-devel cyrus-sasl-gssapi openssl-devel krb5-workstation postgresql-devel && \
echo "#################" && \
echo "### librdkafka ###" && \
echo "#################" && \
mkdir -p /tmp/env-install-workdir/librdkafka && \
cd /tmp/env-install-workdir/librdkafka && \
wget --ca-certificate=/etc/pki/tls/certs/ca-bundle.crt https://github.com/edenhill/librdkafka/archive/v2.4.0.tar.gz && \
tar -xf v2.4.0.tar.gz && \
cd /tmp/env-install-workdir/librdkafka/librdkafka-2.4.0 && \
wget --ca-certificate=/etc/pki/tls/certs/ca-bundle.crt https://github.com/confluentinc/librdkafka/archive/v2.14.0.tar.gz && \
tar -xf v2.14.0.tar.gz && \
cd /tmp/env-install-workdir/librdkafka/librdkafka-2.14.0 && \
./configure && make && make install && \
echo "###################" && \
echo "### pip installs ###" && \
echo "###################" && \
pip install requests==2.31.0 urllib3==1.26.18 setuptools cryptography jsonschema PyJWT psycopg2-binary && \
echo "######################" && \
echo "### confluent-kafka ###" && \
echo "######################" && \
mkdir -p /tmp/env-install-workdir/confluent-kafka && \
cd /tmp/env-install-workdir/confluent-kafka && \
wget --ca-certificate=/etc/pki/tls/certs/ca-bundle.crt https://github.com/confluentinc/confluent-kafka-python/archive/v2.4.0.tar.gz && \
tar -xf v2.4.0.tar.gz && \
cd /tmp/env-install-workdir/confluent-kafka/confluent-kafka-python-2.4.0 && \
CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/opt" python setup.py install && \
pip install -r ${LAMBDA_TASK_ROOT}/requirements.txt --no-binary confluent-kafka && \

@lsulak lsulak May 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

confluent-kafka - what is this about?

It's also part of the requirements.txt file so why it's needed like this here?

@tmikula-dev tmikula-dev May 26, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, if I understand it correctly, the requirements.txt confluent-kafka pins its version. This --no-binary way of forcing source compilation against the system librdkafka, that is set above. That one includes the Kerberos support.

The PyPI has an issue, that it compiles without GSSAPI (would break SASL_SSL authentication).

So requirements holds the version and --no-binary says how pip builds the confluent-kafka dependency. I added a comment into the Dockerfile to be more clear about that step: 9fd16fc

@lsulak lsulak May 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I didn't know about any of this. Thanks!

echo "##############" && \
echo "### cleanup ###" && \
echo "##############" && \
Expand Down
12 changes: 12 additions & 0 deletions requirements-dev.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
-r requirements.txt
psycopg2-binary==2.9.12
pytest==9.0.3
pytest-cov==7.1.0
pytest-mock==3.15.1
pylint==4.0.5
black==26.5.1
mypy==2.1.0
mypy-extensions==1.1.0
moto[s3,secretsmanager,events]==5.2.1
testcontainers==4.14.2
docker==7.1.0
14 changes: 1 addition & 13 deletions requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,21 +1,9 @@
pytest==9.0.3
pytest-cov==7.1.0
pytest-mock==3.15.1
pylint==4.0.5
black==26.5.1
mypy==2.1.0
mypy-extensions==1.1.0
urllib3==2.7.0
cryptography==48.0.0
jsonschema==4.26.0
PyJWT==2.13.0
requests==2.34.2
boto3==1.43.14
aiosql==15.0
botocore==1.43.14
aiosql==15.0
confluent-kafka==2.14.0
moto[s3,secretsmanager,events]==5.2.1
testcontainers==4.14.2
docker==7.1.0
# psycopg2-binary==2.9.10 # Ideal for local development, but not for long-term production use
psycopg2==2.9.12