Skip to content

Commit 0172a9e

Browse files
arledesmaarledesma
committed
Publish image to container registries
Add Github Action for Container build and push Limit number of container layers and reduce size Update COPY chmod to use octal Stop copying DOCS directory into container Reduce number of production container layers Remove linux/arm64 container platform Convert django container to multi-stage build Reduction of image from ~1GB to <550MB Conslidate production/local docker-compose/Dockerfile Move compose/production/* into parent compose/ folder Remove defunct compose/local folder Remove sharing lock from cache mount on RUN The cache was empty when using a `locked` sharing type. Reverting to the default sharing of `shared` as the build should not be affected with the current workflow. https://github.com/moby/buildkit/blob/f2a6e83adcb0295099870489b76d3ce74d6f7f42/frontend/dockerfile/docs/syntax.md#run---mounttypecache > This mount type allows the build container to cache directories for compilers and package managers. |Option |Description| |---------------------|-----------| |`sharing` | One of `shared`, `private`, or `locked`. Defaults to `shared`. A `shared` cache mount can be used concurrently by multiple writers. `private` creates a new mount if there are multiple writers. `locked` pauses the second writer until the first one releases the mount.| Update workflow to run on 'master' branch Limit GITHUB_TOKEN permissions for job Only push the container on push and schedule events ghaction-docker-meta action moved from crazy-max to docker org https://github.com/docker/metadata-action/releases/tag/v3.0.0 Use bind mount rather than cache for wheels cache is not guaranteed. build the wheels and allow pip to cache during the process. bind mount the wheel-dir when installing in django stage Consolidate start commands Remove extraneous instructions from Dockerfile - no need to install the requirements in the build stage - only build the wheels - gecos is for storing metadata about a user (full name, phone number...) - copy of requirements from build is now handled transparently through a bind mount without requiring the additional layer - du of /tmp/wheels was only for debugging the cache mount which is now a bind mount - /tmp/requirements is a bind mount so it does not need to be removed from the stage Cache first stage of multi-stage build By default the mode is set to `min`, which only exports layers to the cache in the final build stage. We want to cache the first stage in order to not always build wheels. Specify ghostwriter:2.2 image in production.yml Add missing EOF newline to docker components Update to default compose in run-unit-tests github workflow job local.yml was replaced with docker-compose.override.yml which is automatically applied when no other compose files are specified
1 parent 19f89bf commit 0172a9e

File tree

33 files changed

+650
-521
lines changed

33 files changed

+650
-521
lines changed

‎.dockerignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,4 @@
22
!.coveragerc
33
!.env
44
!.pylintrc
5+
DOCS/**

‎.gitattributes‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,3 @@
1-
* text=auto
1+
* text=auto eol=lf
2+
*.ico binary
3+
ghostwriter/static/images/** binary

‎.github/workflows/push-container.yml‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
name: push-container
2+
3+
on:
4+
push:
5+
branches:
6+
- 'master'
7+
- 'feature/push-to-container-registry'
8+
- 'feature/push-to-container-registry-v3.0.0'
9+
tags:
10+
- 'v*'
11+
schedule:
12+
# * is a special character in YAML so you have to quote this string
13+
# at 03:00 on the 1st and 15th of the month
14+
- cron: '0 3 1,15 * *'
15+
16+
env:
17+
PLATFORMS: linux/amd64 # multiple platforms can be specified as: linux/amd64,linux/arm64
18+
19+
jobs:
20+
build_and_push_container_image:
21+
runs-on: ubuntu-latest
22+
23+
permissions:
24+
# when permissions are defined only those that are explicitly set will be enabled
25+
# this workflow job currently only requires reading contents and writing packages.
26+
# https://docs.github.com/en/actions/reference/authentication-in-a-workflow#modifying-the-permissions-for-the-github_token
27+
contents: read
28+
packages: write
29+
30+
steps:
31+
- name: Checkout
32+
uses: actions/checkout@v2
33+
34+
- name: Validate secret defined
35+
id: from_secrets
36+
run: |
37+
github_container_push="true";
38+
dockerhub_token_exists="false";
39+
dockerhub_username_exists="false";
40+
dockerhub_namespace_exists="false";
41+
42+
[[ -n "${{ secrets.DOCKERHUB_TOKEN }}" ]] && dockerhub_token_exists="true";
43+
[[ -n "${{ secrets.DOCKERHUB_USERNAME }}" ]] && dockerhub_username_exists="true";
44+
[[ -n "${{ secrets.DOCKERHUB_NAMESPACE }}" ]] && dockerhub_namespace_exists="true";
45+
[[ "true" = "${{ secrets.GITHUB_CONTAINER_PUSH_DISABLED }}" ]] && github_container_push="false";
46+
47+
echo "::set-output name=dockerhub_token_exists::${dockerhub_token_exists}";
48+
echo "::set-output name=dockerhub_username_exists::${dockerhub_username_exists}";
49+
echo "::set-output name=dockerhub_namespace_exists::${dockerhub_namespace_exists}";
50+
echo "::set-output name=github_container_push::${github_container_push}";
51+
52+
- name: Generate container image names
53+
id: generate_image_names
54+
run: |
55+
repository_name="$(basename "${GITHUB_REPOSITORY}")";
56+
images=();
57+
58+
if [[ "${{ steps.from_secrets.outputs.github_container_push }}" = "true" ]];
59+
then
60+
# set GITHUB_CONTAINER_PUSH_DISABLED to a value of true to disable pushing to github container registry
61+
images+=("ghcr.io/${GITHUB_REPOSITORY}");
62+
fi
63+
64+
if [[ -n "${{ secrets.DOCKERHUB_TOKEN }}" ]] && [[ -n "${{ secrets.DOCKERHUB_USERNAME }}" ]] && [[ -n "${{ secrets.DOCKERHUB_NAMESPACE }}" ]];
65+
then
66+
# dockerhub repository should be the same as the github repository name, within the dockerhub namespace (organization or personal)
67+
images+=("${{ secrets.DOCKERHUB_NAMESPACE }}/${repository_name}");
68+
fi
69+
70+
# join the array for Docker meta job to produce image tags
71+
# https://github.com/crazy-max/ghaction-docker-meta#inputs
72+
echo "::set-output name=images::$(IFS=,; echo "${images[*]}")";
73+
74+
- name: Docker ghostwriter meta
75+
id: meta
76+
uses: docker/metadata-action@v3
77+
with:
78+
images: ${{ steps.generate_image_names.outputs.images }}
79+
tags: |
80+
type=schedule,pattern={{date 'YYYYMMDD'}}
81+
type=edge,branch=master
82+
type=ref,event=branch
83+
type=ref,event=pr
84+
type=ref,event=tag
85+
type=semver,pattern={{version}}
86+
type=semver,pattern={{major}}.{{minor}}
87+
type=semver,pattern={{major}}
88+
89+
- name: Docker ghostwriter:postgres meta
90+
id: meta-postgres
91+
uses: docker/metadata-action@v3
92+
with:
93+
images: ${{ steps.generate_image_names.outputs.images }}
94+
flavor: |
95+
prefix=postgres-
96+
tags: |
97+
type=schedule,pattern={{date 'YYYYMMDD'}}
98+
type=edge,branch=master
99+
type=ref,event=branch
100+
type=ref,event=pr
101+
type=ref,event=tag
102+
type=semver,pattern={{version}}
103+
type=semver,pattern={{major}}.{{minor}}
104+
type=semver,pattern={{major}}
105+
106+
- name: Set up QEMU
107+
uses: docker/setup-qemu-action@v1
108+
109+
- name: Set up Docker Buildx
110+
id: buildx
111+
uses: docker/setup-buildx-action@v1
112+
with:
113+
version: latest
114+
115+
- name: Login to DockerHub
116+
uses: docker/login-action@v1
117+
# conditions do not have direct access to github secrets so we check the output of the step from_secrets
118+
if: ${{ steps.from_secrets.outputs.dockerhub_namespace_exists == 'true' && steps.from_secrets.outputs.dockerhub_token_exists == 'true' && steps.from_secrets.outputs.dockerhub_username_exists == 'true' }}
119+
with:
120+
username: ${{ secrets.DOCKERHUB_USERNAME }}
121+
password: ${{ secrets.DOCKERHUB_TOKEN }}
122+
123+
- name: Login to GitHub Container Registry
124+
uses: docker/login-action@v1
125+
if: ${{ steps.from_secrets.outputs.github_container_push == 'true' }}
126+
with:
127+
registry: ghcr.io
128+
username: ${{ github.repository_owner }}
129+
password: ${{ secrets.GITHUB_TOKEN }}
130+
131+
- name: Cache Docker layers
132+
uses: actions/cache@v2
133+
with:
134+
path: /tmp/.buildx-cache
135+
# Caches are scoped to the current branch and parent branch.
136+
# Cache miss can happen on first run of a new branch
137+
# If there is a matching cache key in the default branch then that should be used.
138+
# https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows#matching-a-cache-key
139+
# cache key is a hash of the base and production requirements. Changes to these files will cause a full rebuild.
140+
key: ${{ runner.os }}-buildx-${{ hashFiles('requirements/base.txt', 'requirements/production.txt') }}
141+
restore-keys: |
142+
${{ runner.os }}-buildx-
143+
144+
- name: Build and push - ghostwriter
145+
uses: docker/build-push-action@v2
146+
with:
147+
builder: ${{ steps.buildx.outputs.name }}
148+
context: .
149+
file: ./compose/django/Dockerfile
150+
platforms: ${{ env.PLATFORMS }}
151+
push: ${{ contains(fromJson('["push", "schedule"]'), github.event_name) }}
152+
labels: ${{ steps.meta.outputs.labels }}
153+
tags: ${{ steps.meta.outputs.tags }}
154+
cache-from: type=local,src=/tmp/.buildx-cache
155+
cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max
156+
# type=gha will replace type=local when a buildx release containing
157+
# https://github.com/docker/buildx/commit/5ca0cbff8ed63450a6d4a3b32659e9521d329a43 is published
158+
# https://github.com/docker/buildx/pull/535
159+
# cache-from: type=gha
160+
# cache-to: type=gha
161+
162+
- name: Build and push - ghostwriter:postgres
163+
uses: docker/build-push-action@v2
164+
with:
165+
builder: ${{ steps.buildx.outputs.name }}
166+
context: .
167+
file: ./compose/postgres/Dockerfile
168+
platforms: ${{ env.PLATFORMS }}
169+
push: ${{ contains(fromJson('["push", "schedule"]'), github.event_name) }}
170+
labels: ${{ steps.meta-postgres.outputs.labels }}
171+
tags: ${{ steps.meta-postgres.outputs.tags }}
172+
cache-from: type=local,src=/tmp/.buildx-cache-new
173+
cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max
174+
# type=gha will replace type=local when a buildx release containing
175+
# https://github.com/docker/buildx/commit/5ca0cbff8ed63450a6d4a3b32659e9521d329a43 is published
176+
# https://github.com/docker/buildx/pull/535
177+
# cache-from: type=gha
178+
# cache-to: type=gha
179+
180+
- name: Move cache
181+
# This step can be removed when cache-from/cache-to have been updated to use type=gha
182+
# https://github.com/docker/build-push-action/issues/252
183+
# https://github.com/moby/buildkit/issues/1896
184+
if: always()
185+
run: |
186+
rm -rf /tmp/.buildx-cache
187+
mv /tmp/.buildx-cache-new /tmp/.buildx-cache

‎compose/django/Dockerfile‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
#syntax=docker/dockerfile:1
2+
ARG STAGE=production
3+
4+
# ---------------------------------------------
5+
# BEGIN build image stage
6+
# ---------------------------------------------
7+
FROM python:3.8-alpine as build
8+
ARG STAGE=production
9+
10+
# only update build build when requirements have changed
11+
COPY ./requirements /requirements
12+
# install build dependencies
13+
RUN --mount=type=cache,mode=0755,target=/root/.cache/pip \
14+
apk update \
15+
&& apk add --no-cache build-base \
16+
# psycopg2 dependencies
17+
&& apk add --no-cache --virtual build-deps gcc python3-dev musl-dev \
18+
&& apk add --no-cache postgresql-dev \
19+
# Pillow dependencies
20+
&& apk add --no-cache jpeg-dev zlib-dev freetype-dev lcms2-dev openjpeg-dev tiff-dev tk-dev tcl-dev \
21+
# CFFI dependencies
22+
&& apk add --no-cache libffi-dev py-cffi \
23+
# XLSX dependencies
24+
&& apk add --no-cache libxml2-dev libxslt-dev \
25+
# Rust and Cargo required by the ``cryptography`` Python package - only required during build
26+
&& apk add --no-cache rust \
27+
&& apk add --no-cache cargo \
28+
# build wheels
29+
&& pip install wheel && pip wheel --wheel-dir=/tmp/wheels -r /requirements/${STAGE}.txt \
30+
# remove the virtual package group 'build-deps'
31+
&& apk del build-deps
32+
# ---------------------------------------------
33+
# END build image stage
34+
# ---------------------------------------------
35+
36+
# ---------------------------------------------
37+
# BEGIN django image stage
38+
# ---------------------------------------------
39+
FROM python:3.8-alpine as django
40+
ARG STAGE=production
41+
42+
# stream python output for django logs
43+
ENV PYTHONUNBUFFERED 1
44+
45+
ENV PYTHONPATH="$PYTHONPATH:/app/config"
46+
47+
ARG USER_UID=1000
48+
ARG USER_GID=$USER_UID
49+
RUN if [ -n "$(getent group ${USER_GID})" ]; \
50+
then \
51+
apk --no-cache add shadow; \
52+
groupmod -n "django" "${USER_GID}"; \
53+
else \
54+
addgroup --gid "${USER_GID}" "django"; \
55+
fi && \
56+
if [ -n "$(getent passwd ${USER_UID})" ]; \
57+
then \
58+
apk --no-cache add shadow; \
59+
usermod -l "django" -g "${USER_GID}" -d "/app"; \
60+
else \
61+
adduser \
62+
--home "/app" \
63+
--shell /bin/ash \
64+
--ingroup "django" \
65+
--system \
66+
--disabled-password \
67+
--no-create-home \
68+
--uid "${USER_UID}" \
69+
"django"; \
70+
fi
71+
72+
# install runtime dependencies. `add --no-cache` performs an apk update, adds packages and excludes caching
73+
# in order to not require deletion of apk cache.
74+
RUN apk add --no-cache postgresql-dev \
75+
# Pillow dependencies
76+
jpeg-dev zlib-dev freetype-dev lcms2-dev openjpeg-dev tiff-dev tk-dev tcl-dev \
77+
# CFFI dependencies
78+
libffi-dev py-cffi \
79+
# XLSX dependencies
80+
libxml2-dev libxslt-dev
81+
82+
# combine build and ${STAGE}.txt - remove --no-binary to install our own wheels
83+
RUN --mount=type=bind,target=/tmp/wheels,source=/tmp/wheels,from=build \
84+
--mount=type=bind,target=/requirements,source=/requirements,from=build,readwrite \
85+
--mount=type=cache,mode=0755,target=/root/.cache/pip \
86+
( cat /requirements/base.txt; sed -e 's/--no-binary.*//' -e 's/^-r .*//' /requirements/${STAGE}.txt ) | tee /tmp/requirements.txt >/dev/null \
87+
&& pip install --find-links=/tmp/wheels -r /tmp/requirements.txt \
88+
&& rm -rf /tmp/requirements.txt
89+
# ---------------------------------------------
90+
# END django image stage
91+
# ---------------------------------------------
92+
93+
# ---------------------------------------------
94+
# BEGIN production stage
95+
# ---------------------------------------------
96+
FROM django as django-production
97+
98+
# add our application
99+
COPY --chown=django . /app
100+
101+
# copy the entrypoint and run scripts
102+
RUN for target in /app/compose/django/*; \
103+
do ln "$target" /"$(basename "$target")" \
104+
&& chmod -v 0755 /"$(basename "$target")" \
105+
# remove all carriage returns in the case that a user checks out the files on a windows system
106+
# and has their git core.eol set to native or crlf
107+
&& sed -i 's/\r$//g' /"$(basename "$target")"; \
108+
done \
109+
# due to volumes mounted to these locations we must created and set the ownership of the underlying directory
110+
# so that it is correctly propagated to the named volume
111+
&& mkdir -p "/app/ghostwriter/media" "/app/staticfiles" \
112+
&& chown -R "django": "/app/ghostwriter/media" "/app/staticfiles"
113+
# ---------------------------------------------
114+
# END production stage
115+
# ---------------------------------------------
116+
117+
# ---------------------------------------------
118+
# BEGIN local stage
119+
# ---------------------------------------------
120+
FROM django as django-local
121+
122+
# add our application CMD scripts
123+
COPY --chown=django ./compose/django/ /
124+
125+
# copy the entrypoint and run scripts
126+
RUN find / -maxdepth 1 -type f -exec chmod -v 0755 {} \; \
127+
# remove all carriage returns in the case that a user checks out the files on a windows system
128+
# and has their git core.eol set to native or crlf
129+
&& find / -maxdepth 1 -type f -exec sed -i 's/\r$//g' {} \; \
130+
# due to volumes mounted to these locations we must created and set the ownership of the underlying directory
131+
# so that it is correctly propagated to the named volume
132+
&& mkdir -p "/app/ghostwriter/media" "/app/staticfiles" \
133+
&& chown -R "django": "/app/ghostwriter/media" "/app/staticfiles"
134+
# ---------------------------------------------
135+
# END local stage
136+
# ---------------------------------------------
137+
138+
# ---------------------------------------------
139+
# BEGIN conditional stage
140+
# with buildkit/bake only referenced stages will be built starting from this stage
141+
# ---------------------------------------------
142+
FROM django-${STAGE} as conditional
143+
144+
USER "django"
145+
146+
WORKDIR /app
147+
# ---------------------------------------------
148+
# END conditional stage
149+
# ---------------------------------------------
150+
151+
# ---------------------------------------------
152+
# BEGIN live stage
153+
# ---------------------------------------------
154+
FROM conditional as live
155+
156+
VOLUME ["/app/ghostwriter/media", "/app/staticfiles"]
157+
158+
CMD ["/start"]
159+
ENTRYPOINT ["/entrypoint"]
160+
# ---------------------------------------------
161+
# END live stage
162+
# ---------------------------------------------

compose/production/django/entrypoint renamed to compose/django/entrypoint

File renamed without changes.

compose/local/django/seed_data renamed to compose/django/seed_data

File renamed without changes.

0 commit comments

Comments
 (0)