Skip to content

Commit afcb20a

Browse files
JackLewis-digiratiJackLewis-digirati
authored
Merge pull request #5 from dlcs/feature/cleanup-handler
Create a cleanup handler for the varnish cache to provide 'ban' functionality
2 parents 29f9fb2 + db2212a commit afcb20a

File tree

16 files changed

+643
-51
lines changed

16 files changed

+643
-51
lines changed

‎.github/actions/docker-build-and-push/action.yml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
name: Docker Build & Push
2+
description: Composite GitHub Action to build and push Docker images to the DLCS GitHub Packages repositories.
3+
4+
inputs:
5+
image-name:
6+
description: "Name of the image to push to the GHCR repository."
7+
required: true
8+
dockerfile:
9+
description: "The Dockerfile to build and push."
10+
required: true
11+
context:
12+
description: "The context to use when building the Dockerfile."
13+
required: true
14+
github-token:
15+
description: "The GitHub token used when interacting with GCHR."
16+
required: true
17+
18+
runs:
19+
using: "composite"
20+
steps:
21+
- id: checkout
22+
uses: actions/checkout@v2
23+
- id: docker-setup-buildx
24+
uses: docker/setup-buildx-action@v2
25+
with:
26+
driver-opts: |
27+
image=moby/buildkit:v0.10.6
28+
- id: docker-meta
29+
uses: docker/metadata-action@v4
30+
with:
31+
images: ghcr.io/dlcs/${{ inputs.image-name }}
32+
tags: |
33+
type=ref,event=branch
34+
type=ref,event=pr
35+
type=sha,enable=true,prefix=,format=long
36+
type=semver,pattern={{version}}
37+
type=semver,pattern={{major}}.{{minor}}
38+
type=semver,pattern={{major}}
39+
- id: docker-login
40+
uses: docker/login-action@v2
41+
with:
42+
registry: ghcr.io
43+
username: ${{ github.actor }}
44+
password: ${{ inputs.github-token }}
45+
- id: docker-build-push
46+
uses: docker/build-push-action@v4
47+
with:
48+
context: ${{ inputs.context }}
49+
file: ${{ inputs.dockerfile }}
50+
builder: ${{ steps.docker-setup-buildx.outputs.name }}
51+
tags: ${{ steps.docker-meta.outputs.tags }}
52+
labels: ${{ steps.docker-meta.outputs.labels }}
53+
push: ${{ github.actor != 'dependabot[bot]' }}

‎.github/workflows/build-deploy.yml‎

Lines changed: 17 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -11,45 +11,25 @@ on:
1111
- master
1212

1313
jobs:
14-
build-push:
14+
build-push-varnish:
1515
runs-on: ubuntu-latest
16-
1716
steps:
18-
- name: Check out code
19-
id: checkout
20-
uses: actions/checkout@v2
21-
22-
- name: Set up Docker Buildx
23-
id: buildx
24-
uses: docker/setup-buildx-action@v2
25-
26-
- name: Login to GitHub Container Registry
27-
uses: docker/login-action@v2
17+
- uses: actions/checkout@v3
18+
- uses: ./.github/actions/docker-build-and-push
19+
name: build and push
2820
with:
29-
registry: ghcr.io
30-
username: ${{ github.actor }}
31-
password: ${{ secrets.GITHUB_TOKEN }}
21+
image-name: "dlcs-varnish"
22+
dockerfile: "Dockerfile"
23+
context: "."
24+
github-token: ${{ secrets.GITHUB_TOKEN }}
3225

33-
- name: Docker meta
34-
id: docker_meta
35-
uses: docker/metadata-action@v4
36-
with:
37-
images: ghcr.io/dlcs/dlcs-varnish
38-
tags: |
39-
type=ref,event=branch
40-
type=ref,event=pr
41-
type=semver,pattern={{version}}
42-
type=semver,pattern={{major}}.{{minor}}
43-
type=sha,enable=true,prefix=,format=long
44-
45-
- name: Build and push
46-
id: docker_build
47-
uses: docker/build-push-action@v4
26+
build-push-dlcs-varnish-cleanup:
27+
runs-on: ubuntu-latest
28+
steps:
29+
- uses: actions/checkout@v3
30+
- uses: ./.github/actions/docker-build-and-push
4831
with:
49-
context: .
50-
builder: ${{ steps.buildx.outputs.name }}
51-
push: true
52-
labels: ${{ steps.docker_meta.outputs.labels }}
53-
tags: ${{ steps.docker_meta.outputs.tags }}
54-
55-
32+
image-name: "dlcs-varnish-cleanup"
33+
dockerfile: "varnish-cleanup/Dockerfile"
34+
context: "./varnish-cleanup"
35+
github-token: ${{ secrets.GITHUB_TOKEN }}

‎.gitignore‎

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
# Byte-compiled / optimized / DLL files
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
6+
# C extensions
7+
*.so
8+
9+
# Distribution / packaging
10+
.Python
11+
build/
12+
Scripts/
13+
develop-eggs/
14+
dist/
15+
downloads/
16+
eggs/
17+
.eggs/
18+
lib/
19+
lib64/
20+
parts/
21+
sdist/
22+
var/
23+
wheels/
24+
share/python-wheels/
25+
*.egg-info/
26+
.installed.cfg
27+
*.egg
28+
MANIFEST
29+
30+
# PyInstaller
31+
# Usually these files are written by a python script from a template
32+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
33+
*.manifest
34+
*.spec
35+
36+
# Installer logs
37+
pip-log.txt
38+
pip-delete-this-directory.txt
39+
40+
# Unit test / coverage reports
41+
htmlcov/
42+
.tox/
43+
.nox/
44+
.coverage
45+
.coverage.*
46+
.cache
47+
nosetests.xml
48+
coverage.xml
49+
*.cover
50+
*.py,cover
51+
.hypothesis/
52+
.pytest_cache/
53+
cover/
54+
55+
# Translations
56+
*.mo
57+
*.pot
58+
59+
# Django stuff:
60+
*.log
61+
local_settings.py
62+
db.sqlite3
63+
db.sqlite3-journal
64+
65+
# Flask stuff:
66+
instance/
67+
.webassets-cache
68+
69+
# Scrapy stuff:
70+
.scrapy
71+
72+
# Sphinx documentation
73+
docs/_build/
74+
75+
# PyBuilder
76+
.pybuilder/
77+
target/
78+
79+
# Jupyter Notebook
80+
.ipynb_checkpoints
81+
82+
# IPython
83+
profile_default/
84+
ipython_config.py
85+
86+
# pyenv
87+
# For a library or package, you might want to ignore these files since the code is
88+
# intended to run in multiple environments; otherwise, check them in:
89+
# .python-version
90+
91+
# pipenv
92+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
93+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
94+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
95+
# install all needed dependencies.
96+
#Pipfile.lock
97+
98+
# poetry
99+
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
100+
# This is especially recommended for binary packages to ensure reproducibility, and is more
101+
# commonly ignored for libraries.
102+
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
103+
#poetry.lock
104+
105+
# pdm
106+
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
107+
#pdm.lock
108+
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
109+
# in version control.
110+
# https://pdm.fming.dev/#use-with-ide
111+
.pdm.toml
112+
113+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
114+
__pypackages__/
115+
116+
# Celery stuff
117+
celerybeat-schedule
118+
celerybeat.pid
119+
120+
# SageMath parsed files
121+
*.sage.py
122+
123+
# Environments
124+
.env
125+
.venv
126+
env/
127+
venv/
128+
ENV/
129+
env.bak/
130+
venv.bak/
131+
pyvenv.cfg
132+
133+
# Spyder project settings
134+
.spyderproject
135+
.spyproject
136+
137+
# Rope project settings
138+
.ropeproject
139+
140+
# mkdocs documentation
141+
/site
142+
143+
# mypy
144+
.mypy_cache/
145+
.dmypy.json
146+
dmypy.json
147+
148+
# Pyre type checker
149+
.pyre/
150+
151+
# pytype static type analyzer
152+
.pytype/
153+
154+
# Cython debug symbols
155+
cython_debug/
156+
157+
# PyCharm
158+
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
159+
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
160+
# and can be added to the global gitignore or merged into this file. For a more nuclear
161+
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
162+
.idea/
163+
164+
# General
165+
*.exe

‎Dockerfile‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,13 @@ RUN pip install awscli
1010
COPY start.sh /start.sh
1111
RUN chmod +x /start.sh
1212

13+
WORKDIR /usr/app/src
14+
COPY varnish-cleanup/requirements.txt ./
15+
RUN pip install -r requirements.txt
16+
17+
COPY varnish-cleanup/cleanup_handler.py ./
18+
COPY varnish-cleanup/app ./app
19+
1320
ENV VARNISH_PORT 80
1421
EXPOSE 80
1522

‎README.md‎

Lines changed: 39 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,17 +2,25 @@
22

33
Docker image using disk-backed Varnish instance for caching.
44

5-
On startup it uses the AWS CLI to copy vcl file from location specified by `S3_VCL_FILE` environment variable.
5+
On startup it uses the AWS CLI to copy vcl file from location specified by `S3_VCL_FILE` environment variable.
6+
Optionally, a local file can also be configured for development purposes
67

78
## Configuration
89

9-
The following environment files are expected:
10-
10+
The following environment settings are expected:
1111
* `S3_VCL_FILE` - The location of a vcl file to use. Expected S3Uri as it is used by [aws s3 cp](https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html) command.
1212
* `VARNISH_CACHE_FOLDER` - Folder where disk backed cache is stored.
1313
* `VARNISH_CACHE_SIZE` - Size of cache.
14-
* `VARNISH_PORT` - Which port Varnish should listen on (defaults to 80)
14+
* `AWS_PROFILE` - Required to run locally
15+
* `INCOMING_QUEUE` - the name of the queue that the cleanup handler listens to
16+
17+
The following configuration is optional:
1518

19+
* `VARNISH_ADDRESS` - The location of varnish used by the cleanup handler. Defaults to localhost
20+
* `AWS_REGION` - The AWS region. Defaults to eu-west-1
21+
* `USE_LOCAL_CONFIG` - Whether to use a local config file over S3.
22+
23+
*NOTE:* using `USE_LOCAL_CONFIG` requires a `mount`to be added to the `docker run` containing the VCL
1624
## Running
1725

1826
```bash
@@ -21,10 +29,34 @@ docker build -t dlcs-varnish:local .
2129

2230
# run
2331
docker run -it --rm \
24-
--env AWS_ACCESS_KEY_ID='xxx' \
25-
--env AWS_SECRET_ACCESS_KEY='xxx' \
2632
--env S3_VCL_FILE='s3://my-bucket/varnish-config.vcl' \
2733
--env VARNISH_CACHE_FOLDER='/path/to/folder' \
28-
--env VARNISH_CACHE_SIZE='100M'
34+
--env VARNISH_CACHE_SIZE='100M' \
35+
--env-file='/path/to/env' \
36+
{REQUIRED FOR LOCAL RUNNING}--volume $HOME\.aws\credentials:/root/.aws/credentials:ro \
37+
{OPTIONAL}--mount type=bind,source=.\etc\default.vcl,target=/mnt/varnish/default.vcl \
38+
dlcs-varnish:local
39+
```
40+
# varnish-cleanup
41+
42+
Additionally, there is a standalone docker container for the cleanup handler.
43+
44+
## Configuration
45+
46+
Required:
47+
* `AWS_PROFILE` - Required to run locally
48+
49+
Optional:
50+
* `VARNISH_ADDRESS` - The location of varnish used by the cleanup handler. Defaults to localhost
51+
* `AWS_REGION` - The region used by the cleanup handler. Defaults to eu-west-1
52+
53+
```bash
54+
# build
55+
docker build -t dlcs-varnish-cleanup:local ./varnish-cleanup
56+
57+
# run
58+
docker run -it --rm \
59+
--env-file='/path/to/env'
60+
{REQUIRED FOR LOCAL RUNNING}--volume=$HOME\.aws\credentials:/root/.aws/credentials:ro
2961
dlcs-varnish:local
3062
```

0 commit comments

Comments
 (0)