Skip to content

Commit e55bf6c

Browse files
mart-rmart-r
authored
CU-8699k4eem sync demo webapp (#17)
* CU-8698up3x0: Now depends on medcatv2 * Move to MedCAT v2-based approach * Show that this is v2 on the base template * CU-8698up3x0: Update python to 3.11 and add git installation * CU-8698up3x0: Fix json-serialisation for numpy.float32 * CU-8698up3x0: Use multi-stage build to lower image size * CU-8698up3x0: Make sure to keep pip cache for 2nd stage * CU-8698up3x0: Update pip before installing requirements * CU-8698up3x0: Use fewer layers * CU-8698up3x0: Make sure installed python packages are available in stage 2 * CU-8698up3x0: Allow building image with no GPU components * CU-8698up3x0: Fix requirements file path in new dockerfile * CU-8698up3x0: Bump dependency to v0.2.3 * CU-8698up3x0: Fix medcat2 install - remove non-existent optional part * CU-8698up3x0: Update demo webapp to v0.3.0 * CU-8698up3x0: Update requirements to v0.3.1 * Move to MedCAT v2 0.3.3 * CU-8698up3x0: Small whitespace fix * CU-8698up3x0: Add running MedCAT (v2) version to header * CU-8698up3x0: Fix version string in base render * CU-8698up3x0: Update medcat installation version (to v0.8.0) * CU-8698up3x0: Fix imports to correct root package * CU-8698up3x0: Fix medcat version context * CU-8698up3x0: Clear things after install * CU-8698up3x0: Add default for version being run in template * CU-8698up3x0: Fix renamed ents attribute * CU-8699gb53b: Add endpoint for UMLS api-key validation * CU-8699gb53b: Update requirements to latest medcat * CU-8699gb53b: Use latest compatible python (3.12) * CU-8699gb53b: Update Dockerfile to use a smaller build / image * CU-8699gb53b: Update Dockerfile to use python 3.12 correctly * CU-8699gb53b: Add template for UMLS API key entry * CU-8699gb53b: Fix UMLS API validation * CU-8699k4eem: Update dependency to latest / v2 based * CU-8699k4eem: Add a simple workflow to check demo web app * CU-8699k4eem: Make integration tests executable * CU-8699k4eemL Fix workflow time test for demo * CU-8699k4eem: Move bulk of tests to python script * CU-8699k4eem: Add response output to tests * CU-8699k4eem: Fix teardown step (typo in compose file name) * CU-8699k4eem: Add more robust test for output
1 parent d9b504a commit e55bf6c

File tree

12 files changed

+321
-21
lines changed

12 files changed

+321
-21
lines changed

‎.github/workflows/medcat-demo-app_build.yml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
name: medcat-demo - Test
2+
3+
on:
4+
push:
5+
branches: [ main ]
6+
pull_request:
7+
branches: [ main ]
8+
9+
defaults:
10+
run:
11+
working-directory: ./medcat-demo-app
12+
13+
jobs:
14+
integration-test:
15+
runs-on: ubuntu-latest
16+
17+
steps:
18+
- uses: actions/checkout@v3
19+
20+
- name: Set up Docker Compose
21+
run: sudo apt-get update && sudo apt-get install -y docker-compose
22+
23+
- name: Build and start containers
24+
run: docker-compose -f docker-compose-test.yml up -d --build
25+
26+
- name: Run integration test
27+
run: ./tests/test_integration.sh
28+
29+
- name: Check container logs for errors
30+
run: |
31+
docker-compose logs medcatweb
32+
docker-compose logs medcatweb | grep -i 'error' && exit 1 || true
33+
34+
- name: Tear down
35+
run: docker-compose -f docker-compose-test.yml down

‎medcat-demo-app/docker-compose-test.yml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
version: '3.4'
2+
3+
services:
4+
medcatweb:
5+
build:
6+
context: ./webapp
7+
command: >
8+
bash -c "/etc/init.d/cron start &&
9+
python /webapp/manage.py runserver 0.0.0.0:8000"
10+
volumes:
11+
- ../medcat-v2/tests/resources:/webapp/models
12+
ports:
13+
- "8000:8000"
14+
env_file:
15+
- ./envs/env_medmen_test
16+
tty: true

‎medcat-demo-app/envs/env_medmen_test‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
MODEL_PACK_PATH=/webapp/models/mct2_model_pack.zip

‎medcat-demo-app/tests/test_integration.py‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# tests/test_integration.py
2+
3+
import requests
4+
5+
from bs4 import BeautifulSoup
6+
7+
URL = "http://localhost:8000/"
8+
session = requests.Session()
9+
10+
# GET the page to get the CSRF token
11+
resp = session.get(URL)
12+
soup = BeautifulSoup(resp.text, "html.parser")
13+
csrf = soup.find("input", {"name": "csrfmiddlewaretoken"}).get("value")
14+
15+
disease = "kidney failure"
16+
17+
text = f"Patient had been diagnosed with acute {disease} the week before"
18+
19+
# POST with the token and same session (cookies preserved)
20+
resp = session.post(URL, data={
21+
"text": text,
22+
"csrfmiddlewaretoken": csrf})
23+
24+
print(f"RESPOONSE:\n{resp.text}")
25+
26+
soup = BeautifulSoup(resp.text, "html.parser")
27+
28+
annotations = soup.select("div.entities mark.entity")
29+
assert annotations, "No annotations found in the response"
30+
assert any(disease in mark.text.lower() for mark in annotations), (
31+
f"Disease '{disease}' not found in annotations")
32+
33+
assert disease in resp.text

‎medcat-demo-app/tests/test_integration.sh‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
#!/bin/bash
2+
set -euo pipefail
3+
4+
echo "Waiting for service to start..."
5+
6+
# install bs4 for tests
7+
python -m pip install beautifulsoup4
8+
9+
# Wait until curl doesn't fail
10+
for i in {1..60}; do
11+
if curl -s --fail http://localhost:8000 > /dev/null; then
12+
echo "Service is up"
13+
break
14+
else
15+
echo "Waiting ($i)..."
16+
sleep 2
17+
fi
18+
done
19+
20+
python tests/test_integration.py

‎medcat-demo-app/webapp/Dockerfile‎

Lines changed: 48 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,37 +1,69 @@
1-
FROM python:3.7
1+
# Stage 1: Build stage (dependencies and compilation)
2+
FROM python:3.12-slim as build
23

34
# Create the required folders
45
RUN mkdir -p /webapp/models
56

6-
# Copy everything
7+
# Copy the application code
78
COPY . /webapp
89

10+
# Install dependencies for building (git, etc.)
11+
RUN apt-get update && apt-get install -y --no-install-recommends \
12+
git \
13+
build-essential \
14+
apt-utils \
15+
cron \
16+
sqlite3 \
17+
libsqlite3-dev
18+
19+
# Install Python dependencies
20+
ARG USE_CPU_TORCH=false
21+
# NOTE: Allow building without GPU so as to lower image size (disabled by default)
22+
RUN pip install -U pip && if [ "$USE_CPU_TORCH" = "true" ]; then \
23+
pip install -r /webapp/requirements.txt --extra-index-url https://download.pytorch.org/whl/cpu/; \
24+
else \
25+
pip install -r /webapp/requirements.txt; \
26+
fi
27+
28+
# Get the spacy model (for later copy)
29+
RUN python -m spacy download en_core_web_md
30+
31+
# Stage 2: Final (production) image
32+
FROM python:3.12-slim as final
33+
34+
# Install runtime dependencies (you don’t need git in production)
35+
RUN apt-get update && apt-get install -y --no-install-recommends \
36+
cron \
37+
sqlite3 \
38+
libsqlite3-dev && apt-get autoremove
39+
40+
# Create the required folders (if not created already)
41+
RUN mkdir -p /webapp/models && mkdir -p /medcat_data
42+
43+
# Copy only necessary files from build stage
44+
COPY --from=build /webapp /webapp
45+
# COPY --from=build /root/.cache /root/.cache # Copy pip cache if needed
46+
47+
# Copy Python site-packages (installed by pip) from build stage
48+
COPY --from=build /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages
49+
50+
# Set environment variables
951
ENV VOCAB_URL=https://medcat.rosalind.kcl.ac.uk/media/vocab.dat
1052
ENV CDB_URL=https://medcat.rosalind.kcl.ac.uk/media/cdb-medmen-v1.dat
1153

1254
ENV CDB_PATH=/webapp/models/cdb.dat
1355
ENV VOCAB_PATH=/webapp/models/vocab.dat
1456

15-
# Create the data directory
16-
RUN mkdir -p /medcat_data
17-
1857
# Set the pythonpath
1958
WORKDIR /webapp
2059

21-
RUN pip install -r requirements.txt
22-
23-
# Get the spacy model
24-
RUN python -m spacy download en_core_web_md
60+
# Create the db backup cron job (copied from your setup)
61+
COPY etc/cron.d/db-backup-cron /etc/cron.d/db-backup-cron
62+
RUN chmod 0644 /etc/cron.d/db-backup-cron && crontab /etc/cron.d/db-backup-cron
2563

26-
# Build the db
64+
# Run migrations and collect static (could be in entrypoint script)
2765
RUN python manage.py makemigrations && \
2866
python manage.py makemigrations demo && \
2967
python manage.py migrate && \
3068
python manage.py migrate demo && \
3169
python manage.py collectstatic --noinput
32-
33-
# Create the db backup cron job
34-
RUN apt-get update && apt-get install -y --no-install-recommends apt-utils cron sqlite3 libsqlite3-dev
35-
COPY etc/cron.d/db-backup-cron /etc/cron.d/db-backup-cron
36-
RUN chmod 0644 /etc/cron.d/db-backup-cron
37-
RUN crontab /etc/cron.d/db-backup-cron

‎medcat-demo-app/webapp/demo/forms.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,3 +46,8 @@ class Meta:
4646
"funder": forms.TextInput(attrs={"size": 40}),
4747
"use_case": forms.Textarea(attrs={"rows": 5, "cols": 40}),
4848
}
49+
50+
51+
class UMLSApiKeyForm(forms.Form):
52+
apikey = forms.CharField(label='UMLS API Key',
53+
widget=forms.TextInput(attrs={'size': 50}))

‎medcat-demo-app/webapp/demo/templates/base.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
</head>
1616
<body>
1717
<nav class="navbar navbar-expand-lg navbar-light bg-light">
18-
<a class="navbar-brand" href="/">MedCAT</a>
18+
<a class="navbar-brand" href="/">MedCAT v2 (running v{{ medcat_version|default:"N/A" }})</a>
1919

2020
<button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
2121
<span class="navbar-toggler-icon"></span>

‎medcat-demo-app/webapp/demo/templates/umls_api_key_entry.html‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
{% extends 'base.html' %}
2+
{% load static %}
3+
4+
{% block style %}
5+
<link rel="stylesheet" href="{% static 'css/home.css' %}">
6+
<link rel="stylesheet" href="{% static 'css/annotations.css' %}">
7+
<link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.8.2/css/all.css" integrity="sha384-oS3vJWv+0UjzBfQzYUhtDYW+Pj2yciDJxpsK1OYPAYjqT085Qq/1cq5FLXAZQ7Ay" crossorigin="anonymous">
8+
<link rel="icon" href="{% static 'image/favicon.ico' %}">
9+
{% endblock %}
10+
11+
{% block body %}
12+
<div class="container-fluid">
13+
<h5>Please enter your UMLS API key to verify your license:</h5>
14+
15+
{% if message %}
16+
<div class="alert alert-info">
17+
{{ message }}
18+
</div>
19+
{% endif %}
20+
21+
{% if form.errors %}
22+
<div class="alert alert-danger">
23+
<ul>
24+
{% for field in form %}
25+
{% for error in field.errors %}
26+
<li><strong>{{ field.label }}:</strong> {{ error }}</li>
27+
{% endfor %}
28+
{% endfor %}
29+
{% for error in form.non_field_errors %}
30+
<li>{{ error }}</li>
31+
{% endfor %}
32+
</ul>
33+
</div>
34+
{% endif %}
35+
36+
<form method="post">
37+
{% csrf_token %}
38+
<div class="form-group">
39+
{{ form.apikey.label_tag }}
40+
{{ form.apikey }}
41+
</div>
42+
<button type="submit" class="btn btn-primary">Verify API Key</button>
43+
</form>
44+
45+
<p class="mt-3 text-muted">
46+
You can find your API key by logging into your UMLS account and visiting your <a href="https://uts.nlm.nih.gov/uts/profile" target="_blank">UMLS Profile</a>.
47+
</p>
48+
</div>
49+
{% endblock %}

‎medcat-demo-app/webapp/demo/urls.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,5 +5,6 @@
55
urlpatterns = [
66
path('', show_annotations, name='train_annotations'),
77
path('auth-callback', validate_umls_user, name='validate-umls-user'),
8+
path('auth-callback-api', validate_umls_api_key, name='validate-umls-api-key'),
89
path('download-model', download_model, name="download-model")
910
]

0 commit comments

Comments
 (0)