Implements backend API, integrates with frontend (#4)

This PR adds the following:
- implements the entire meta2onto API (sorry, probably should've broken
it up...)
- updates endpoint paths, field references in the frontend to match the
backend
- adds a facility for the database to load the latest dump on startup,
similar to ECCO
- downloads the latest database dump if it's not already present on
invoking `run_stack.sh`
- note that the dump is currently around 850mb and I expect that to grow
- if it's missing or doesn't match the remote's exact file size in bytes
(I presume this is enough to ensure it's different), you'll be prompted
with a message to download it that includes its actual remote size
- adds `memcached` for caching expensive responses

Regarding the frontend, here are a few significant changes:
- the Cart component now directly downloads the response from
`/api/cart/download` rather than getting a link and fetching that
- the Autocomplete listing now shows the ontology ID alongside the
result (feel free to remove if you like; that was mostly for me); it
also passes the entity's name to the search page rather than its ID to
match the mockup.
- I vaguely recall them saying that they wanted to select an ontology ID
from the list, but perhaps that's not the case? Right now, the search
will match multiple ontology entities as long as their names are the
same.
- The initial results are currently capped at 50, but due to some joins
that exclude the actual number of results can end up winnowing it down
to ~47. I'm looking into it return exactly as many results as you ask
for while not blowing up the size early in the query.
- Search and Cart were updated to reflect the field names I return from
the backend

## Caveats

Note that the backend API structure is very much a work in progress. For
example, I'd like to combine the GEO metadata series table with the
regular Series table, decide between whether it's going to be flat or
nested, normalize how queries and pagination are done (there are some
custom endpoints, for example), etc. There's also currently no formal
type definition for the API; I intend to add that, as well as Swagger or
ReDoc to surface that information better, as well as add an OpenAPI
schema endpoint for programmatic use.

The data model is also kind of all over the place, due to not knowing
how (or, in many cases, if) the data i was provided would be used in the
app. I have a lot of cleanup to do both on the model and the interface,
but I imagine that it'll be best to address it in a future PR.

The way that searches are conducted is currently complicated, but I
figure I should get the PR out first. I'll explain it in a subsequent
comment on this PR.


## Trying it Out

Since I was only given a subset of ontology terms for testing, only a
small subset of entities in the ontology will actually be able to be
matched to results. (They're all disease entities from the MONDO
ontology, and only a small selection of them at that.) Here's a few
entries that you can search for that actually return responses, pulled
from the `api_searchterms` table joined with `api_ontologyterms`:

Ontology ID           |Query Phrase
------------------|--------------------------------
MONDO:0000270|lower respiratory tract disorder
MONDO:0000637|musculoskeletal system cancer
MONDO:0001416|female reproductive organ cancer

---------

Co-authored-by: Vincent Rubinetti <[email protected]>

Author: falquaddoomi

.gitignore

@@ -217,3 +217,7 @@ __marimo__/
 
 # MacOS stuff
 .DS_Store
+
+# project-specific ignores
+# ignore dumpfiles that aren't explicitly checked in
+/db-exports/*.dump

backend/Dockerfile

@@ -1,5 +1,4 @@
-# Use Python 3.10 slim image as base
-FROM python:3.10-slim
+FROM python:3.12-slim
 
 # Set working directory
 WORKDIR /app

backend/pyproject.toml

@@ -10,8 +10,14 @@ dependencies = [
     "django-rest-framework>=0.1.0",
     "drf-nested-routers>=0.95.0",
     "gunicorn>=23.0.0",
+    "pandas>=2.3.3",
+    "tqdm>=4.67.1",
     "pgpq>=0.9.0",
     "psycopg[binary]>=3.2.10",
+    "django-filter>=25.2",
+    "python-memcached>=1.62",
+    "pymemcache>=4.0.0",
+    "duckdb>=1.4.2",
 ]
 
 # [build-system]

backend/src/api/__init__.py

@@ -0,0 +1,9 @@
+from django.contrib import admin
+
+from api.models import Organism, Platform, Sample, Series, SeriesRelations
+
+admin.site.register(Organism)
+admin.site.register(Platform)
+admin.site.register(Sample)
+admin.site.register(Series)
+admin.site.register(SeriesRelations)

backend/src/api/admin.py

@@ -0,0 +1,6 @@
+from django.apps import AppConfig
+
+
+class ApiConfig(AppConfig):
+    default_auto_field = "django.db.models.BigAutoField"
+    name = "api"

backend/src/api/apps.py

@@ -0,0 +1,80 @@
+import io
+from pathlib import Path
+
+from django.core.management.base import BaseCommand
+from django.db import connection, transaction
+
+import pyarrow.parquet as pq
+import pyarrow.csv as pacsv
+import pyarrow.compute as pc
+
+from api.models import SearchTerm
+
+class Command(BaseCommand):
+    help = "Import SearchTerm rows from a Parquet file using PostgreSQL COPY"
+
+    def add_arguments(self, parser):
+        parser.add_argument("parquet_path", help="Path to meta2onto_example_predictions.parquet")
+        parser.add_argument(
+            "--table",
+            default=SearchTerm._meta.db_table,
+            help="Target DB table name (default: SearchTerm._meta.db_table)",
+        )
+
+    def handle(self, *args, **options):
+        parquet_path = Path(options["parquet_path"])
+        table_name = options["table"]
+
+        # first, truncate SearchTerm
+        self.stdout.write(f"Truncating table {table_name} ...")
+        with connection.cursor() as cursor:
+            cursor.execute(f"TRUNCATE TABLE {table_name} RESTART IDENTITY CASCADE;")
+        self.stdout.write(self.style.SUCCESS(f"Table {table_name} truncated."))
+
+        # 1) Read Parquet
+        self.stdout.write(f"Reading Parquet file: {parquet_path}")
+        table = pq.read_table(parquet_path)
+
+        # 2) Select and rename columns to match DB schema
+        # Parquet: term, ID, prob, log2(prob/prior), related_words
+        # DB:      term, sample_id, prob, log2_prob_prior, related_words
+        table = table.select(["term", "ID", "prob", "log2(prob/prior)", "related_words"])
+        table = table.rename_columns(
+            ["term", "sample_id", "prob", "log2_prob_prior", "related_words"]
+        )
+
+        # # If ID might be missing / null, ensure it's numeric or null
+        # # (Arrow usually infers this correctly; adjust if needed)
+        # if table["sample_id"].type not in (pc.field("dummy", pc.int64()).type,):
+        #     # Try to cast to int64; errors='ignore' will produce nulls for bad values
+        #     table = table.set_column(
+        #         table.schema.get_field_index("sample_id"),
+        #         "sample_id",
+        #         pc.cast(table["sample_id"], pc.int64()),
+        #     )
+
+        # 3) Write to an in-memory CSV with header
+        self.stdout.write("Converting Arrow table to CSV in memory...")
+        buf = io.BytesIO()
+        pacsv.write_csv(table, buf)
+        buf.seek(0)
+
+        # 4) COPY into PostgreSQL using psycopg3's copy()
+        self.stdout.write(f"Copying into table {table_name} ...")
+        cols = ["term", "sample_id", "prob", "log2_prob_prior", "related_words"]
+        copy_sql = f"""
+            COPY {table_name} ({", ".join(cols)})
+            FROM STDIN WITH (FORMAT csv, HEADER true)
+        """
+
+        # buf is bytes; psycopg3 Copy.write() accepts bytes for text/binary COPY
+        with transaction.atomic():
+            with connection.cursor() as cursor:
+                with cursor.copy(copy_sql) as copy:  # <-- psycopg3 API
+                    while True:
+                        chunk = buf.read(1024 * 1024)  # 1 MB chunks
+                        if not chunk:
+                            break
+                        copy.write(chunk)
+
+        self.stdout.write(self.style.SUCCESS("Import completed successfully."))