Skip to content

Commit 9112dca

Browse files
trey-staffordtrey-stafford
authored
Merge pull request #20 from nsidc/quarto-docs
Init quarto for docs
2 parents 1b61524 + 0dc615b commit 9112dca

16 files changed

+388
-113
lines changed

‎README.md‎

Lines changed: 3 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -5,25 +5,7 @@
55
# Data Access Tool (DAT) Backend
66

77
Backend services for the
8-
[Data Access Tool UI](https://github.com/nsidc/data-access-tool-ui) that is
9-
deployed to NSIDC's Virtual Machine (VM) infrastructure via the
10-
[Data Access Tool VM](https://github.com/nsidc/data-access-tool-vm) project.
11-
12-
The DAT Backend is composed of:
13-
14-
- A Flask-based API provides endpoints that power the
15-
[Data Access Tool UI](https://github.com/nsidc/data-access-tool-ui). For
16-
example,
17-
18-
- The`/api/downloader-script` endpoint provides the `python_script.py`
19-
download script for data granules matching a user's filters.
20-
- The `/api/get-links/` endpoint that provides the
21-
[getLinks](https://github.com/nasa/earthdata-download/blob/main/docs/GET_LINKS.md)
22-
service required for DAT integration with the
23-
[NASA Earthdata Downloader](https://github.com/nasa/earthdata-download).
24-
25-
- Docker compose configuration for the DAT backend, which includes
26-
[nginx proxy server configuration](./nginx).
8+
[Data Access Tool UI](https://github.com/nsidc/data-access-tool-ui).
279

2810
## Level of Support
2911

@@ -40,11 +22,11 @@ contact nsidc@nsidc.org for more information.
4022
[docker compose](https://docs.docker.com/compose/)
4123
- Access to NSIDC's internal Virtual Machine infrastructure. It is expected that
4224
this backend system be deployed via the
43-
[data-access-tool-vm](https://github.com/nsidc/data-access-tool-ui) project.
25+
[data-access-tool-vm](https://github.com/nsidc/data-access-tool-vm) project.
4426

4527
## Contributing
4628

47-
See [./doc/DEVELOPMENT.md](./doc/DEVELOPMENT.md).
29+
Contributing documentation can be viewed by running `quarto preview docs/`
4830

4931
## Credit
5032

‎doc/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
/.quarto/
2+
/_site/

‎doc/DEVELOPMENT.md‎

Lines changed: 0 additions & 57 deletions
This file was deleted.

‎doc/_quarto.yml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
project:
2+
type: website
3+
4+
website:
5+
title: "Data Access Tool Backend Documentation"
6+
site-url: "https://nsidc.github.io/data-access-tool-backend"
7+
repo-url: &repo-url "https://github.com/nsidc/data-access-tool-backend"
8+
repo-actions: ["edit", "issue"]
9+
navbar:
10+
left:
11+
- href: getting_started.md
12+
text: "Getting Started"
13+
- href: testing.md
14+
text: "Testing"
15+
- href: developing_with_edd.qmd
16+
text: "Developing with EDD"
17+
- href: envvars.md
18+
- href: logs.md
19+
- href: releasing.md
20+
- href: docs.md
21+
text: "Docs"
22+
23+
page-footer:
24+
left: "© NSIDC 2025"
25+
right: "Built with [Quarto](https://quarto.org/)"
26+
27+
format:
28+
html:
29+
theme:
30+
- cosmo
31+
- brand
32+
css: styles.css
33+
toc: true

doc/developing_with_edd.md renamed to doc/developing_with_edd.qmd

Lines changed: 18 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ See the
88
for detailed information on how Earthdata Download works.
99

1010
Routes supporting the EDD can be found in
11-
[./src/dat_backend/routes/earthdata_download/](./src/dat_backend/routes/earthdata_download/).
11+
`./src/dat_backend/routes/earthdata_download/`.
1212

1313
## EDD interaction overview
1414

@@ -32,7 +32,7 @@ between the DAT UI, the backend, and Earthdata Download.
3232
the last couple of steps until the results are exhausted. The user now has all
3333
of the granules they requested downloaded to their computer.
3434

35-
```mermaid
35+
```{mermaid}
3636
sequenceDiagram
3737
actor user
3838
participant DAT-UI
@@ -64,18 +64,21 @@ To authenticate on behalf of users, the DAT must be registered with the
6464
Earthdata Login system. To manage the DAT EDL app, visit this page:
6565
<https://urs.earthdata.nasa.gov/apps/data_access_tool>.
6666

67-
> [!NOTE] You must have admin access to the application via your personal
68-
> Earthdata Login account to manage the DAT app. Reach out to NSIDC tech
69-
> services or another DAT developer to add you as an admin if needed.
67+
::: {.callout-note}
68+
69+
You must have admin access to the application via your personal Earthdata Login
70+
account to manage the DAT app. Reach out to NSIDC tech services or another DAT
71+
developer to add you as an admin if needed.
72+
73+
:::
7074

7175
The `data_access_tool` EDL app page provides the ability to manage application
7276
admins, set redirection URIs, reset the application password, and more.
7377

7478
The DAT backend requires that the `EARTHDATA_APP_USERNAME`,
7579
`EARTHDATA_APP_PASSWORD`, and `EARTHDATA_APP_CLIENT_ID` envvars be set with
76-
values provided by the EDL app. These are stored in NSIDC's vault instance and
77-
provided as envvars in VMs provisioned by the
78-
[data-access-tool-vm](https://github.com/nsidc/data-access-tool-vm) project.
80+
values provided by the EDL app. See [Environment Variables](envvars.md) for more
81+
information about environment variables used by the DAT backend.
7982

8083
## Running the EDD in dev
8184

@@ -88,9 +91,13 @@ nvm use
8891
npm run start
8992
```
9093

91-
> [!NOTE] On Linux, starting the EDD in dev mode may not properly associate
92-
> `earthdata-download://` deep links with the dev instance of the EDD. See
93-
> https://github.com/nasa/earthdata-download/issues/62
94+
::: {.callout-note}
95+
96+
On Linux, starting the EDD in dev mode may not properly associate
97+
`earthdata-download://` deep links with the dev instance of the EDD. See
98+
https://github.com/nasa/earthdata-download/issues/62
99+
100+
:::
94101

95102
## get-links service
96103

‎doc/docs.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
Documentation is written as markdown and rendered as a website using
2+
[Quarto](https://quarto.org/).
3+
4+
To contribute documentation, update an existing markdown file in `docs/`, or add
5+
a new one.
6+
7+
If adding a new one, include it in the `navbar` entry in `docs/_quarto.yml`.
8+
9+
Preview your changes with:
10+
11+
```bash
12+
quarto preview docs/
13+
```

‎doc/envvars.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Environment Variables
2+
3+
Environment variables are used to configure the DAT backend at runtime. These
4+
environment variables are provided via NSIDC's
5+
[Vault](https://www.hashicorp.com/en/products/vault) instance via virtual
6+
machines provisioned with the
7+
[data-access-tool-vm](https://github.com/nsidc/data-access-tool-vm) project.
8+
9+
Environment variables are exposed to the DAT's components through
10+
`docker compose` configuration files. E.g., see the `docker-compose.yml`'s
11+
`environment` sections.
12+
13+
- `EARTHDATA_APP_USERNAME`: DAT's Earthdata Login App username (See
14+
[Earthdata login](developing_with_edd.qmd#earthdata-login)).
15+
- `EARTHDATA_APP_PASSWORD`: DAT's Earthdata Login App username (See
16+
[Earthdata login](developing_with_edd.qmd#earthdata-login)).
17+
- `EARTHDATA_APP_CLIENT_ID`: DAT's Earthdata Login App client ID (See
18+
[Earthdata login](developing_with_edd.qmd#earthdata-login)).
19+
- `DAT_FLASK_SECRET_KEY`: Secret key used by Flask for session management
20+
(required by Earthdata Auth endpoints). See
21+
<https://flask.palletsprojects.com/en/stable/quickstart/#sessions>.
22+
- `LOGS_SHARE_PATH`: Local path to directory where logs are stored. See
23+
[Logs](logs.md) for more information.

‎doc/getting_started.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Contributing
2+
3+
- Strive to create [tests](testing.md) for all new features/bugfixes.
4+
- New or modified features should be accompanied by adequate
5+
[documentation](docs.md).
6+
- Follow the recommended
7+
[git flow](https://docs.github.com/en/get-started/using-github/github-flow#following-github-flow).
8+
9+
# Setting up a dev environment
10+
11+
## Development on a dev VM at NSIDC
12+
13+
This is the recommended approach to development. See the
14+
[data-access-tool-vm](https://github.com/nsidc/data-access-tool-vm) project
15+
documentation for how to get setup.
16+
17+
## Local development
18+
19+
Local development (not on an NISDC development VM) is not recommended. However,
20+
it is possible.
21+
22+
### Conda environment
23+
24+
It can be useful to install the `dat-backend` conda environment locally so that
25+
tools like `quarto` and `bump-my-version` can be used. To do so,
26+
27+
```bash
28+
conda env create -f environment.yml
29+
conda activate dat-backend
30+
```
31+
32+
### Docker stack
33+
34+
The `docker-compose.local.yml` can be used to run the stack locally:
35+
36+
::: {.callout-note}
37+
38+
You will have to manually setup necessary [Environment Variables](envvars.md)
39+
first.
40+
41+
:::
42+
43+
```bash
44+
ln -s docker-compose.local.yml docker-compose.override.yml
45+
docker compose up -d
46+
```
47+
48+
Now you should be able to visit the API documentation page at
49+
<https://localhost/>.
50+
51+
::: {.callout-note}
52+
53+
Docker logs can be followed via:
54+
55+
```bash
56+
docker compose logs -f
57+
```
58+
59+
:::

‎doc/index.qmd‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
---
2+
title: "Data Access Tool Backend Documentation"
3+
---
4+
5+
Contributor documentation for the Data Access Tool (DAT) Backend.
6+
7+
The DAT backend is deployed to NSIDC's Virtual Machine (VM) infrastructure via
8+
the [Data Access Tool VM](https://github.com/nsidc/data-access-tool-vm) project.
9+
10+
The DAT Backend is composed of:
11+
12+
- A Flask-based API provides endpoints that power the
13+
[Data Access Tool UI](https://github.com/nsidc/data-access-tool-ui). For
14+
example,
15+
16+
- The`/api/downloader-script` endpoint provides the `python_script.py`
17+
download script for data granules matching a user's filters.
18+
- The `/api/get-links/` endpoint that provides the
19+
[getLinks](https://github.com/nasa/earthdata-download/blob/main/docs/GET_LINKS.md)
20+
service required for DAT integration with the
21+
[NASA Earthdata Downloader](https://github.com/nasa/earthdata-download).
22+
23+
- Docker compose configuration for the DAT backend, which includes
24+
nginx proxy server configuration.
25+
26+
27+
## Background
28+
29+
The Data Access Tool backend service is based on the
30+
[hermes-api](https://bitbucket.org/nsidc/hermes-api/src) project at NSIDC, which
31+
provided a backend API for Data Access Tool data orders through an ECS-based
32+
system.
33+
34+
With ECS scheduled to be decommissioned in July 2026, this new backend service
35+
was developed to replace hermes.
36+
37+
This project provides some of the same functionality as hermes (e.g., the Python
38+
download script endpoint, `/api/downloader-script`), removes some functionality
39+
(e.g., ECS-based ordering and direct delivery to Google Drive), and adds one new
40+
key feature: support for the NASA
41+
[Earthdata Download](https://github.com/nasa/earthdata-download/) application.
42+
43+
This service went into production at NSIDC alongside the
44+
[data-access-tool-ui](https://github.com/nsidc/data-access-tool-ui/) v3.1.0 on
45+
April 16, 2025.
46+
47+
::: {.callout-note}
48+
49+
The Data Access Tool project has been referred to as "Everest" and "Hermes" at
50+
NSIDC. Generally speaking, "Hermes" refers to the legacy backend services that
51+
powered the DAT UI <3.1.0. "Everest" referred to the frontend UI. In practice,
52+
these terms were sometimes used interchangeably.
53+
54+
:::

‎doc/logs.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Logs
2+
3+
The Data Access Tool backend produces logs from two primary sources:
4+
5+
- Flask/API application logs produced through the use of e.g., `app.logger.info`
6+
and unhandled exceptions.
7+
- In dev, the Flask dev server produces logs that can be viewed with
8+
`docker compose logs`.
9+
- In non-dev environments, the `gunicorn` server produces logs that can be
10+
viewed with `docker compose logs`.
11+
- NGINX `error` and `access` server logs, which are written to local disk.
12+
13+
## NGINX Server logs
14+
15+
Logs are written to local disk, in `./logs/server/`.
16+
17+
It is expected that these logs be backed-up to a networked share drive (defined
18+
through the `LOGS_SHARE_PATH` [environment variable](envvars.md)) through
19+
`logrotation`. The `dat.access.log` is intended to be used for application
20+
metrics, and these data should be retained for recordkeeping. The
21+
[data-access-tool-vm](https://github.com/nsidc/data-access-tool-vm) project sets
22+
up logrotation for NSIDC deployments.
23+
24+
Each line of the NGINX access logs are formatted as JSON so that the
25+
`/api/metrics` endpoint can easily parse the records.
26+
27+
An example line from an access log:
28+
29+
```json
30+
{
31+
"http_user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0",
32+
"args": "",
33+
"uri": "/api/status",
34+
"http_referer": "https://dev.data-access-tool.trst2284.dev.int.nsidc.org/",
35+
"body_bytes_sent": "551",
36+
"status": "200",
37+
"request": "GET /api/status HTTP/1.1",
38+
"time_iso8601": "2025-04-14T14:31:03+00:00",
39+
"remote_addr": "111.111.111.111",
40+
"x_forwarded_for": "12.345.678.911, 111.111.111.111"
41+
}
42+
```

0 commit comments

Comments
 (0)