Skip to content

Commit c0e0926

Browse files
mmalenicmmalenic
authored
Merge pull request #260 from umccr/refactor/lambda-axum
refactor: htsget-lambda and misc deployment changes
2 parents aa51483 + c45b092 commit c0e0926

File tree

30 files changed

+565
-1262
lines changed

30 files changed

+565
-1262
lines changed

‎Cargo.lock‎

Lines changed: 27 additions & 28 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎deploy/Dockerfile‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM rust:1.76-slim AS builder
1+
FROM rust:1.81-slim AS builder
22

33
LABEL org.opencontainers.image.source=https://github.com/umccr/htsget-rs
44
LABEL org.opencontainers.image.url=https://github.com/umccr/htsget-rs/pkgs/container/htsget-rs
@@ -17,9 +17,9 @@ RUN cargo build --all-features --release && \
1717

1818
FROM gcr.io/distroless/cc-debian12
1919

20-
COPY --from=builder /build/target/release/htsget-actix /usr/local/bin/htsget-actix
20+
COPY --from=builder /build/target/release/htsget-axum /usr/local/bin/htsget-axum
2121

2222
ENV HTSGET_TICKET_SERVER_ADDR 0.0.0.0:8080
2323
ENV HTSGET_DATA_SERVER_ADDR 0.0.0.0:8081
2424

25-
CMD [ "htsget-actix" ]
25+
CMD [ "htsget-axum" ]

‎deploy/Dockerfile.dockerignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,12 @@
11
*
22

33
!/htsget-actix
4+
!/htsget-axum
45
!/htsget-config
56
!/htsget-http
67
!/htsget-lambda
78
!/htsget-search
9+
!/htsget-storage
810
!/htsget-test
911
!/Cargo.toml
1012
!/Cargo.lock

‎deploy/README.md‎

Lines changed: 26 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -13,24 +13,29 @@ The CDK code in this directory constructs a CDK app from [`HtsgetLambdaStack`][h
1313
[`bin/settings.ts`][htsget-settings]:
1414

1515
#### HtsgetSettings
16+
1617
These are general settings for the CDK deployment.
1718

18-
| Name | Description | Type |
19-
|----------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------|
20-
| <span id="config">`config`</span> | The location of the htsget-rs server config. This must be specified. This config file configures the htsget-rs server. See [htsget-config] for a list of available server configuration options. | `string` |
21-
| <span id="domain">`domain`</span> | The domain name for the Route53 Hosted Zone that the htsget-rs server will be under. This must be specified. A hosted zone with this name will either be looked up or created depending on the value of [`lookupHostedZone?`](#lookupHostedZone). | `string` |
22-
| <span id="authorizer">`authorizer`</span> | Deployment options related to the authorizer. Note that this option allows specifying an AWS [JWT authorizer][jwt-authorizer]. The JWT authorizer automatically verifies tokens issued by a Cognito user pool. | [`HtsgetJwtAuthSettings`](#htsgetjwtauthsettings) |
23-
| <span id="subDomain">`subDomain?`</span> | The domain name prefix to use for the htsget-rs server. Together with the [`domain`](#domain), this specifies url that the htsget-rs server will be reachable under. Defaults to `"htsget"`. | `string` |
24-
| <span id="s3BucketResources">`s3BucketResources?`</span> | The resources that are affected by the bucket policy with actions: `["s3:List*", "s3:Get*"]`. If this is not specified, it defaults to `["arn:aws:s3:::*"]`. This affects which buckets are allowed to be accessed with the policy. | `string[]` |
25-
| <span id="lookupHostedZone">`lookupHostedZone?`</span> | Whether to lookup the hosted zone with the domain name. Defaults to `true`. If `true`, attempts to lookup an existing hosted zone using the domain name. Set this to `false` if you want to create a new hosted zone with the domain name. | `boolean` |
19+
| Name | Description | Type |
20+
|--------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------|
21+
| <span id="config">`config`</span> | The location of the htsget-rs server config. This must be specified. This config file configures the htsget-rs server. See [htsget-config] for a list of available server configuration options. | `string` |
22+
| <span id="domain">`domain`</span> | The domain name for the Route53 Hosted Zone that the htsget-rs server will be under. This must be specified. A hosted zone with this name will either be looked up or created depending on the value of [`lookupHostedZone?`](#lookupHostedZone). | `string` |
23+
| <span id="authorizer">`authorizer`</span> | Deployment options related to the authorizer. Note that this option allows specifying an AWS [JWT authorizer][jwt-authorizer]. The JWT authorizer automatically verifies tokens issued by a Cognito user pool. | [`HtsgetJwtAuthSettings`](#htsgetjwtauthsettings) |
24+
| <span id="subDomain">`subDomain?`</span> | The domain name prefix to use for the htsget-rs server. Together with the [`domain`](#domain), this specifies url that the htsget-rs server will be reachable under. Defaults to `"htsget"`. | `string` |
25+
| <span id="s3BucketResources">`s3BucketResources`</span> | The buckets to serve data from. If this is not specified, this defaults to `[]`. This affects which buckets are allowed to be accessed by the policy actions which are `["s3:List*", "s3:Get*"]`. Note that this option does not create buckets, it only gives permission to access them, see the `createS3Buckets` option. This option must be specified to allow `htsget-rs` to access data in buckets that are not created in this stack. | `string[]` |
26+
| <span id="lookupHostedZone">`lookupHostedZone?`</span> | Whether to lookup the hosted zone with the domain name. Defaults to `true`. If `true`, attempts to lookup an existing hosted zone using the domain name. Set this to `false` if you want to create a new hosted zone with the domain name. | `boolean` |
27+
| <span id="createS3Bucket">`createS3Bucket?`</span> | Whether to create a test bucket. Defaults to true. Buckets are created with [`RemovalPolicy.RETAIN`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.RemovalPolicy.html). The correct access permissions are automatically added. | `boolean` |
28+
| <span id="bucketName">`bucketName?`</span> | The name of the bucket created using `createS3Bucket`. The name defaults to an automatically generated CDK name, use this option to override that. This option only has an affect is `createS3Buckets` is true. | `string` |
29+
| <span id="copyTestData">`copyTestData?`</span> | Whether to copy test data into the bucket. Defaults to true. This copies the example data under the `data` directory to those buckets. This option only has an affect is `createS3Buckets` is true. | `boolean` |
2630

2731
#### HtsgetJwtAuthSettings
32+
2833
These settings are used to determine if the htsget API gateway endpoint is configured to have a JWT authorizer or not.
2934

3035
| Name | Description | Type |
31-
|---------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|------------|
36+
| ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
3237
| <span id="public">`public`</span> | Whether this deployment is public. If this is `true` then no authorizer is present on the API gateway and the options below have no effect. | `boolean` |
33-
| <span id="jwtAudience">`jwtAudience?`</span> | A list of the intended recipients of the JWT. A valid JWT must provide an aud that matches at least one entry in this list. | `string[]` |
38+
| <span id="jwtAudience">`jwtAudience?`</span> | A list of the intended recipients of the JWT. A valid JWT must provide an aud that matches at least one entry in this list. | `string[]` |
3439
| <span id="cogUserPoolId?">`cogUserPoolId?`</span> | The cognito user pool id for the authorizer. If this is not set, then a new user pool is created. No user pool is created if [`public`](#public) is true. | `string` |
3540

3641
The [`HtsgetSettings`](#htsgetsettings) are passed into [`HtsgetLambdaStack`][htsget-lambda-stack] in order to change the deployment config. An example of a public instance deployment
@@ -49,7 +54,7 @@ After installing the basic dependencies, complete the following steps:
4954

5055
1. Login to AWS and define `CDK_DEFAULT_*` env variables (if not defined already). You must be authenticated with your AWS cloud to run this step.
5156
2. Install [cargo-lambda], as it is used to compile artifacts that are uploaded to aws lambda.
52-
3. Define which configuration to use for htsget-rs as stated in the configuration section.
57+
3. Define which configuration to use for htsget-rs as stated in the configuration section.
5358

5459
Below is a summary of commands to run in this directory:
5560

@@ -67,6 +72,11 @@ npm install
6772

6873
### Deploy to AWS
6974

75+
> [!IMPORTANT]
76+
> The default deployment is designed to work out of the box. A bucket with a CDK-generated name is created with test
77+
> data from the [`data`][data] directory. All deployment settings can be tweaked using the [`settings.ts`][htsget-settings].
78+
> The only option that must be specified in the `domain`, which determines the domain name to serve htsget-rs at.
79+
7080
CDK should be bootstrapped once, if this hasn't been done before:
7181

7282
```sh
@@ -79,6 +89,10 @@ Then to deploy the stack, run:
7989
npx cdk deploy
8090
```
8191

92+
> [!WARNING]
93+
> By default this deployment will create a public instance of htsget-rs. Anyone will be able to query the server
94+
> without authorizing unless you modify the `HtsgetJwtAuthSettings` settings.
95+
8296
### Testing the endpoint
8397

8498
When the deployment is finished, the htsget endpoint can be tested by querying it. If a JWT authorizer is configured,
@@ -173,3 +187,4 @@ and a [MinIO][minio] deployment.
173187
[rust]: https://www.rust-lang.org/tools/install
174188
[zig]: https://ziglang.org/
175189
[zig-getting-started]: https://ziglang.org/learn/getting-started/
190+
[data]: ../data

‎deploy/bin/settings.ts‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,16 +4,18 @@ import { HtsgetSettings } from "../lib/htsget-lambda-stack";
44
* Settings to use for the htsget deployment.
55
*/
66
export const SETTINGS: HtsgetSettings = {
7-
config: "config/dev_umccr.toml",
7+
config: "config/example_deploy.toml",
8+
// Specify the domain to serve htsget-rs under.
89
domain: "dev.umccr.org",
910
subDomain: "htsget",
10-
s3BucketResources: [
11-
"arn:aws:s3:::org.umccr.demo.sbeacon-data/*",
12-
"arn:aws:s3:::org.umccr.demo.htsget-rs-data/*",
13-
],
14-
lookupHostedZone: true,
11+
s3BucketResources: [],
12+
lookupHostedZone: false,
13+
createS3Bucket: true,
14+
copyTestData: true,
15+
// Override the bucket name.
16+
// bucketName: "bucket",
1517
jwtAuthorizer: {
16-
// Set this to true if you want a public instance.
18+
// Set this to false if you want a private instance.
1719
public: false,
1820
// jwtAudience: ["audience"],
1921
// cogUserPoolId: "user-pool-id",

‎deploy/config/dev_umccr.toml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,11 @@ contact_url = "https://umccr.org/"
2323
documentation_url = "https://github.com/umccr/htsget-rs"
2424
environment = "dev"
2525

26+
[[resolvers]]
27+
regex = '^(org.umccr.dev.htsget-rs-test-data)/(?P<key>.*)$'
28+
substitution_string = '$key'
29+
storage = 'S3'
30+
2631
[[resolvers]]
2732
regex = '^(umccr-10c-data-dev)/(?P<key>.*)$'
2833
substitution_string = '$key'

‎deploy/config/example_deploy.toml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
ticket_server_cors_allow_headers = "All"
2+
ticket_server_cors_allow_origins = []
3+
ticket_server_cors_allow_methods = "All"
4+
ticket_server_cors_allow_credentials = true
5+
ticket_server_cors_max_age = 300
6+
7+
data_server_enabled = false
8+
9+
name = "umccr-htsget-rs"
10+
version = "0.1"
11+
organization_name = "UMCCR"
12+
organization_url = "https://umccr.org/"
13+
contact_url = "https://umccr.org/"
14+
documentation_url = "https://github.com/umccr/htsget-rs"
15+
environment = "dev"
16+
17+
[[resolvers]]
18+
regex = '^(?P<bucket>.*?)/(?P<key>.*)$'
19+
substitution_string = '$key'
20+
storage = 'S3'

0 commit comments

Comments
 (0)