Skip to content

refactor: c4gh config #267

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Sep 24, 2024
Merged

refactor: c4gh config #267

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c8d624c
refactor(config)!: remove object type and specify keys directly
mmalenic Sep 23, 2024
415d687
refactor!: make storage config more explicit
mmalenic Sep 23, 2024
713c007
refactor(config): slight adjustment to reduce conditionally compiled …
mmalenic Sep 23, 2024
4839016
style: add newline
mmalenic Sep 24, 2024
1210175
fix: build errors with conditional flags
mmalenic Sep 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

16 changes: 8 additions & 8 deletions deploy/config/dev_umccr.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,39 +26,39 @@ environment = "dev"
[[resolvers]]
regex = '^(org.umccr.dev.htsget-rs-test-data)/(?P<key>.*)$'
substitution_string = '$key'
storage = 'S3'
storage.type = 'S3'

[[resolvers]]
regex = '^(umccr-10c-data-dev)/(?P<key>.*)$'
substitution_string = '$key'
storage = 'S3'
storage.type = 'S3'

[[resolvers]]
regex = '^(umccr-10f-data-dev)/(?P<key>.*)$'
substitution_string = '$key'
storage = 'S3'
storage.type = 'S3'

[[resolvers]]
regex = '^(umccr-10g-data-dev)/(?P<key>.*)$'
substitution_string = '$key'
storage = 'S3'
storage.type = 'S3'

[[resolvers]]
regex = '^(umccr-agha-test-dev)/(?P<key>.*)$'
substitution_string = '$key'
storage = 'S3'
storage.type = 'S3'

[[resolvers]]
regex = '^(umccr-research-dev)/(?P<key>.*)$'
substitution_string = '$key'
storage = 'S3'
storage.type = 'S3'

[[resolvers]]
regex = '^(umccr-primary-data-dev)/(?P<key>.*)$'
substitution_string = '$key'
storage = 'S3'
storage.type = 'S3'

[[resolvers]]
regex = '^(umccr-validation-prod)/(?P<key>.*)$'
substitution_string = '$key'
storage = 'S3'
storage.type = 'S3'
2 changes: 1 addition & 1 deletion deploy/config/example_deploy.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,4 @@ environment = "dev"
[[resolvers]]
regex = '^(?P<bucket>.*?)/(?P<key>.*)$'
substitution_string = '$key'
storage = 'S3'
storage.type = 'S3'
6 changes: 3 additions & 3 deletions deploy/config/prod_umccr.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,14 @@ environment = "prod"
[[resolvers]]
regex = '^(umccr-research-dev)/(?P<key>.*)$'
substitution_string = '$key'
storage = 'S3'
storage.type = 'S3'

[[resolvers]]
regex = '^(umccr-validation-prod)/(?P<key>.*)$'
substitution_string = '$key'
storage = 'S3'
storage.type = 'S3'

[[resolvers]]
regex = '^(umccr-primary-data-prod)/(?P<key>.*)$'
substitution_string = '$key'
storage = 'S3'
storage.type = 'S3'
4 changes: 2 additions & 2 deletions deploy/config/public_umccr.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@ environment = 'public'
[[resolvers]]
regex = '^(org.umccr.demo.sbeacon-data)/CINECA_UK1/(?P<key>.*)$'
substitution_string = 'CINECA_UK1/$key'
storage = 'S3'
storage.type = 'S3'

[[resolvers]]
regex = '^(org.umccr.demo.htsget-rs-data)/(?P<type>bam|cram|vcf|bcf|crypt4gh|mixed)/(?P<key>.*)$'
substitution_string = '$type/$key'
storage = 'S3'
storage.type = 'S3'
50 changes: 28 additions & 22 deletions htsget-config/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,33 +151,34 @@ For more information about regex options see the [regex crate](https://docs.rs/r
Each resolver also maps to a certain storage backend. This storage backend can be used to set query IDs which are served from local storage, from S3-style bucket storage, or from HTTP URLs.
To set the storage backend for a resolver, add a `[resolvers.storage]` table. Some storage backends require feature flags to be set when compiling htsget-rs.

To use `LocalStorage`, set `storage = 'Local'`. This will derive the values for the fields below from the `data_server` config:
To use `LocalStorage`, set `type = 'Local'` under `[resolvers.storage]`, and specify any additional options from below:

| Option | Description | When `storage = 'Local'` | Type | Default |
|---------------------|-------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------|------------------------------|--------------------|
| `scheme` | The scheme present on URL tickets. | Derived from `data_server_key` and `data_server_cert`. If no key and cert are present, then uses `Http`, otherwise uses `Https`. | Either `'Http'` or `'Https'` | `'Http'` |
| `authority` | The authority present on URL tickets. This should likely match the `data_server_addr`. | Same as `data_server_addr`. | URL authority | `'127.0.0.1:8081'` |
| `local_path` | The local filesystem path which the data server uses to respond to tickets. This should likely match the `data_server_local_path`. | Same as `data_server_local_path`. | Filesystem path | `'./'` |
| `path_prefix` | The path prefix which the URL tickets will have. This should likely match the `data_server_serve_at` path. | Same as `data_server_serve_at`. | URL path | `''` |
| Option | Description | Type | Default |
|--------------------------|-------------------------------------------------------------------------------------------------------------------------------------|------------------------------|--------------------|
| `scheme` | The scheme present on URL tickets. | Either `'Http'` or `'Https'` | `'Http'` |
| `authority` | The authority present on URL tickets. This should likely match the `data_server_addr`. | URL authority | `'127.0.0.1:8081'` |
| `local_path` | The local filesystem path which the data server uses to respond to tickets. This should likely match the `data_server_local_path`. | Filesystem path | `'./'` |
| `path_prefix` | The path prefix which the URL tickets will have. This should likely match the `data_server_serve_at` path. | URL path | `''` |
| `use_data_server_config` | Whether to use the data server config to fill in the above values. This overrides any other options specified from this table. | Boolean | `false` |

To use `S3Storage`, build htsget-rs with the `s3-storage` feature enabled, and set `storage = 'S3'`. This will derive the value for `bucket` from the `regex` component of the `resolvers`:
To use `S3Storage`, build htsget-rs with the `s3-storage` feature enabled, set `type = 'S3'` under `[resolvers.storage]`, and specify any additional options from below:

| Option | Description | When `storage = 'S3'` | Type | Default |
|--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------|---------|----------------------------------------|
| `bucket` | The AWS S3 bucket where resources can be retrieved from. | Derived from the `resolvers` `regex` property. This uses the first capture group in the `regex` as the `bucket`. | String | `''` |
| `endpoint` | A custom endpoint to override the default S3 service address. This is useful for using S3 locally or with storage backends such as MinIO. See [MinIO](#minio). | Not set, uses regular AWS S3 services. | String | Not set, uses regular AWS S3 services. |
| `path_style` | The S3 path style to request from the storage backend. If `true`, "path style" is used, e.g. `host.com/bucket/object.bam`, otherwise `bucket.host.com/object` style is used. | `false` | Boolean | `false` |
| Option | Description | Type | Default |
|--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|---------------------------------------------------------------------------------------------------------------------------|
| `bucket` | The AWS S3 bucket where resources can be retrieved from. | String | Derived from the `resolvers` `regex` property if empty. This uses the first capture group in the `regex` as the `bucket`. |
| `endpoint` | A custom endpoint to override the default S3 service address. This is useful for using S3 locally or with storage backends such as MinIO. See [MinIO](#minio). | String | Not set, uses regular AWS S3 services. |
| `path_style` | The S3 path style to request from the storage backend. If `true`, "path style" is used, e.g. `host.com/bucket/object.bam`, otherwise `bucket.host.com/object` style is used. | Boolean | `false` |

`UrlStorage` is another storage backend which can be used to serve data from a remote HTTP URL. When using this storage backend, htsget-rs will fetch data from a `url` which is set in the config. It will also forward any headers received with the initial query, which is useful for authentication.
To use `UrlStorage`, build htsget-rs with the `url-storage` feature enabled, and set the following options under `[resolvers.storage]`:
To use `UrlStorage`, build htsget-rs with the `url-storage` feature enabled, set `type = 'Url'` under `[resolvers.storage]`, and specify any additional options from below:

| Option | Description | Type | Default |
|--------------------------------------|------------------------------------------------------------------------------------------------------------------------------|--------------------------|-----------------------------------------------------------------------------------------------------------------|
| <span id="url">`url`</span> | The URL to fetch data from. | HTTP URL | `"https://127.0.0.1:8081/"` |
| <span id="url">`response_url`</span> | The URL to return to the client for fetching tickets. | HTTP URL | `"https://127.0.0.1:8081/"` |
| `forward_headers` | When constructing the URL tickets, copy HTTP headers received in the initial query. | Boolean | `true` |
| `header_blacklist` | List of headers that should not be forwarded | Array of headers | `[]` |
| `tls` | Additionally enables client authentication, or sets non-native root certificates for TLS. See [TLS](#tls) for more details. | TOML table | TLS is always allowed, however the default performs no client authentication and uses native root certificates. |
| Option | Description | Type | Default |
|--------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|--------------------------|-----------------------------------------------------------------------------------------------------------------|
| <span id="url">`url`</span> | The URL to fetch data from. | HTTP URL | `"https://127.0.0.1:8081/"` |
| <span id="url">`response_url`</span> | The URL to return to the client for fetching tickets. | HTTP URL | `"https://127.0.0.1:8081/"` |
| `forward_headers` | When constructing the URL tickets, copy HTTP headers received in the initial query. | Boolean | `true` |
| `header_blacklist` | List of headers that should not be forwarded. | Array of headers | `[]` |
| `tls` | Additionally enables client authentication, or sets non-native root certificates for TLS. See [TLS](#tls) for more details. | TOML table | TLS is always allowed, however the default performs no client authentication and uses native root certificates. |

When using `UrlStorage`, the following requests will be made to the `url`.
* `GET` request to fetch only the headers of the data file (e.g. `GET /data.bam`, with `Range: bytes=0-<end_of_bam_header>`).
Expand All @@ -192,7 +193,9 @@ For example, a `resolvers` value of:
[[resolvers]]
regex = '^(example_bucket)/(?P<key>.*)$'
substitution_string = '$key'
storage = 'S3'
[resolvers.storage]
type = 'S3'
# Uses the first capture group in the regex as the bucket.
```
Will use "example_bucket" as the S3 bucket if that resolver matches, because this is the first capture group in the `regex`.
Note, to use this feature, at least one capture group must be defined in the `regex`.
Expand All @@ -206,6 +209,7 @@ regex = '.*'
substitution_string = '$0'

[resolvers.storage]
type = 'Local'
scheme = 'Http'
authority = '127.0.0.1:8081'
local_path = './'
Expand All @@ -220,6 +224,7 @@ regex = '.*'
substitution_string = '$0'

[resolvers.storage]
type = 'S3'
bucket = 'bucket'
```

Expand All @@ -231,6 +236,7 @@ regex = ".*"
substitution_string = "$0"

[resolvers.storage]
type = 'Url'
url = "http://localhost:8080"
response_url = "https://example.com"
forward_headers = true
Expand Down
5 changes: 4 additions & 1 deletion htsget-config/examples/config-files/c4gh.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,7 @@ regex = ".*"
substitution_string = "$0"

[resolvers.storage]
object_type = { private_key = "data/c4gh/keys/bob.sec", recipient_public_key = "data/c4gh/keys/alice.pub" } # pragma: allowlist secret
type = 'Local'

private_key = "data/c4gh/keys/bob.sec" # pragma: allowlist secret
recipient_public_key = "data/c4gh/keys/alice.pub"
3 changes: 2 additions & 1 deletion htsget-config/examples/config-files/s3_storage.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,9 @@ data_server_enabled = false
[[resolvers]]
regex = '^(bucket)/(?P<key>.*)$'
substitution_string = '$key'
storage = 'S3'
storage.type = 'S3'

# Or, set the bucket manually
#[resolvers.storage]
#type = 'S3'
#bucket = 'bucket'
4 changes: 4 additions & 0 deletions htsget-config/examples/config-files/tls_data_server.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,7 @@ data_server_tls.key = "key.pem"
[[resolvers]]
regex = ".*"
substitution_string = "$0"

[resolvers.storage]
type = 'Local'
use_data_server_config = true
1 change: 1 addition & 0 deletions htsget-config/examples/config-files/tls_ticket_server.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,5 @@ regex = ".*"
substitution_string = "$0"

[resolvers.storage]
type = 'S3'
bucket = "bucket"
1 change: 1 addition & 0 deletions htsget-config/examples/config-files/url_storage.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ regex = ".*"
substitution_string = "$0"

[resolvers.storage]
type = 'Url'
url = "http://127.0.0.1:8081"
response_url = "https://127.0.0.1:8081"
forward_headers = true
Expand Down
6 changes: 4 additions & 2 deletions htsget-config/src/config/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -774,13 +774,15 @@ pub(crate) mod tests {
data_server_serve_at = "/path"

[[resolvers]]
storage = "Local"
[resolvers.storage]
type = "Local"
use_data_server_config = true
"#,
|config| {
assert_eq!(config.resolvers.len(), 1);

assert!(matches!(config.resolvers.first().unwrap().storage(),
Storage::Local { local_storage } if local_storage.local_path() == "path" && local_storage.scheme() == Http && local_storage.authority() == &Authority::from_static("127.0.0.1:8080") && local_storage.path_prefix() == "/path"));
Storage::Local(local_storage) if local_storage.local_path() == "path" && local_storage.scheme() == Http && local_storage.authority() == &Authority::from_static("127.0.0.1:8080") && local_storage.path_prefix() == "/path"));
},
);
}
Expand Down
Loading