refactor: rename type to backend and clarify experimental feature flag

mmalenic · mmalenic · commit aae1b2c6d242 · 2024-09-30T12:08:23.000+10:00
diff --git a/deploy/config/dev_umccr.toml b/deploy/config/dev_umccr.toml
@@ -26,39 +26,39 @@ environment = "dev"
 [[resolvers]]
 regex = '^(org.umccr.dev.htsget-rs-test-data)/(?P<key>.*)$'
 substitution_string = '$key'
-storage.type = 'S3'
+storage.backend = 'S3'
 
 [[resolvers]]
 regex = '^(umccr-10c-data-dev)/(?P<key>.*)$'
 substitution_string = '$key'
-storage.type = 'S3'
+storage.backend = 'S3'
 
 [[resolvers]]
 regex = '^(umccr-10f-data-dev)/(?P<key>.*)$'
 substitution_string = '$key'
-storage.type = 'S3'
+storage.backend = 'S3'
 
 [[resolvers]]
 regex = '^(umccr-10g-data-dev)/(?P<key>.*)$'
 substitution_string = '$key'
-storage.type = 'S3'
+storage.backend = 'S3'
 
 [[resolvers]]
 regex = '^(umccr-agha-test-dev)/(?P<key>.*)$'
 substitution_string = '$key'
-storage.type = 'S3'
+storage.backend = 'S3'
 
 [[resolvers]]
 regex = '^(umccr-research-dev)/(?P<key>.*)$'
 substitution_string = '$key'
-storage.type = 'S3'
+storage.backend = 'S3'
 
 [[resolvers]]
 regex = '^(umccr-primary-data-dev)/(?P<key>.*)$'
 substitution_string = '$key'
-storage.type = 'S3'
+storage.backend = 'S3'
 
 [[resolvers]]
 regex = '^(umccr-validation-prod)/(?P<key>.*)$'
 substitution_string = '$key'
-storage.type = 'S3'
+storage.backend = 'S3'
diff --git a/deploy/config/example_deploy.toml b/deploy/config/example_deploy.toml
@@ -17,4 +17,4 @@ environment = "dev"
 [[resolvers]]
 regex = '^(?P<bucket>.*?)/(?P<key>.*)$'
 substitution_string = '$key'
-storage.type = 'S3'
+storage.backend = 'S3'
diff --git a/deploy/config/prod_umccr.toml b/deploy/config/prod_umccr.toml
@@ -22,14 +22,14 @@ environment = "prod"
 [[resolvers]]
 regex = '^(umccr-research-dev)/(?P<key>.*)$'
 substitution_string = '$key'
-storage.type = 'S3'
+storage.backend = 'S3'
 
 [[resolvers]]
 regex = '^(umccr-validation-prod)/(?P<key>.*)$'
 substitution_string = '$key'
-storage.type = 'S3'
+storage.backend = 'S3'
 
 [[resolvers]]
 regex = '^(umccr-primary-data-prod)/(?P<key>.*)$'
 substitution_string = '$key'
-storage.type = 'S3'
+storage.backend = 'S3'
diff --git a/deploy/config/public_umccr.toml b/deploy/config/public_umccr.toml
@@ -11,9 +11,9 @@ environment = 'public'
 [[resolvers]]
 regex = '^(org.umccr.demo.sbeacon-data)/CINECA_UK1/(?P<key>.*)$'
 substitution_string = 'CINECA_UK1/$key'
-storage.type = 'S3'
+storage.backend = 'S3'
 
 [[resolvers]]
 regex = '^(org.umccr.demo.htsget-rs-data)/(?P<type>bam|cram|vcf|bcf|crypt4gh|mixed)/(?P<key>.*)$'
 substitution_string = '$type/$key'
-storage.type = 'S3'
+storage.backend = 'S3'
diff --git a/htsget-actix/README.md b/htsget-actix/README.md
@@ -50,7 +50,7 @@ are exposed in the public API.
 This crate has the following features:
 * `s3-storage`: used to enable `S3Storage` functionality.
 * `url-storage`: used to enable `UrlStorage` functionality.
-* `experimental`: used to enable experimental features like `C4GHStorage`.
+* `experimental`: used to enable experimental features that aren't necessarily part of the htsget spec, such as Crypt4GH support through `C4GHStorage`.
 
 ## Benchmarks
 Benchmarks for this crate written using [Criterion.rs][criterion-rs], and aim to compare the performance of this crate with the
diff --git a/htsget-axum/README.md b/htsget-axum/README.md
@@ -171,7 +171,7 @@ htsget-rs. It also contains the data block server which fetches data from a `Loc
 This crate has the following features:
 * `s3-storage`: used to enable `S3Storage` functionality.
 * `url-storage`: used to enable `UrlStorage` functionality.
-* `experimental`: used to enable experimental features like `C4GHStorage`.
+* `experimental`: used to enable experimental features that aren't necessarily part of the htsget spec, such as Crypt4GH support through `C4GHStorage`.
 
 ## License
 
diff --git a/htsget-config/README.md b/htsget-config/README.md
@@ -151,7 +151,7 @@ For more information about regex options see the [regex crate](https://docs.rs/r
 Each resolver also maps to a certain storage backend. This storage backend can be used to set query IDs which are served from local storage, from S3-style bucket storage, or from HTTP URLs.
 To set the storage backend for a resolver, add a `[resolvers.storage]` table. Some storage backends require feature flags to be set when compiling htsget-rs.
 
-To use `LocalStorage`, set `type = 'Local'` under `[resolvers.storage]`, and specify any additional options from below:
+To use `LocalStorage`, set `backend = 'Local'` under `[resolvers.storage]`, and specify any additional options from below:
 
 | Option                   | Description                                                                                                                         | Type                         | Default            |
 |--------------------------|-------------------------------------------------------------------------------------------------------------------------------------|------------------------------|--------------------|
@@ -161,7 +161,7 @@ To use `LocalStorage`, set `type = 'Local'` under `[resolvers.storage]`, and spe
 | `path_prefix`            | The path prefix which the URL tickets will have. This should likely match the `data_server_serve_at` path.                          | URL path                     | `''`               |
 | `use_data_server_config` | Whether to use the data server config to fill in the above values. This overrides any other options specified from this table.      | Boolean                      | `false`            |
 
-To use `S3Storage`, build htsget-rs with the `s3-storage` feature enabled, set `type = 'S3'` under `[resolvers.storage]`, and specify any additional options from below:
+To use `S3Storage`, build htsget-rs with the `s3-storage` feature enabled, set `backend = 'S3'` under `[resolvers.storage]`, and specify any additional options from below:
 
 | Option       | Description                                                                                                                                                                   | Type    | Default                                                                                                                   |
 |--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|---------------------------------------------------------------------------------------------------------------------------|
@@ -170,7 +170,7 @@ To use `S3Storage`, build htsget-rs with the `s3-storage` feature enabled, set `
 | `path_style` | The S3 path style to request from the storage backend. If `true`, "path style" is used, e.g. `host.com/bucket/object.bam`, otherwise `bucket.host.com/object` style is used.  | Boolean | `false`                                                                                                                   |
 
 `UrlStorage` is another storage backend which can be used to serve data from a remote HTTP URL. When using this storage backend, htsget-rs will fetch data from a `url` which is set in the config. It will also forward any headers received with the initial query, which is useful for authentication. 
-To use `UrlStorage`, build htsget-rs with the `url-storage` feature enabled, set `type = 'Url'` under `[resolvers.storage]`, and specify any additional options from below:
+To use `UrlStorage`, build htsget-rs with the `url-storage` feature enabled, set `backend = 'Url'` under `[resolvers.storage]`, and specify any additional options from below:
 
 | Option                               | Description                                                                                                                 | Type                     | Default                                                                                                         |
 |--------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|--------------------------|-----------------------------------------------------------------------------------------------------------------|
@@ -195,7 +195,7 @@ regex = '^(example_bucket)/(?P<key>.*)$'
 substitution_string = '$key'
 
 [resolvers.storage]
-type = 'S3'
+backend = 'S3'
 # Uses the first capture group in the regex as the bucket.
 ```
 Will use "example_bucket" as the S3 bucket if that resolver matches, because this is the first capture group in the `regex`.
@@ -210,7 +210,7 @@ regex = '.*'
 substitution_string = '$0'
 
 [resolvers.storage]
-type = 'Local'
+backend = 'Local'
 scheme = 'Http'
 authority = '127.0.0.1:8081'
 local_path = './'
@@ -225,7 +225,7 @@ regex = '.*'
 substitution_string = '$0'
 
 [resolvers.storage]
-type = 'S3'
+backend = 'S3'
 bucket = 'bucket'
 ```
 
@@ -237,7 +237,7 @@ regex = ".*"
 substitution_string = "$0"
 
 [resolvers.storage]
-type = 'Url'
+backend = 'Url'
 url = "http://localhost:8080"
 response_url = "https://example.com"
 forward_headers = true
@@ -276,7 +276,7 @@ regex = '.*'
 substitution_string = '$0'
 
 [resolvers.storage]
-type = 'S3'
+backend = 'S3'
 bucket = 'bucket'
 
 [resolvers.allow_guard]
@@ -486,7 +486,7 @@ regex = '.*'
 substitution_string = '$0'
 
 [resolvers.storage]
-type = 'S3'
+backend = 'S3'
 bucket = 'bucket'
 endpoint = 'http://127.0.0.1:9000'
 path_style = true
@@ -523,7 +523,7 @@ regex = '.*'
 substitution_string = '$0'
 
 [resolvers.storage]
-type = 'Local'
+backend = 'Local'
 private_key = 'data/c4gh/keys/bob.sec' # pragma: allowlist secret
 recipient_public_key = 'data/c4gh/keys/alice.pub'
 ```
@@ -544,7 +544,7 @@ regex, and changing it by using a substitution string.
 This crate has the following features:
 * `s3-storage`: used to enable `S3Storage` functionality.
 * `url-storage`: used to enable `UrlStorage` functionality.
-* `experimental`: used to enable experimental features like `C4GHStorage`.
+* `experimental`: used to enable experimental features that aren't necessarily part of the htsget spec, such as Crypt4GH support through `C4GHStorage`.
 
 ## License
 
diff --git a/htsget-config/examples/config-files/c4gh.toml b/htsget-config/examples/config-files/c4gh.toml
@@ -9,7 +9,7 @@ regex = ".*"
 substitution_string = "$0"
 
 [resolvers.storage]
-type = 'Local'
+backend = 'Local'
 
 private_key = "data/c4gh/keys/bob.sec" # pragma: allowlist secret
 recipient_public_key = "data/c4gh/keys/alice.pub"
diff --git a/htsget-config/examples/config-files/s3_storage.toml b/htsget-config/examples/config-files/s3_storage.toml
@@ -11,9 +11,9 @@ data_server_enabled = false
 [[resolvers]]
 regex = '^(bucket)/(?P<key>.*)$'
 substitution_string = '$key'
-storage.type = 'S3'
+storage.backend = 'S3'
 
 # Or, set the bucket manually
 #[resolvers.storage]
-#type = 'S3'
+#backend = 'S3'
 #bucket = 'bucket'
diff --git a/htsget-config/examples/config-files/tls_data_server.toml b/htsget-config/examples/config-files/tls_data_server.toml
@@ -12,5 +12,5 @@ regex = ".*"
 substitution_string = "$0"
 
 [resolvers.storage]
-type = 'Local'
+backend = 'Local'
 use_data_server_config = true
diff --git a/htsget-config/examples/config-files/tls_ticket_server.toml b/htsget-config/examples/config-files/tls_ticket_server.toml
@@ -12,5 +12,5 @@ regex = ".*"
 substitution_string = "$0"
 
 [resolvers.storage]
-type = 'S3'
+backend = 'S3'
 bucket = "bucket"
diff --git a/htsget-config/examples/config-files/url_storage.toml b/htsget-config/examples/config-files/url_storage.toml
@@ -16,7 +16,7 @@ regex = ".*"
 substitution_string = "$0"
 
 [resolvers.storage]
-type = 'Url'
+backend = 'Url'
 url = "http://127.0.0.1:8081"
 response_url = "https://127.0.0.1:8081"
 forward_headers = true
diff --git a/htsget-config/src/config/mod.rs b/htsget-config/src/config/mod.rs
@@ -775,7 +775,7 @@ pub(crate) mod tests {
 
     [[resolvers]]
     [resolvers.storage]
-    type = "Local"
+    backend = "Local"
     use_data_server_config = true
     "#,
       |config| {
diff --git a/htsget-config/src/resolver.rs b/htsget-config/src/resolver.rs
@@ -678,7 +678,7 @@ mod tests {
       vec![(
         "HTSGET_RESOLVERS",
         "[{ regex=regex, substitution_string=substitution_string, \
-        storage={ type=S3, bucket=bucket }, \
+        storage={ backend=S3, bucket=bucket }, \
         allow_guard={ allow_reference_names=[chr1], allow_fields=[QNAME], allow_tags=[RG], \
         allow_formats=[BAM], allow_classes=[body], allow_interval_start=100, \
         allow_interval_end=1000 } }]",
diff --git a/htsget-config/src/storage/local.rs b/htsget-config/src/storage/local.rs
@@ -126,7 +126,7 @@ mod tests {
         regex = "regex"
 
         [resolvers.storage]
-        type = "Local"
+        backend = "Local"
         local_path = "path"
         scheme = "HTTPS"
         path_prefix = "path"
diff --git a/htsget-config/src/storage/mod.rs b/htsget-config/src/storage/mod.rs
@@ -30,7 +30,7 @@ impl ResolvedId {
 
 /// Specify the storage backend to use as config values.
 #[derive(Serialize, Deserialize, Debug, Clone)]
-#[serde(tag = "type")]
+#[serde(tag = "backend")]
 #[non_exhaustive]
 pub enum Storage {
   #[serde(alias = "local", alias = "LOCAL")]
@@ -61,7 +61,7 @@ pub(crate) mod tests {
       r#"
       [[resolvers]]
       [resolvers.storage]
-      type = "Local"
+      backend = "Local"
       regex = "regex"
       "#,
       |config| {
@@ -79,7 +79,7 @@ pub(crate) mod tests {
     test_config_from_env(
       vec![(
         "HTSGET_RESOLVERS",
-        "[{storage={ type=Local, use_data_server_config=true}}]",
+        "[{storage={ backend=Local, use_data_server_config=true}}]",
       )],
       |config| {
         assert!(matches!(
@@ -97,7 +97,7 @@ pub(crate) mod tests {
       r#"
       [[resolvers]]
       [resolvers.storage]
-      type = "S3"
+      backend = "S3"
       regex = "regex"
       "#,
       |config| {
@@ -114,7 +114,7 @@ pub(crate) mod tests {
   #[test]
   fn config_storage_tagged_s3_env() {
     test_config_from_env(
-      vec![("HTSGET_RESOLVERS", "[{storage={ type=S3 }}]")],
+      vec![("HTSGET_RESOLVERS", "[{storage={ backend=S3 }}]")],
       |config| {
         assert!(matches!(
           config.resolvers().first().unwrap().storage(),
diff --git a/htsget-config/src/storage/object/c4gh.rs b/htsget-config/src/storage/object/c4gh.rs
@@ -114,7 +114,7 @@ mod tests {
 
   #[test]
   fn config_local_storage_c4gh() {
-    test_c4gh_storage_config(r#"type = "Local""#, |config| {
+    test_c4gh_storage_config(r#"backend = "Local""#, |config| {
       assert!(matches!(
             config.resolvers().first().unwrap().storage(),
             Storage::Local(local_storage) if local_storage.object_type().keys().is_some()
@@ -127,7 +127,7 @@ mod tests {
   fn config_s3_storage_c4gh() {
     test_c4gh_storage_config(
       r#"
-        type = "S3"
+        backend = "S3"
         bucket = "bucket"
         "#,
       |config| {
@@ -144,7 +144,7 @@ mod tests {
   fn config_url_storage_c4gh() {
     test_c4gh_storage_config(
       r#"
-        type = "Url"
+        backend = "Url"
         url = "https://example.com/"
         response_url = "https://example.com/"
         forward_headers = false
diff --git a/htsget-config/src/storage/s3.rs b/htsget-config/src/storage/s3.rs
@@ -61,7 +61,7 @@ mod tests {
         regex = "regex"
 
         [resolvers.storage]
-        type = "S3"
+        backend = "S3"
         bucket = "bucket"
         "#,
       |config| {
diff --git a/htsget-config/src/storage/url.rs b/htsget-config/src/storage/url.rs
@@ -261,7 +261,7 @@ mod tests {
         regex = "regex"
 
         [resolvers.storage]
-        type = "Url"
+        backend = "Url"
         url = "https://example.com/"
         response_url = "https://example.com/"
         forward_headers = false
diff --git a/htsget-http/README.md b/htsget-http/README.md
@@ -38,7 +38,7 @@ These functions take query and endpoint information, and process it using [htsge
 This crate has the following features:
 * `s3-storage`: used to enable `S3Storage` functionality.
 * `url-storage`: used to enable `UrlStorage` functionality.
-* `experimental`: used to enable experimental features like `C4GHStorage`.
+* `experimental`: used to enable experimental features that aren't necessarily part of the htsget spec, such as Crypt4GH support through `C4GHStorage`.
 
 [warp]: https://github.com/seanmonstar/warp
 [htsget-search]: ../htsget-search
diff --git a/htsget-lambda/README.md b/htsget-lambda/README.md
@@ -46,7 +46,7 @@ library code, and it instead uses `htsget-axum`. Please use that crate for funct
 This crate has the following features:
 * `s3-storage`: used to enable `S3Storage` functionality.
 * `url-storage`: used to enable `UrlStorage` functionality.
-* `experimental`: used to enable experimental features like `C4GHStorage`.
+* `experimental`: used to enable experimental features that aren't necessarily part of the htsget spec, such as Crypt4GH support through `C4GHStorage`.
 
 ## License
 
diff --git a/htsget-search/README.md b/htsget-search/README.md
@@ -59,7 +59,7 @@ used to process requests.
 This crate has the following features:
 * `s3-storage`: used to enable `S3Storage` functionality.
 * `url-storage`: used to enable `UrlStorage` functionality.
-* `experimental`: used to enable experimental features like `C4GHStorage`.
+* `experimental`: used to enable experimental features that aren't necessarily part of the htsget spec, such as Crypt4GH support through `C4GHStorage`.
 
 ## Minimising Byte Ranges
 
diff --git a/htsget-storage/README.md b/htsget-storage/README.md
@@ -49,7 +49,7 @@ and [url] modules implement the `Storage` functionality.
 This crate has the following features:
 * `s3-storage`: used to enable `S3Storage` functionality.
 * `url-storage`: used to enable `UrlStorage` functionality.
-* `experimental`: used to enable experimental features like `C4GHStorage`.
+* `experimental`: used to enable experimental features that aren't necessarily part of the htsget spec, such as Crypt4GH support through `C4GHStorage`.
 
 [local]: src/local.rs
 [s3]: src/s3.rs
diff --git a/htsget-test/README.md b/htsget-test/README.md
