Skip to content

Commit 29b826b

Browse files
landreevlandreev
authored
Merge pull request #11940 from GlobalDataverseCommunityConsortium/Fix-dataset-storageDriver_api
Fixes for #11716
2 parents ec346f3 + c9d8924 commit 29b826b

File tree

11 files changed

+113
-43
lines changed

11 files changed

+113
-43
lines changed

‎doc/release-notes/11695-11940-change-api-get-storage-driver.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
## Get Dataset/Dataverse Storage Driver API
2+
3+
### Changed Json response - breaking change!
4+
5+
The API for getting the Storage Driver info has been changed/extended.
6+
/api/datasets/{identifier}/storageDriver
7+
/api/admin/dataverse/{dataverse-alias}/storageDriver
8+
Rather than returning just the name/id of the driver (with the key "message"), the api call now returns a JSONObject with the driver's "name", "type" and "label", and booleans indicating whether the driver has "directUpload", "directDownload", and/or "uploadOutOfBand" enabled.
9+
10+
This change also affects the /api/admin/dataverse/{dataverse-alias}/storageDriver api call. In addition, this call now supports an optional ?getEffective=true to find the effective storageDriver (the driver that will be used for new datasets in the collection)
11+
12+
See also [the guides](https://dataverse-guide--11664.org.readthedocs.build/en/11664/api/native-api.html#configure-a-dataset-to-store-all-new-files-in-a-specific-file-store), #11695, and #11664.

‎doc/release-notes/11695-change-api-get-storage-driver.md‎

Lines changed: 0 additions & 12 deletions
This file was deleted.

‎doc/sphinx-guides/source/admin/dataverses-datasets.rst‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -56,17 +56,19 @@ To direct new files (uploaded when datasets are created or edited) for all datas
5656

5757
(Note that for ``dataverse.files.store1.label=MyLabel``, you should pass ``MyLabel``.)
5858

59-
The current driver can be seen using::
59+
A store assigned directly to a collection can be seen using::
6060

6161
curl -H "X-Dataverse-key: $API_TOKEN" http://$SERVER/api/admin/dataverse/$dataverse-alias/storageDriver
6262

63-
Or to recurse the chain of parents to find the effective storageDriver::
63+
This may be null. To get the effective storageDriver for a collection, which may be inherited from a parent collection or be the installation default, you can use::
6464

6565
curl -H "X-Dataverse-key: $API_TOKEN" http://$SERVER/api/admin/dataverse/$dataverse-alias/storageDriver?getEffective=true
66+
67+
This will never be null.
6668

67-
(Note that for ``dataverse.files.store1.label=MyLabel``, ``store1`` will be returned.)
69+
(Note that for ``dataverse.files.store1.label=MyLabel``, the JSON response will include "name":"store1" and "label":"MyLabel".)
6870

69-
and can be reset to the default store with::
71+
To delete a store assigned directly to a collection (so that the colllection's effective store is inherted from it's parent or is the global default), use::
7072

7173
curl -H "X-Dataverse-key: $API_TOKEN" -X DELETE http://$SERVER/api/admin/dataverse/$dataverse-alias/storageDriver
7274
@@ -261,15 +263,15 @@ To identify invalid data values in specific datasets (if, for example, an attemp
261263
Configure a Dataset to Store All New Files in a Specific File Store
262264
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
263265

264-
Configure a dataset to use a specific file store (this API can only be used by a superuser) ::
266+
Configure an individual dataset to use a specific file store (this API can only be used by a superuser) ::
265267
266268
curl -H "X-Dataverse-key: $API_TOKEN" -X PUT -d $storageDriverLabel http://$SERVER/api/datasets/$dataset-id/storageDriver
267269
268-
The current driver can be seen using::
270+
The effective store can be seen using::
269271

270272
curl http://$SERVER/api/datasets/$dataset-id/storageDriver
271273

272-
It can be reset to the default store as follows (only a superuser can do this) ::
274+
To remove an assigned store, and allow the dataset to inherit the store from it's parent collection, use the following (only a superuser can do this) ::
273275

274276
curl -H "X-Dataverse-key: $API_TOKEN" -X DELETE http://$SERVER/api/datasets/$dataset-id/storageDriver
275277

‎doc/sphinx-guides/source/api/changelog.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,14 @@ v6.9
1414
- The way to set per-format size limits for tabular ingest has changed. JSON input is now used. See :ref:`:TabularIngestSizeLimit`.
1515
- In the past, the settings API would accept any key and value. This is no longer the case because validation has been added. See :ref:`settings_put_single`, for example.
1616
- For GET /api/notifications/all the JSON response has changed breaking the backward compatibility of the API.
17+
- For GET /api/admin/dataverse/{dataverse-alias}/storageDriver and /api/datasets/{identifier}/storageDriver the driver name is no longer returned in data.message. Instead, it is returned as data.name (along with other information about the storageDriver).
1718

1819
v6.8
1920
----
2021

2122
- For POST /api/files/{id}/metadata passing an empty string ("description":"") or array ("categories":[]) will no longer be ignored. Empty fields will now clear out the values in the file's metadata. To ignore the fields simply do not include them in the JSON string.
2223
- For PUT /api/datasets/{id}/editMetadata the query parameter "sourceInternalVersionNumber" has been removed and replaced with "sourceLastUpdateTime" to verify that the data being edited hasn't been modified and isn't stale.
2324
- For GET /api/dataverses/$dataverse-alias/links the JSON response has changed breaking the backward compatibility of the API.
24-
- For GET /api/admin/dataverse/{dataverse-alias}/storageDriver and /api/datasets/{identifier}/storageDriver the driver name is no longer returned in data.message. This value is now returned in data.name.
2525
- For PUT /api/dataverses/$dataverse-alias/inputLevels custom input levels that had been previously set will no longer be deleted. To delete input levels send an empty list (deletes all), then send the new/modified list.
2626
- For GET /api/externalTools and /api/externalTools/{id} the responses are now formatted as JSON (previously the toolParameters and allowedApiCalls were a JSON object and array (respectively) that were serialized as JSON strings) and any configured "requirements" are included.
2727

‎src/main/java/edu/harvard/iq/dataverse/api/Admin.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2275,9 +2275,9 @@ public Response getStorageDriver(@Context ContainerRequestContext crc, @PathPara
22752275
}
22762276

22772277
if (getEffective != null && getEffective) {
2278-
return ok(JsonPrinter.jsonStorageDriver(dataverse.getEffectiveStorageDriverId(), null));
2278+
return ok(JsonPrinter.jsonStorageDriver(dataverse.getEffectiveStorageDriverId()));
22792279
} else {
2280-
return ok(JsonPrinter.jsonStorageDriver(dataverse.getStorageDriverId(), null));
2280+
return ok(JsonPrinter.jsonStorageDriver(dataverse.getStorageDriverId()));
22812281
}
22822282
}
22832283

‎src/main/java/edu/harvard/iq/dataverse/api/Datasets.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3662,7 +3662,7 @@ public Response getFileStore(@Context ContainerRequestContext crc, @PathParam("i
36623662
return error(Response.Status.NOT_FOUND, "No such dataset");
36633663
}
36643664

3665-
return ok(JsonPrinter.jsonStorageDriver(dataset.getEffectiveStorageDriverId(), dataset));
3665+
return ok(JsonPrinter.jsonStorageDriver(dataset.getEffectiveStorageDriverId()));
36663666
}
36673667

36683668
@PUT

‎src/main/java/edu/harvard/iq/dataverse/dataaccess/StorageIO.java‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -53,10 +53,11 @@ public abstract class StorageIO<T extends DvObject> {
5353

5454
static final String INGEST_SIZE_LIMIT = "ingestsizelimit";
5555
static final String PUBLIC = "public";
56-
static final String TYPE = "type";
57-
static final String UPLOAD_REDIRECT = "upload-redirect";
58-
static final String UPLOAD_OUT_OF_BAND = "upload-out-of-band";
59-
protected static final String DOWNLOAD_REDIRECT = "download-redirect";
56+
public static final String TYPE = "type";
57+
public static final String UPLOAD_REDIRECT = "upload-redirect";
58+
public static final String UPLOAD_OUT_OF_BAND = "upload-out-of-band";
59+
public static final String DOWNLOAD_REDIRECT = "download-redirect";
60+
public static final String LABEL = "label";
6061

6162

6263
public StorageIO() {
@@ -675,10 +676,11 @@ public boolean isDataverseAccessible() {
675676
return true;
676677
}
677678

678-
protected static String getConfigParamForDriver(String driverId, String parameterName) {
679+
public static String getConfigParamForDriver(String driverId, String parameterName) {
679680
return getConfigParamForDriver(driverId, parameterName, null);
680681
}
681-
protected static String getConfigParamForDriver(String driverId, String parameterName, String defaultValue) {
682+
683+
public static String getConfigParamForDriver(String driverId, String parameterName, String defaultValue) {
682684
return System.getProperty("dataverse.files." + driverId + "." + parameterName, defaultValue);
683685
}
684686

‎src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java‎

Lines changed: 7 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
import edu.harvard.iq.dataverse.authorization.users.User;
1818
import edu.harvard.iq.dataverse.branding.BrandingUtil;
1919
import edu.harvard.iq.dataverse.dataaccess.DataAccess;
20+
import edu.harvard.iq.dataverse.dataaccess.StorageIO;
2021
import edu.harvard.iq.dataverse.dataset.DatasetType;
2122
import edu.harvard.iq.dataverse.dataset.DatasetUtil;
2223
import edu.harvard.iq.dataverse.datasetversionsummaries.*;
@@ -1670,19 +1671,14 @@ public static JsonArrayBuilder jsonTemplateInstructions(Map<String, String> temp
16701671
return jsonArrayBuilder;
16711672
}
16721673

1673-
public static JsonObjectBuilder jsonStorageDriver(String storageDriverId, Dataset dataset) {
1674+
public static JsonObjectBuilder jsonStorageDriver(String storageDriverId) {
16741675
JsonObjectBuilder jsonObjectBuilder = new NullSafeJsonBuilder();
16751676
jsonObjectBuilder.add("name", storageDriverId);
1676-
jsonObjectBuilder.add("type", DataAccess.getDriverType(storageDriverId));
1677-
jsonObjectBuilder.add("label", DataAccess.getStorageDriverLabelFor(storageDriverId));
1678-
if (dataset != null) {
1679-
jsonObjectBuilder.add("directUpload", DataAccess.uploadToDatasetAllowed(dataset, storageDriverId));
1680-
try {
1681-
jsonObjectBuilder.add("directDownload", DataAccess.getStorageIO(dataset).downloadRedirectEnabled());
1682-
} catch (IOException ex) {
1683-
logger.fine("Failed to get Storage IO for dataset " + ex.getMessage());
1684-
}
1685-
}
1677+
jsonObjectBuilder.add("type", StorageIO.getConfigParamForDriver(storageDriverId, StorageIO.TYPE));
1678+
jsonObjectBuilder.add("label", StorageIO.getConfigParamForDriver(storageDriverId, StorageIO.LABEL));
1679+
jsonObjectBuilder.add("directUpload", Boolean.parseBoolean(StorageIO.getConfigParamForDriver(storageDriverId, StorageIO.UPLOAD_REDIRECT)));
1680+
jsonObjectBuilder.add("directDownload", Boolean.parseBoolean(StorageIO.getConfigParamForDriver(storageDriverId, StorageIO.DOWNLOAD_REDIRECT)));
1681+
jsonObjectBuilder.add("uploadOutOfBand", Boolean.parseBoolean(StorageIO.getConfigParamForDriver(storageDriverId, StorageIO.UPLOAD_OUT_OF_BAND)));
16861682

16871683
return jsonObjectBuilder;
16881684
}

‎src/test/java/edu/harvard/iq/dataverse/api/DataversesIT.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2740,8 +2740,8 @@ public void testGetStorageDriver() {
27402740
.body("data.name", CoreMatchers.notNullValue())
27412741
.body("data.type", CoreMatchers.notNullValue())
27422742
.body("data.label", CoreMatchers.notNullValue())
2743-
.body("data.directUpload", CoreMatchers.nullValue())
2744-
.body("data.directDownload", CoreMatchers.nullValue())
2743+
.body("data.directUpload", CoreMatchers.notNullValue())
2744+
.body("data.directDownload", CoreMatchers.notNullValue())
27452745
.statusCode(200);
27462746

27472747
// Root without default is undefined

‎src/test/java/edu/harvard/iq/dataverse/api/S3AccessIT.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -156,7 +156,7 @@ public void testNonDirectUpload() {
156156
updatedStorageDriver.then().assertThat()
157157
.body("data.type", CoreMatchers.notNullValue())
158158
.body("data.label", CoreMatchers.notNullValue())
159-
.body("data.directUpload", CoreMatchers.nullValue())
159+
.body("data.directUpload", CoreMatchers.notNullValue())
160160
.statusCode(200);
161161

162162
Response createDatasetResponse = UtilIT.createRandomDatasetViaNativeApi(dataverseAlias, apiToken);

0 commit comments

Comments
 (0)