Skip to content

Commit 13a9427

Browse files
lmarinilmarini
authored
Merge pull request #250 from clowder-framework/release/1.18.0
Release/1.18.0
2 parents 4152a29 + 0e7e260 commit 13a9427

32 files changed

+2050
-2194
lines changed

‎CHANGELOG.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,23 @@ All notable changes to this project will be documented in this file.
44
The format is based on [Keep a Changelog](http://keepachangelog.com/)
55
and this project adheres to [Semantic Versioning](http://semver.org/).
66

7+
## 1.18.0 - 2021-07-08
8+
9+
### Added
10+
- Added folder and folder id to API call `GET /api/datasets/:id/files`. [#34](https://github.com/clowder-framework/clowder/issues/34)
11+
- Ability to queue archive / unarchive for full datasets.
12+
- API status endpoint `GET /api/status` will now show what storage type is used and for superadmins will show more
13+
information about the backend storage.
14+
- `GET /api/files/bulkRemove` now returns status of files deleted, not found, no permission, or errors.
15+
16+
### Fixed
17+
- When uploading a file, any extractors marked disabled at the space level would be ignored. [#246](https://github.com/clowder-framework/clowder/issues/246)
18+
- RabbitMQ will not use connection if it does not exist.
19+
- Previews returns 404 if preview is not found `GET /api/previews/:id`.
20+
- Added index for comments, will speed up index creation.
21+
- If using S3 storage in docker, it was not reflected correctly in the docker-compose file.
22+
- Docker image for mongo-init now based on python:3.7-slim to reduce size.
23+
724
## 1.17.0 - 2021-04-29
825

926
### Fixed
@@ -19,6 +36,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
1936
### Changed
2037
- Updated Sphinx dependencies due to security and changes in required packages.
2138

39+
- Updated the three.js libraries for the FBX previewer
40+
2241
## 1.16.0 - 2021-03-31
2342

2443
### Fixed
@@ -86,6 +105,7 @@ script to fix this.
86105
## 1.14.0 - 2021-01-07
87106

88107
### Added
108+
- Added a previewer for FBX files.
89109
- Added a new `/api/reports/metrics/extractors` report for summarizing extractor usage by user. Database administrators
90110
can use `scripts/updates/UpdateUserId.js` to assign user IDs to older extraction event records based on resource ownership
91111
in order to improve the accuracy of the report for older data.

‎CONTRIBUTING.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
# How to contribute
22

33
We would love to get contributions from you! To keep the process manageable we please ask you to follow the workflow
4-
below. By participating in this project, you agree to abide by the [code of conduct](https://clowderframework.org/docs/Clowder-CoC.pdf).
5-
Before your code can be accepted, you will have to sign a [CLA](https://clowderframework.org/docs/Clowder-CLA.pdf)
4+
below. By participating in this project, you agree to abide by the [code of conduct](https://clowderframework.org/pdf/Clowder-CoC.pdf).
5+
Before your code can be accepted, you will have to sign a [CLA](https://clowderframework.org/pdf/Clowder-CLA.pdf)
66
and mail it to [email protected].
77

88
Most of the core development happens on [NCSA Bitbucket][bitbucket]. The `master` and `develop` branches are pushed to
@@ -81,4 +81,4 @@ Developer documentation is available in [Confluence][confluence]. User documenta
8181
[bitbucket]: https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS
8282
[bamboo]: https://opensource.ncsa.illinois.edu/bamboo/browse/CATS
8383
[userdocs]: https://clowderframework.org/docs/
84-
[confluence]: https://opensource.ncsa.illinois.edu/confluence/display/CATS/Home
84+
[confluence]: https://opensource.ncsa.illinois.edu/confluence/display/CATS/Home

‎app/api/Datasets.scala‎

Lines changed: 122 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -1,40 +1,34 @@
11
package api
22

3-
import java.io._
4-
import java.io.{File => JFile}
5-
import java.net.URL
6-
import java.security.{DigestInputStream, MessageDigest}
7-
import java.text.SimpleDateFormat
8-
import java.util.{Calendar, Date}
9-
3+
import _root_.util._
104
import api.Permission.Permission
11-
import java.util.zip._
12-
13-
import javax.inject.{Inject, Singleton}
5+
import controllers.Utils.https
146
import controllers.{Previewers, Utils}
157
import jsonutils.JsonUtil
168
import models._
179
import org.apache.commons.codec.binary.Hex
1810
import org.json.JSONObject
1911
import play.api.Logger
20-
import play.api.Play.{configuration, current, routes}
12+
import play.api.Play.{configuration, current}
2113
import play.api.i18n.Messages
14+
import play.api.libs.Files
2215
import play.api.libs.concurrent.Execution.Implicits._
2316
import play.api.libs.iteratee.Enumerator
24-
import play.api.libs.json._
2517
import play.api.libs.json.Json._
26-
import play.api.mvc.{Action, AnyContent, MultipartFormData, SimpleResult}
27-
import services._
28-
import _root_.util._
29-
import controllers.Utils.https
30-
import org.json.simple.{JSONArray, JSONObject => SimpleJSONObject}
31-
import org.json.simple.parser.JSONParser
32-
import play.api.libs.Files
33-
import play.api.libs.Files.TemporaryFile
18+
import play.api.libs.json._
19+
import play.api.mvc.{AnyContent, MultipartFormData, SimpleResult}
3420
import scalax.file.Path.createTempFile
21+
import services._
3522

36-
import scala.concurrent.{ExecutionContext, Future}
23+
import java.io._
24+
import java.net.URL
25+
import java.security.{DigestInputStream, MessageDigest}
26+
import java.text.SimpleDateFormat
27+
import java.util.zip._
28+
import java.util.{Calendar, Date}
29+
import javax.inject.{Inject, Singleton}
3730
import scala.collection.mutable.{ListBuffer, Map => MutaMap}
31+
import scala.concurrent.{ExecutionContext, Future}
3832

3933
/**
4034
* Dataset API.
@@ -933,6 +927,7 @@ class Datasets @Inject()(
933927
}
934928
}
935929

930+
936931
def getMetadataDefinitions(id: UUID, currentSpace: Option[String]) = PermissionAction(Permission.AddMetadata, Some(ResourceRef(ResourceRef.dataset, id))) { implicit request =>
937932
implicit val user = request.user
938933
datasets.get(id) match {
@@ -1134,47 +1129,84 @@ class Datasets @Inject()(
11341129
private def getFilesWithinFolders(id: UUID, serveradmin: Boolean=false, max: Int = -1): List[JsValue] = {
11351130
val output = new ListBuffer[JsValue]()
11361131
var resultCount = 0
1132+
var current_folder : Option[Folder] = None
11371133
datasets.get(id) match {
11381134
case Some(dataset) => {
1139-
folders.findByParentDatasetId(id).map { folder =>
1135+
folders.findByParentDatasetId(id).foreach { folder =>
1136+
current_folder = Some(folder)
11401137
files.get(folder.files).found.foreach(file => {
11411138
if (max < 0 || resultCount < max) {
1142-
output += jsonFile(file, serveradmin)
1139+
output += jsonFile(file, serveradmin, Some(folder))
11431140
resultCount += 1
11441141
}
11451142
})
1143+
print("done with folder")
11461144
}
11471145
}
11481146
case None => Logger.error(s"Error getting dataset $id")
11491147
}
11501148
output.toList
11511149
}
11521150

1153-
def jsonFile(file: models.File, serverAdmin: Boolean = false): JsValue = {
1154-
val defaultMap = Map(
1155-
"id" -> file.id.toString,
1156-
"filename" -> file.filename,
1157-
"contentType" -> file.contentType,
1158-
"date-created" -> file.uploadDate.toString(),
1159-
"size" -> file.length.toString)
11601151

1161-
// Only include filepath if using DiskByte storage and user is serverAdmin
1162-
val jsonMap = file.loader match {
1163-
case "services.filesystem.DiskByteStorageService" => {
1164-
if (serverAdmin)
1165-
Map(
1166-
"id" -> file.id.toString,
1167-
"filename" -> file.filename,
1168-
"filepath" -> file.loader_id,
1169-
"contentType" -> file.contentType,
1170-
"date-created" -> file.uploadDate.toString(),
1171-
"size" -> file.length.toString)
1172-
else
1173-
defaultMap
1152+
def jsonFile(file: models.File, serverAdmin: Boolean = false, folder : Option[Folder] = None): JsValue = {
1153+
folder match {
1154+
case Some(f) => {
1155+
val folderMap : JsValue = Json.obj("id"->f.id, "name"->f.name)
1156+
val defaultMap : JsValue = Json.obj(
1157+
"id" -> file.id.toString,
1158+
"filename" -> file.filename,
1159+
"contentType" -> file.contentType,
1160+
"date-created" -> file.uploadDate.toString(),
1161+
"folders"->folderMap,
1162+
"size" -> file.length.toString)
1163+
1164+
// Only include filepath if using DiskByte storage and user is serverAdmin
1165+
val jsonMap = file.loader match {
1166+
case "services.filesystem.DiskByteStorageService" => {
1167+
if (serverAdmin)
1168+
Json.obj(
1169+
"id" -> file.id.toString,
1170+
"filename" -> file.filename,
1171+
"filepath" -> file.loader_id,
1172+
"contentType" -> file.contentType,
1173+
"date-created" -> file.uploadDate.toString(),
1174+
"folders"->folderMap,
1175+
"size" -> file.length.toString)
1176+
else
1177+
defaultMap
1178+
}
1179+
case _ => defaultMap
1180+
}
1181+
toJson(jsonMap)
11741182
}
1175-
case _ => defaultMap
1183+
case None => {
1184+
val defaultMap = Map(
1185+
"id" -> file.id.toString,
1186+
"filename" -> file.filename,
1187+
"contentType" -> file.contentType,
1188+
"date-created" -> file.uploadDate.toString(),
1189+
"size" -> file.length.toString)
1190+
1191+
// Only include filepath if using DiskByte storage and user is serverAdmin
1192+
val jsonMap = file.loader match {
1193+
case "services.filesystem.DiskByteStorageService" => {
1194+
if (serverAdmin)
1195+
Map(
1196+
"id" -> file.id.toString,
1197+
"filename" -> file.filename,
1198+
"filepath" -> file.loader_id,
1199+
"contentType" -> file.contentType,
1200+
"date-created" -> file.uploadDate.toString(),
1201+
"size" -> file.length.toString)
1202+
else
1203+
defaultMap
1204+
}
1205+
case _ => defaultMap
1206+
}
1207+
toJson(jsonMap)
1208+
}
11761209
}
1177-
toJson(jsonMap)
11781210
}
11791211

11801212
//Update Dataset Information code starts
@@ -3076,6 +3108,51 @@ class Datasets @Inject()(
30763108
}
30773109

30783110
}
3111+
3112+
/**
3113+
* Recursively submit requests to archive the contents of the given dataset
3114+
* @param id dataset to archive
3115+
* @return
3116+
*/
3117+
def queueArchival(id: UUID) = PermissionAction(Permission.ArchiveDataset, Some(ResourceRef(ResourceRef.dataset, id)))(parse.json) { implicit request =>
3118+
val reqParams = (request.body \ "parameters").asOpt[JsObject].getOrElse(JsObject(Seq.empty[(String, JsValue)]))
3119+
val parameters = reqParams + FileService.ARCHIVE_PARAMETER
3120+
datasets.get(id) match {
3121+
case Some(ds) => {
3122+
val host = Utils.baseUrl(request)
3123+
datasets.recursiveArchive(ds, host, parameters, request.apiKey, request.user)
3124+
sinkService.logDatasetArchiveEvent(ds, request.user)
3125+
Ok(toJson(Map("status" -> "success")))
3126+
}
3127+
case None => {
3128+
Logger.error("Error getting dataset " + id)
3129+
NotFound
3130+
}
3131+
}
3132+
}
3133+
3134+
3135+
/**
3136+
* Recursively submit requests to unarchive the contents of the given dataset
3137+
* @param id dataset to unarchive
3138+
* @return
3139+
*/
3140+
def queueUnarchival(id: UUID) = PermissionAction(Permission.ArchiveDataset, Some(ResourceRef(ResourceRef.dataset, id)))(parse.json) { implicit request =>
3141+
val reqParams = (request.body \ "parameters").asOpt[JsObject].getOrElse(JsObject(Seq.empty[(String, JsValue)]))
3142+
val parameters = reqParams + FileService.UNARCHIVE_PARAMETER
3143+
datasets.get(id) match {
3144+
case Some(ds) => {
3145+
val host = Utils.baseUrl(request)
3146+
datasets.recursiveArchive(ds, host, parameters, request.apiKey, request.user)
3147+
sinkService.logDatasetUnarchiveEvent(ds, request.user)
3148+
Ok(toJson(Map("status" -> "success")))
3149+
}
3150+
case None => {
3151+
Logger.error("Error getting dataset " + id)
3152+
NotFound
3153+
}
3154+
}
3155+
}
30793156
}
30803157

30813158
object ActivityFound extends Exception {}

‎app/api/Files.scala‎

Lines changed: 36 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1611,24 +1611,49 @@ class Files @Inject()(
16111611
}
16121612

16131613
def bulkDeleteFiles() = PrivateServerAction (parse.json) {implicit request=>
1614+
var filesToCheck : ListBuffer[String] = ListBuffer.empty[String]
1615+
var filesNotExist : ListBuffer[String] = ListBuffer.empty[String]
1616+
var filesDeleted : ListBuffer[String] = ListBuffer.empty[String]
1617+
var filesErrorDeleted: ListBuffer[String] = ListBuffer.empty[String]
16141618
request.user match {
16151619
case Some(user) => {
16161620
val fileIds = request.body.\("fileIds").asOpt[List[String]].getOrElse(List.empty[String])
1621+
filesToCheck.appendAll(fileIds)
16171622
if (fileIds.isEmpty){
16181623
BadRequest("No file ids supplied")
16191624
} else {
16201625
var resourceRefList: ListBuffer[ResourceRef] = ListBuffer.empty[ResourceRef]
16211626
for (fileId <- fileIds) {
16221627
if (UUID.isValid(fileId)) {
1623-
val current_resource_ref = ResourceRef(ResourceRef.file, UUID(fileId))
1624-
resourceRefList += current_resource_ref
1628+
files.get(UUID(fileId)) match {
1629+
case Some(currentFile) => {
1630+
val current_resource_ref = ResourceRef(ResourceRef.file, UUID(fileId))
1631+
resourceRefList += current_resource_ref
1632+
}
1633+
case None => {
1634+
filesToCheck -= fileId
1635+
filesNotExist += fileId
1636+
}
1637+
}
1638+
} else {
1639+
filesToCheck -= fileId
1640+
filesNotExist += fileId
16251641
}
16261642
}
16271643
val filesIdsCanDelete = Permission.checkPermissions(request.user, Permission.DeleteFile, resourceRefList.toList).approved.map(_.id)
16281644
for (id <- filesIdsCanDelete) {
1629-
files.removeFile(id,Utils.baseUrl(request), request.apiKey, request.user)
1645+
val id_removed = files.removeFile(id,Utils.baseUrl(request), request.apiKey, request.user)
1646+
if (id_removed == true) {
1647+
filesToCheck -= id.stringify
1648+
filesDeleted += id.stringify
1649+
} else {
1650+
filesToCheck -= id.stringify
1651+
filesErrorDeleted += id.stringify
1652+
}
16301653
}
1631-
Ok(toJson(Map("status" -> "success")))
1654+
Ok(toJson(Map("deleted"->filesDeleted.toList, "not found"->filesNotExist.toList,
1655+
"error deleting"->filesErrorDeleted.toList,"no permission"->filesToCheck.toList)))
1656+
// Ok(toJson(Map("status" -> "success")))
16321657
}
16331658
}
16341659
case None => {
@@ -1885,7 +1910,7 @@ class Files @Inject()(
18851910
}
18861911
case None => {
18871912
Logger.error("Error getting file " + id)
1888-
InternalServerError
1913+
NotFound
18891914
}
18901915
}
18911916
}
@@ -1896,11 +1921,14 @@ class Files @Inject()(
18961921
case Some(file) => {
18971922
files.setStatus(id, FileStatus.PROCESSED)
18981923
sinkService.logFileUnarchiveEvent(file, user)
1924+
1925+
// Only increment download date, to avoid immediate auto-archive
1926+
files.incrementDownloads(id, user, true)
18991927
Ok(toJson(Map("status" -> "success")))
19001928
}
19011929
case None => {
19021930
Logger.error("Error getting file " + id)
1903-
InternalServerError
1931+
NotFound
19041932
}
19051933
}
19061934
}
@@ -1916,7 +1944,7 @@ class Files @Inject()(
19161944
}
19171945
case None => {
19181946
Logger.error("Error getting file " + id)
1919-
InternalServerError
1947+
NotFound
19201948
}
19211949
}
19221950
}
@@ -1932,7 +1960,7 @@ class Files @Inject()(
19321960
}
19331961
case None => {
19341962
Logger.error("Error getting file " + id)
1935-
InternalServerError
1963+
NotFound
19361964
}
19371965
}
19381966
}

‎app/api/Permissions.scala‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ object Permission extends Enumeration {
2929
DeleteDataset,
3030
EditDataset,
3131
PublicDataset,
32+
ArchiveDataset,
3233
AddResourceToDataset,
3334
RemoveResourceFromDataset,
3435
ExecuteOnDataset,
@@ -120,7 +121,8 @@ object Permission extends Enumeration {
120121
CreateSensor, DeleteSensor, AddGeoStream, DeleteGeoStream, AddDatapoints,
121122
CreateRelation, ViewRelation, DeleteRelation,
122123
CreateVocabulary, DeleteVocabulary, EditVocabulary,
123-
CreateVocabularyTerm, DeleteVocabularyTerm, EditVocabularyTerm
124+
CreateVocabularyTerm, DeleteVocabularyTerm, EditVocabularyTerm,
125+
ArchiveFile, ArchiveDataset
124126
)
125127

126128
lazy val files: FileService = DI.injector.getInstance(classOf[FileService])

0 commit comments

Comments
 (0)