485 public datasets visible when not logged in (#832)

* adding new file

* modal allows selecting public

* changing front end to allow changing to PUBLIC

* changing to public now works

* adding new routes for public datasets

* routers for public files and datasets added
download is available, though button will not work on page when added

* running codegen for new files

* running codegen for new files

* does not work right now

* small fixes, a few typos

* working now

* new direction if public

* adding PublicDataset

will probably need to refactor content from Dataset so that they can both import the same things, with 'publicView' as boolean that will determine what is shown

* fixing indents

* adding public folders

* error on folders and details

* folders do not show
details not working

* moving layout to new component
this should make having a public dataset easier

* public dataset view uses DatasetLayout now

* console logs, should remove later

* cannot read properties of undefined?

* cannot read properties of undefined?

* fix package lock

* error gone but with hardcoded limit,not sure why

* works now with limit

* this seemed to get rid of the error

* public view works, shows only public datasets
changes coming to PublicDataset need some refactoring of Dataset component

* fixing import, working on publicDataset component now

* files table has option for public now, will use different route

* publicactions menu added
removing tabs we don't use for public datasets

* new tab panel components

* removing unused components

* adding public file

* adding actions and states for public folders
gradually switching public views to use different states than private ones

* fixing package lock

* fixing wrong method called for public dataset details
we see file, some features removed that a public file should not have

* public file - no editing or modifying user metadata

* index now has authenticated and public
using booleans instead of statuses here for now, might change later

* adding public and authenticated to index
these will be used for searching later

* adding new routes for public search
from public pages, the search will only return stuff that is public

* adding tests

* sort of works

* using a status field rather than 2 booleans since both cannot be true

* tests pass for public datasets,
need tests for authenticated added

* adding public and authenticated to the user clause

* public no longer private router, but problems with layout

* errors on files, using new publicLayout for later

* public file actions
not tested yet,

* dataset works, need to fix file

* public file added
tabs removed for public dataset

* public file page shows up, has error

* something breaks on page, might be related to visualizations

* errors seem to be on visualization dab

* routes had dependency that was not needed

* public file menu boolean, not all options should appear
adding public visualizations, we need that and metadata eventually

* adding public visualizations

* public visualization view

* adding to reducers index, fixing names

* will need to use another file actions menu

* removing some unused imports
adding public metadata

* deleting the metadata public routers
just adding method to the public files and datasets

* public metadata

* adding metadata

* fixing public dataset details

* no more error on public file

* user metadata tab should work now
checks if public view, uses other route if public

* adding method to public_datasets.js for metadata
listener metadata tab

* public layout removes links we do not use in this view

* change explore to public

* some fake data will be public
change to DatasetIn

* fix public metadata extraction tab for file
missing import

* using public layout in public dataset

* download public file version

* route to download public dataset

* HACK ADDED - probably not a good idea
if the user is public, an email is returned for
[email protected]

* register or login for public page

* adding authenticated status

* return public datasets

* have some authenticated datasets as well

* fixing auth for public datasets

* checking for public dataset files

* changing auth for public or authenticated things

* adding filestatus for easier checks

* changing status of files?

* adding has public access method

* new method

* adding public as well as authenticated

* can now view files if they are public
but it looks like the viewer can do things they should not be able to

* check public access method added, fixed to work for files

* viewer cannot run extraction on file

* formatting

* running codegen

* formatting

* fixing package lock, ws wrong

* formatting

* remove extraction history tab on public view
fixing update of status on dataset patch

* fix file summary on public file

* reverting change

* fixing file status change, looks like that broke
removing unnecessary file status field
just using downloadResource

* return public access

* fixing public visualization

* fixed the broken visualization on public files

* formatting

* does not work

* think this fixed it

* putting downloadPublicResource back in, errors not fixed

* putting downloadPublicResource back in, errors not fixed

* adding publicView option to the various visualizations. this means the right URL will be used if we are seeing this in public view. otherwise the raw bytes do not download

* fix package lock

* removing console logs

* removing console logs

* removing console logs

* catch case if authorization is None and no public access for file

* role was not working for dataset, fixed there but still not showing on tab

* does not break on front end, still needs clean up

* back end fixes after merge, some authorizations were not checked correctly.
files did not switch to public or authenticated properly

* fix file tab extractor

* file metadata from listeners now displays for public files

* button back for add metadata dataset

* ran codegen

* public metadata definitions

* metadata should work now for public when not signed in

* fixed text visualization, endpoint for public_files had a typo

* fixing more typoes

* formatting

* codegen

* removing search for now

* increment should be false for getting public files

* adding public check to dataset files

* check public or authenticated in dataset routes

* don't use public route

* removing console logs

* fake data should populate both PUBLIC and AUTHENTICATED datasets

* fixing download

* fixing type on public/files
dataset download fixed
download file and file version fixed
public elasticsearch added to router

* removing unused methods

* remove divider

* codegen

* put divider back in
public dataset card allows dataset download when public

* removed divider

* can download public file from file menu now

* remove unused method
black formatting

* black formatting

* black formatting

---------

Co-authored-by: Chen Wang <[email protected]>

Author: tcnichol

backend/app/deps/authorization_deps.py

@@ -5,14 +5,41 @@
 from app.keycloak_auth import get_current_username
 from app.models.authorization import RoleType, AuthorizationDB
 from app.models.datasets import DatasetDB, DatasetStatus
-from app.models.files import FileDB
-from app.models.groups import GroupDB
+from app.models.files import FileOut, FileDB, FileStatus
+from app.models.groups import GroupOut, GroupDB
 from app.models.metadata import MetadataDB
 from app.models.pyobjectid import PyObjectId
 from app.routers.authentication import get_admin
 from app.routers.authentication import get_admin_mode
 
 
+async def check_public_access(
+    resource_id: str,
+    resource_type: str,
+    role: RoleType,
+    current_user=Depends(get_current_username),
+) -> bool:
+    has_public_access = False
+    if role == RoleType.VIEWER:
+        if resource_type == "dataset":
+            if (
+                dataset := await DatasetDB.get(PydanticObjectId(resource_id))
+            ) is not None:
+                if (
+                    dataset.status == DatasetStatus.PUBLIC.name
+                    or dataset.status == DatasetStatus.AUTHENTICATED.name
+                ):
+                    has_public_access = True
+        elif resource_type == "file":
+            if (file := await FileDB.get(PydanticObjectId(resource_id))) is not None:
+                if (
+                    file.status == FileStatus.PUBLIC.name
+                    or file.status == FileStatus.AUTHENTICATED.name
+                ):
+                    has_public_access = True
+    return has_public_access
+
+
 async def get_role(
     dataset_id: str,
     current_user=Depends(get_current_username),
@@ -31,6 +58,11 @@ async def get_role(
             AuthorizationDB.user_ids == current_user,
         ),
     )
+    public_access = await check_public_access(
+        dataset_id, "dataset", RoleType.VIEWER, current_user
+    )
+    if authorization is None and public_access:
+        return RoleType.VIEWER
     return authorization.role
 
 
@@ -55,15 +87,11 @@ async def get_role_by_file(
             if (
                 dataset := await DatasetDB.get(PydanticObjectId(file.dataset_id))
             ) is not None:
-                if dataset.status == DatasetStatus.AUTHENTICATED.name:
-                    auth_dict = {
-                        "creator": dataset.author.email,
-                        "dataset_id": file.dataset_id,
-                        "user_ids": [current_user],
-                        "role": RoleType.VIEWER,
-                    }
-                    authenticated_auth = AuthorizationDB(**auth_dict)
-                    return authenticated_auth
+                if (
+                    dataset.status == DatasetStatus.AUTHENTICATED.name
+                    or dataset.status == DatasetStatus.PUBLIC.name
+                ):
+                    return RoleType.VIEWER
                 else:
                     raise HTTPException(
                         status_code=403,
@@ -199,6 +227,7 @@ async def __call__(
             ) is not None:
                 if (
                     current_dataset.status == DatasetStatus.AUTHENTICATED.name
+                    or current_dataset.status == DatasetStatus.PUBLIC.name
                     and self.role == "viewer"
                 ):
                     return True
@@ -249,7 +278,15 @@ async def __call__(
                     detail=f"User `{current_user} does not have `{self.role}` permission on file {file_id}",
                 )
             else:
-                raise HTTPException(status_code=404, detail=f"File {file_id} not found")
+                if (
+                    file.status == FileStatus.PUBLIC.name
+                    or file.status == FileStatus.AUTHENTICATED.name
+                ) and self.role == RoleType.VIEWER:
+                    return True
+                else:
+                    raise HTTPException(
+                        status_code=404, detail=f"File {file_id} not found"
+                    )
 
 
 class MetadataAuthorization:

backend/app/main.py

@@ -30,22 +30,27 @@
 from app.models.users import UserDB, UserAPIKeyDB, ListenerAPIKeyDB
 from app.models.visualization_config import VisualizationConfigDB
 from app.models.visualization_data import VisualizationDataDB
-from app.routers import folders, groups, status
+from app.routers import folders, groups, public_folders, status
 from app.routers import (
     users,
     authorization,
     metadata,
+    public_metadata,
     files,
+    public_files,
     metadata_files,
     datasets,
+    public_datasets,
     metadata_datasets,
     authentication,
     keycloak,
     elasticsearch,
+    public_elasticsearch,
     listeners,
     feeds,
     jobs,
     visualization,
+    public_visualization,
     thumbnails,
 )
 
@@ -113,12 +118,22 @@
     tags=["metadata"],
     dependencies=[Depends(get_current_username)],
 )
+api_router.include_router(
+    public_metadata.router,
+    prefix="/public_metadata",
+    tags=["public_metadata"],
+)
 api_router.include_router(
     files.router,
     prefix="/files",
     tags=["files"],
     dependencies=[Depends(get_current_username)],
 )
+api_router.include_router(
+    public_files.router,
+    prefix="/public_files",
+    tags=["public_files"],
+)
 api_router.include_router(
     metadata_files.router,
     prefix="/files",
@@ -132,14 +147,27 @@
     dependencies=[Depends(get_current_username)],
 )
 api_router.include_router(
-    metadata_datasets.router, prefix="/datasets", tags=["metadata"]
+    public_datasets.router,
+    prefix="/public_datasets",
+    tags=["public_datasets"],
+)
+api_router.include_router(
+    metadata_datasets.router,
+    prefix="/datasets",
+    tags=["metadata"],
+    dependencies=[Depends(get_current_username)],
 )
 api_router.include_router(
     folders.router,
     prefix="/folders",
     tags=["folders"],
     dependencies=[Depends(get_current_username)],
 )
+api_router.include_router(
+    public_folders.router,
+    prefix="/public_folders",
+    tags=["public_folders"],
+)
 api_router.include_router(
     listeners.router,
     prefix="/listeners",
@@ -164,6 +192,11 @@
     tags=["elasticsearch"],
     dependencies=[Depends(get_current_username)],
 )
+api_router.include_router(
+    public_elasticsearch.router,
+    prefix="/public_elasticsearch",
+    tags=["public_elasticsearch"],
+)
 api_router.include_router(
     feeds.router,
     prefix="/feeds",
@@ -182,6 +215,11 @@
     tags=["visualizations"],
     dependencies=[Depends(get_current_username)],
 )
+api_router.include_router(
+    public_visualization.router,
+    prefix="/public_visualizations",
+    tags=["public_visualizations"],
+)
 api_router.include_router(
     thumbnails.router,
     prefix="/thumbnails",

backend/app/models/datasets.py

@@ -27,6 +27,7 @@ class DatasetStatus(AutoName):
 class DatasetBase(BaseModel):
     name: str = "N/A"
     description: Optional[str] = None
+    status: str = DatasetStatus.PRIVATE.name
 
 
 class DatasetIn(DatasetBase):
@@ -65,7 +66,7 @@ class DatasetDBViewList(View, DatasetBase):
     modified: datetime = Field(default_factory=datetime.utcnow)
     auth: List[AuthorizationDB]
     thumbnail_id: Optional[PydanticObjectId] = None
-    status: Optional[str]
+    status: str = DatasetStatus.PRIVATE.name
 
     class Settings:
         source = DatasetDB

backend/app/models/files.py

@@ -1,7 +1,7 @@
 from datetime import datetime
 from enum import Enum
 from typing import Optional, List
-
+from enum import Enum, auto
 from beanie import Document, View, PydanticObjectId
 from pydantic import Field, BaseModel
 
@@ -10,6 +10,19 @@
 from app.models.users import UserOut
 
 
+class AutoName(Enum):
+    def _generate_next_value_(name, start, count, last_values):
+        return name
+
+
+class FileStatus(AutoName):
+    PRIVATE = auto()
+    PUBLIC = auto()
+    AUTHENTICATED = auto()
+    DEFAULT = auto()
+    TRIAL = auto()
+
+
 class StorageType(str, Enum):
     """Depending on the StorageType,the file may need different properties such as local path or URL.
     Also, some StorageTypes do not support versioning or anonymous sharing."""
@@ -47,6 +60,7 @@ class Settings:
 
 class FileBase(BaseModel):
     name: str = "N/A"
+    status: str = FileStatus.PRIVATE.name
 
 
 class FileIn(FileBase):

backend/app/models/search.py

@@ -46,3 +46,4 @@ class ElasticsearchEntry(BaseModel):
     bytes: Optional[int]
     # metadata fields
     metadata: Optional[List[dict]] = []
+    status: Optional[str]

backend/app/routers/authorization.py

@@ -1,3 +1,5 @@
+import datetime
+
 from beanie import PydanticObjectId
 from beanie.operators import Or, In
 from bson import ObjectId
@@ -92,7 +94,10 @@ async def get_dataset_role(
         if (
             current_dataset := await DatasetDB.get(PydanticObjectId(dataset_id))
         ) is not None:
-            if current_dataset.status == DatasetStatus.AUTHENTICATED.name:
+            if (
+                current_dataset.status == DatasetStatus.AUTHENTICATED.name
+                or current_dataset.status == DatasetStatus.PUBLIC.name
+            ):
                 public_authorization_in = {
                     "dataset_id": PydanticObjectId(dataset_id),
                     "role": RoleType.VIEWER,