feat: add functions for project-dataset operations (#82)

* feat(experiment): add JuliaHub._project_datasets

* upload WIP

* end-to-end wip

* wip docs & cleanup

* cleanup

* get rid of separate type

* add basic Dataset constructor tests

* make sure Dataset only throws JuliaHubError

* in mocking, .tags is sometimes Any[]

* only silently capture JuliaHubErrors in datasets()

* add a return

* don't rely on string rep of a parametric type

* bad merge

* fix

* fix current tests

* fixes & tests for auth

* add unit tests for listing APIs

* basic upload_project_dataset unit tests

* add live tests for projects

* add option to run tests

* format

* docs: use [sources]

* docstrings

* auth docs

* changelog

* make JET happy?

* 🤦

* fix project_dataset print

* Update src/authentication.jl

* don't allow missing in authenticate()

* fix doctests

---------

Co-authored-by: Sebastian Pfitzner <[email protected]>

Author: mortenpi

CHANGELOG.md

@@ -6,6 +6,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 ### Added
 
+* With JuliaHub instances v6.9 and above, JuliaHub.jl now supports project-dataset operations. ([#15], [#82])
+
+  This includes the following new features:
+
+  - Authentication objects can now be associated with projects.
+    If the `JULIAHUB_PROJECT_UUID` environment variable is set, JuliaHub.jl will pick it up automatically..
+  - The `project_dataset` and `project_datasets` functions allow for listing datasets attached to a project.
+  - `upload_project_dataset` can be used to upload a new version of a dataset.
+
 * All the public API names are now correctly marked `public` in Julia 1.11 and above. ([#83])
 
 ### Changed
@@ -134,6 +143,7 @@ Initial package release.
 [#12]: https://github.com/JuliaComputing/JuliaHub.jl/issues/12
 [#13]: https://github.com/JuliaComputing/JuliaHub.jl/issues/13
 [#14]: https://github.com/JuliaComputing/JuliaHub.jl/issues/14
+[#15]: https://github.com/JuliaComputing/JuliaHub.jl/issues/15
 [#18]: https://github.com/JuliaComputing/JuliaHub.jl/issues/18
 [#28]: https://github.com/JuliaComputing/JuliaHub.jl/issues/28
 [#31]: https://github.com/JuliaComputing/JuliaHub.jl/issues/31
@@ -150,6 +160,7 @@ Initial package release.
 [#53]: https://github.com/JuliaComputing/JuliaHub.jl/issues/53
 [#58]: https://github.com/JuliaComputing/JuliaHub.jl/issues/58
 [#74]: https://github.com/JuliaComputing/JuliaHub.jl/issues/74
+[#82]: https://github.com/JuliaComputing/JuliaHub.jl/issues/82
 [#83]: https://github.com/JuliaComputing/JuliaHub.jl/issues/83
 [#84]: https://github.com/JuliaComputing/JuliaHub.jl/issues/84
 [#86]: https://github.com/JuliaComputing/JuliaHub.jl/issues/86

docs/Project.toml

@@ -10,7 +10,7 @@ URIs = "5c2747f8-b7ea-4ff2-ba2e-563bfd36b1d4"
 
 [compat]
 Changelog = "1"
-Documenter = "1"
+Documenter = "1.9"
 DocumenterMermaid = "0.1"
 
 [sources]

docs/make.jl

@@ -9,6 +9,10 @@ import TimeZones
 ENV["TZ"] = "America/New_York"
 JuliaHub._LOCAL_TZ[] = TimeZones.localzone()
 
+# Patching of the API responses. Also sets JuliaHub.__AUTH__.
+include("../test/mocking.jl")
+
+# We don't want doctests to interfere with each other
 DocMeta.setdocmeta!(
     JuliaHub, :DocTestSetup,
     quote
@@ -18,8 +22,29 @@ DocMeta.setdocmeta!(
     recursive=true,
 )
 
-# Patching of the API responses. Also sets JuliaHub.__AUTH__.
-include("../test/mocking.jl")
+# For project-related APIs, we need a different authentication object.
+# So we set up small setup and teardown functions here too.
+const DEFAULT_PROJECT_AUTH = mockauth(
+    URIs.URI("https://juliahub.com");
+    api_version=v"0.2.0",
+    project_id=UUIDs.UUID("cd6c9ee3-d15f-414f-a762-7e1d3faed835"),
+)
+projectauth_setup!() = JuliaHub.__AUTH__[] = DEFAULT_PROJECT_AUTH
+projectauth_teardown!() = JuliaHub.__AUTH__[] = DEFAULT_GLOBAL_MOCK_AUTH
+
+function env_setup!(; project=false)
+    ENV["JULIA_PKG_SERVER"] = "juliahub.com"
+    if project
+        ENV["JULIAHUB_PROJECT_UUID"] = "b1a95ba8-43e6-4eb6-b280-3c5cbe0fa0b9"
+    end
+    return nothing
+end
+function env_teardown!()
+    delete!(ENV, "JULIA_PKG_SERVER")
+    delete!(ENV, "JULIAHUB_PROJECT_UUID")
+    return nothing
+end
+
 # The following setup function is reused in both at-setup blocks, but also in
 # doctestsetup.
 function setup_job_results_file!()
@@ -65,6 +90,7 @@ const PAGES_REFERENCE = [
     "reference/job-submission.md",
     "reference/jobs.md",
     "reference/datasets.md",
+    "reference/projects.md",
     "reference/exceptions.md",
 ]
 Mocking.apply(mocking_patch) do

docs/src/guides/authentication.md

@@ -43,6 +43,8 @@ When working with JuliaHub.jl in JuliaHub cloud environment, such as in JuliaHub
 The `JULIA_PKG_SERVER` should always be correctly set up, and the `auth.toml` file will also be present and up-to-date.
 This means that [`authenticate`](@ref) should automatically be able to authenticate your session, and you should never be prompted for interactive authentication.
 
+This also extends to JuliaHub Projects support --- if an IDE has been launched withing the context of a project, the environment will have the `JULIAHUB_PROJECT_UUID` environment set, and the authentication object will be automatically linked to the correct project.
+
 !!! tip "No need to authenticate()"
 
     As in a JuliaHub environment everything is already set up and pointing to the correct server, you do not need to call [`authenticate`](@ref) yourself, nor set up `JULIA_PKG_SERVER`.

docs/src/reference/datasets.md

@@ -41,15 +41,16 @@ The versions are indexed with a linear list of integers starting from `1`.
 ## Reference
 
 ```@docs
-JuliaHub.Dataset
-JuliaHub.DatasetVersion
-JuliaHub.datasets
-JuliaHub.DatasetReference
-JuliaHub.dataset
-JuliaHub.download_dataset
-JuliaHub.upload_dataset
-JuliaHub.update_dataset
-JuliaHub.delete_dataset
+Dataset
+DatasetVersion
+datasets
+DatasetReference
+dataset
+download_dataset
+upload_dataset
+update_dataset
+delete_dataset
+DatasetProjectLink
 ```
 
 ## Index

docs/src/reference/exceptions.md

@@ -24,6 +24,8 @@ InvalidRequestError
 JuliaHubConnectionError
 JuliaHubError
 PermissionError
+ProjectNotSetError
+InvalidJuliaHubVersion
 ```
 
 ## Index

docs/src/reference/projects.md

@@ -0,0 +1,46 @@
+```@meta
+CurrentModule=JuliaHub
+```
+
+# Projects
+
+These APIs allow you to interact with datasets that have been attached to projects.
+
+* [`project_datasets`](@ref) and [`project_dataset`](@ref) let you list and access datasets linked to a project
+* [`upload_project_dataset`](@ref) allows uploading new versions of project-linked datasets
+
+## Automatic project authentication
+
+The [`Authentication`](@ref) object can be associated with a default project UUID, which will
+then be used to for all _project_ operations, unless an explicit `project` gets passed to
+override the default.
+
+Importantly, [`JuliaHub.authenticate`](@ref) will automatically pick up the the JuliaHub
+project UUID from the `JULIAHUB_PROJECT_UUID` environment variable. This means in JuliaHub
+cloud jobs and IDEs, it is not necessary to manually set the project, and JuliaHub.jl
+will automatically.
+However, you can opt-out of this behavior by explicitly passing a `project=nothing` to
+[`JuliaHub.authenticate`](@ref).
+
+You can always verify that your operations are running in the context of the correct project
+by checking the [`Authentication`](@ref) object, e.g. via [`current_authentication`](@ref):
+
+```jldoctest; setup = :(using JuliaHub; Main.projectauth_setup!()), teardown = :(Main.projectauth_teardown!())
+julia> JuliaHub.current_authentication()
+JuliaHub.Authentication("https://juliahub.com", "username", *****; project_id = "cd6c9ee3-d15f-414f-a762-7e1d3faed835")
+```
+
+## Reference
+
+```@docs
+project_datasets
+project_dataset
+upload_project_dataset
+ProjectReference
+```
+
+## Index
+
+```@index
+Pages = ["project_datasets.md"]
+```

src/JuliaHub.jl

@@ -11,7 +11,7 @@ import SHA
 import TimeZones
 import TOML
 import URIs
-import UUIDs
+using UUIDs: UUIDs, UUID
 
 # We cache the local timezone in a global, so that we don't have to call
 # TimeZones.localzone() every time we do a TZ operation. However, we only
@@ -34,6 +34,7 @@ include("jobs/request.jl")
 include("jobs/logging.jl")
 include("jobs/logging-kafka.jl")
 include("jobs/logging-legacy.jl")
+include("projects.jl")
 
 # JuliaHub.jl follows the convention that all private names are
 # prefixed with an underscore.