diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 33fea112..7272f41b 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1,6 +1,7 @@ * @bcumming @msimberg @RMeli docs/access/jupyterlab.md @rsarm -docs/services/firecrest @jpdorsch @ekouts +docs/services/firecrest.md @jpdorsch @ekouts @theely @iB0nes +docs/services/devportal.md @jpdorsch @ekouts @theely @iB0nes docs/services/kubernetes @eliaoggian docs/software/communication @Madeeks @msimberg docs/software/devtools/linaro @jgphpc diff --git a/.github/actions/spelling/allow.txt b/.github/actions/spelling/allow.txt index 66e88382..b240e9e1 100644 --- a/.github/actions/spelling/allow.txt +++ b/.github/actions/spelling/allow.txt @@ -76,6 +76,8 @@ NICs NVMe Nordend OpenFabrics +OAuth +OIDC OSS OSSs OTP diff --git a/.github/actions/spelling/patterns.txt b/.github/actions/spelling/patterns.txt index 44ba3e6e..0ad1687f 100644 --- a/.github/actions/spelling/patterns.txt +++ b/.github/actions/spelling/patterns.txt @@ -14,6 +14,7 @@ Säntis ScaLAPACK VSCode aarch64 +WSO2 # markdown figure !\[.*\]\(.*\) diff --git a/docs/services/firecrest.md b/docs/access/firecrest.md similarity index 81% rename from docs/services/firecrest.md rename to docs/access/firecrest.md index 2e5d7018..93b140cf 100644 --- a/docs/services/firecrest.md +++ b/docs/access/firecrest.md @@ -3,9 +3,9 @@ FirecREST is a RESTful API for programmatically accessing High-Performance Computing resources, developed at CSCS. -Users can make use of FirecREST to automate access to HPC, enabling [CI/CD pipelines](https://eth-cscs.github.io/firecrest-v2/use_cases/CI-pipeline/), [workflow managers](https://github.com/eth-cscs/firecrest/tree/master/examples/airflow-operators), and other tools against HPC resources. +Users can make use of FirecREST to automate access to HPC, enabling [CI/CD pipelines](https://eth-cscs.github.io/firecrest-v2/use_cases/CI-pipeline/), [workflow orchestrators](https://eth-cscs.github.io/firecrest-v2/use_cases/workflow-orchestrator/), and other tools against HPC resources. -Additionally, scientific platform developers can integrate FirecREST into [web-enabled portals](https://my.hpcp.cscs.ch) and [applications](https://github.com/eth-cscs/firecrest/tree/master/examples/UI-client-credentials), allowing them to securely access authenticated and authorized CSCS services such as job submission and data transfer on HPC systems. +Additionally, scientific platform developers can integrate FirecREST into [web-enabled portals](https://eth-cscs.github.io/firecrest-ui/home/) and [web UI applications](https://eth-cscs.github.io/firecrest-v2/use_cases/UI-client-credentials/), allowing them to securely access authenticated and authorized CSCS services such as job submission and data transfer on HPC systems. Users can make HTTP requests to perform the following operations: @@ -55,7 +55,7 @@ FirecREST is available for all three major [Alps platforms][ref-alps-platforms], ### Clients and access tokens -For authenticating requests to FirecREST, **client applications** use an **access token** instead of directly using the user's credentials. +For authenticating requests to FirecREST, [client applications][ref-devportal-application] use an **access token** instead of directly using the user's credentials. The access token is a signed JSON Web Token ([JWT](https://jwt.io/introduction)) which contains user information and is only valid for a short time (5 minutes). Behind the API, all commands launched by the client will use the account of the user that registered the client, inheriting their access rights. @@ -65,77 +65,14 @@ Every client has a client ID (Consumer Key) and a secret (Consumer Secret) that ``` curl -s -X POST https://auth.cscs.ch/auth/realms/firecrest-clients/protocol/openid-connect/token \ --data "grant_type=client_credentials" \ - --data "client_id=" \ - --data "client_secret=" + --data "client_id=" \ + --data "client_secret=" ``` You can manage your client application on the [CSCS Developer Portal][ref-devportal]. -[](){#ref-devportal} -### CSCS Developer Portal -The [Developer Portal](https://developer.cscs.ch) facilitates CSCS users to manage subscriptions to an API at CSCS (such as FirecREST v1/v2). - -Start by browsing to [developer.cscs.ch](https://developer.cscs.ch), then sign in by clicking the "SIGN-IN" button on the top right hand corner of the page. - -Once logged in, you will see a list of APIs that are available to your user. - -!!! Warning - You might not see version 1 or version 2 of some API. You will be able to see all the versions when you *subscribe* your Application to the API. - -### Creating an Application - -Click on the "Applications" button at the top of the screen to manage your Applications. - -![FirecREST Main Page](../images/firecrest/f7t-apis.png) - -To create a new application, click on the "ADD NEW APPLICATION" button at the top of the Applications page, and complete the mandatory fields (marked with `*`). -Make sure to give the application a unique name and select the number of requests per minute. -When finished, click on the "Save" button. - -!!! note - To subscribe to an API you need at least one application, for which it is possible to use the DefaultApplication. - -!!! note - The quota of requests per minute will be shared by all subscribers to the Application over all APIs. - -### Configuring Production Keys - -Once the Application is created, create the Production Keys (`Client ID` and `Client Secret`) by clicking on "Production Keys" - -![FirecREST production keys](../images/firecrest/f7t-keys.png) - - -Use this if this is your first FirecREST application, or if you wish to create new keys. - -* click on the "Generate Keys" button at the bottom of the page - -![FirecREST existing keys](../images/firecrest/f7t-generate-keys.png) - -Once the keys are generated, you will see the pair "Consumer Key" and "Consumer Secret". - -![FirecREST keys](../images/firecrest/f7t-keys-overview.png) - -!!! warning - Store this pair of credentials securely, these are the access keys to your resources at CSCS. - -### Subscribe to an API - -Once you have set up your Application, is time to subscribe it to an API. - -To do so: - -* (8a) click on the "Subscriptions" option on the left panel -* (8b) click the :fontawesome-solid-circle-plus: "Subscribe APIS" button -* (8c) choose the API you want to subscribe to by clicking the "Subscribe" button - -![FirecREST subscriptions](../images/firecrest/f7t-api-subscriptions.png) - -Back on the Subscription Management page, you can review your active subscriptions and APIs that your Application has access to. - -![FirecREST subscription management](../images/firecrest/f7t-api-subscriptions-management.png) - -To use your Application to access FirecREST, follow the [API documentation](https://eth-cscs.github.io/firecrest-v2/openapi). +To use your client credentials to access FirecREST, follow the [API documentation](https://eth-cscs.github.io/firecrest-v2/openapi). ## Getting Started @@ -443,8 +380,9 @@ A staging area is used for external transfers and downloading/uploading a file f ## Further Information -* [FirecREST UI for HPC Platform](https://my.hpcp.cscs.ch) -* [FirecREST UI for ML Platform](https://my.mlp.cscs.ch) +* [HPC Platform Dashboard](https://my.hpcp.cscs.ch) +* [ML Platform Dashboard](https://my.mlp.cscs.ch) +* [C&W Platform Dashboard](https://my.cwp.cscs.ch) * [FirecREST OpenAPI Specification](https://eth-cscs.github.io/firecrest-v2/openapi) * [FirecREST Official Docs](https://eth-cscs.github.io/firecrest-v2) * [Documentation of pyFirecREST](https://pyfirecrest.readthedocs.io/) diff --git a/docs/access/index.md b/docs/access/index.md index 68f7fe7e..ec54f1d4 100644 --- a/docs/access/index.md +++ b/docs/access/index.md @@ -26,6 +26,12 @@ This documentation guides users through the process of accessing CSCS systems an [:octicons-arrow-right-24: SSH][ref-ssh] +- :material-fire-circle: __FirecREST__ + + FirecREST is a RESTful API for programmatically accessing High-Performance Computing resources. + + [:octicons-arrow-right-24: FirecREST][ref-firecrest] + - :simple-jupyter: __JupyterLab__ JupyterLab is a feature-rich notebook authoring application and editing environment. diff --git a/docs/images/firecrest/f7t-existing-keys.png b/docs/images/firecrest/f7t-existing-keys.png deleted file mode 100644 index 94ea86a5..00000000 Binary files a/docs/images/firecrest/f7t-existing-keys.png and /dev/null differ diff --git a/docs/images/services/devportal-api-docs.png b/docs/images/services/devportal-api-docs.png new file mode 100644 index 00000000..d7f28248 Binary files /dev/null and b/docs/images/services/devportal-api-docs.png differ diff --git a/docs/images/services/devportal-api-overview.png b/docs/images/services/devportal-api-overview.png new file mode 100644 index 00000000..9387f3e5 Binary files /dev/null and b/docs/images/services/devportal-api-overview.png differ diff --git a/docs/images/firecrest/f7t-api-subscriptions-management.png b/docs/images/services/devportal-api-subscriptions-management.png similarity index 100% rename from docs/images/firecrest/f7t-api-subscriptions-management.png rename to docs/images/services/devportal-api-subscriptions-management.png diff --git a/docs/images/firecrest/f7t-api-subscriptions.png b/docs/images/services/devportal-api-subscriptions.png similarity index 100% rename from docs/images/firecrest/f7t-api-subscriptions.png rename to docs/images/services/devportal-api-subscriptions.png diff --git a/docs/images/firecrest/f7t-apis.png b/docs/images/services/devportal-apis.png similarity index 100% rename from docs/images/firecrest/f7t-apis.png rename to docs/images/services/devportal-apis.png diff --git a/docs/images/services/devportal-complete-new-app.png b/docs/images/services/devportal-complete-new-app.png new file mode 100644 index 00000000..06f73fbe Binary files /dev/null and b/docs/images/services/devportal-complete-new-app.png differ diff --git a/docs/images/services/devportal-create-new-app.png b/docs/images/services/devportal-create-new-app.png new file mode 100644 index 00000000..b9621906 Binary files /dev/null and b/docs/images/services/devportal-create-new-app.png differ diff --git a/docs/images/firecrest/f7t-generate-keys.png b/docs/images/services/devportal-generate-keys.png similarity index 100% rename from docs/images/firecrest/f7t-generate-keys.png rename to docs/images/services/devportal-generate-keys.png diff --git a/docs/images/firecrest/f7t-keys-overview.png b/docs/images/services/devportal-keys-overview.png similarity index 100% rename from docs/images/firecrest/f7t-keys-overview.png rename to docs/images/services/devportal-keys-overview.png diff --git a/docs/images/firecrest/f7t-keys.png b/docs/images/services/devportal-keys.png similarity index 100% rename from docs/images/firecrest/f7t-keys.png rename to docs/images/services/devportal-keys.png diff --git a/docs/images/services/devportal-regenerate-keys.png b/docs/images/services/devportal-regenerate-keys.png new file mode 100644 index 00000000..92e1aa93 Binary files /dev/null and b/docs/images/services/devportal-regenerate-keys.png differ diff --git a/docs/index.md b/docs/index.md index a87abe0f..e45aa211 100644 --- a/docs/index.md +++ b/docs/index.md @@ -55,6 +55,8 @@ The Alps Research infrastructure hosts multiple platforms and clusters targeting [:octicons-arrow-right-24: Using SSH][ref-ssh] + [:octicons-arrow-right-24: FirecREST API][ref-firecrest] + ## Tutorials and Guides @@ -91,7 +93,7 @@ Learn by doing with our guides and tutorials. Learn about individual services - [:octicons-arrow-right-24: FirecREST API](services/firecrest.md) + [:octicons-arrow-right-24: Developer Portal][ref-devportal] [:octicons-arrow-right-24: CI/CD for external projects](services/cicd.md) diff --git a/docs/services/devportal.md b/docs/services/devportal.md new file mode 100644 index 00000000..462b88a9 --- /dev/null +++ b/docs/services/devportal.md @@ -0,0 +1,149 @@ +[](){#ref-devportal} +# Developer Portal + +The [Developer Portal](https://developer.cscs.ch) facilitates CSCS users to manage client applications' subscriptions to an API at CSCS, such as [FirecREST][ref-firecrest] or [CI/CD][ref-cicd]. + +## Terminology + +### API + +An **API** is an interface exposed by CSCS to their users to allow them accessing protected resources (computing, storage, accounting data, etc.) in an automated fashion. + +Examples of automated access to HPC resources are: + +- A cronjob that triggers a CI/CD pipeline to test software periodically +- A workflow manager that executes workflows asynchronously +- A web application that performs unattended data transfer after a job finishes + +[](){#ref-devportal-application} +### Application + +An **application** (or "*client application*" or "*client*") is a software created by users that consumes the resources exposed by an API. + +At CSCS, it requires [OIDC/OAuth2 access tokens](https://datatracker.ietf.org/doc/html/rfc6749#section-1.4) to authenticate against an [Identity Provider](#identity-provider-idp) (IdP) in order to access the protected resources. + +An application *consumes* API resources by *subscribing* to the API. At CSCS, applications can be subscribed to multiple APIs. + +### Identity Provider (IdP) + +An **identity provider** is an entity that authenticates an application and allows accessing protected resources. In this case, CSCS provides an IdP for all its APIs. + +### Production Keys + +Applications must identify against the CSCS IdP. To do this, users must create the **production keys** (*Consumer ID* - or *Client ID* -, and *Consumer Secret* - or *Client Secret*) that are going to be used to authenticate. + +!!! warning + The pair Consumer Key and Secret represents a personal credential to access CSCS resources, therefore can't be shared and must be stored securely (i.e.: not exposed in public repositories, forums, documentation, etc) + +## Getting started + +Start by browsing to [developer.cscs.ch](https://developer.cscs.ch), then sign in by clicking the `SIGN-IN` button on the top right hand corner of the page. + +Once logged in, you will see a list of APIs that are available to your user. + +!!! warning + You might not see version 1 or version 2 of some APIs. You will be able to see all the versions when you *subscribe* your Application to the API. + +### Creating an Application + +Click on the `Applications` option at the top of the screen to manage your applications. + +![FirecREST Main Page](../images/services/devportal-apis.png) + +To create a new application, click on the `ADD NEW APPLICATION` button at the top of the Applications view + +![Create New App](../images/services/devportal-create-new-app.png) + +Complete the mandatory fields (marked with `*`) and make sure to give the application a unique name and select the number of requests per minute. + +When finished, click on the `SAVE` button. + +!!! note + On your first login you will find the `DefaultApplication` which is created by default + +!!! note + The quota of requests per minute will be shared by all subscribers to the Application over all APIs + +### Configuring Production Keys + +Once the Application is created, create the Production Keys by clicking on `Production Keys` in the left panel: + +![FirecREST production keys](../images/services/devportal-keys.png) + +Click on the `Generate Keys` button at the bottom of the page to generate the production keys of your application: + +![FirecREST existing keys](../images/services/devportal-generate-keys.png) + +Once the keys are generated, you will see the pair `Consumer Key` (also known as "client id") and `Consumer Secret` (also known as "client secret"). + +![FirecREST keys](../images/services/devportal-keys-overview.png) + +!!! note + At the moment, only FIRECREST keys are available for the users + +### Subscribe to an API + +Once you have set up your Application, is time to subscribe it to an API. + +To do so: + +* (8.a) click on the :fontawesome-solid-rss: `Subscriptions` option on the left panel +* (8.b) click the :fontawesome-solid-circle-plus: `SUBSCRIBE APIS` button +* (8.c) choose the [business plan][ref-devportal-api-info] and the version of the API you want to subscribe to by clicking the `SUBSCRIBE` button on the right-side of the requested API. + +![Devportal subscriptions](../images/services/devportal-api-subscriptions.png) + +## Manage your applications + +Back on the Subscription Management page, you can review your active subscriptions and APIs that your Application has access to. + +![Devportal subscription management](../images/services/devportal-api-subscriptions-management.png) + +### Regenerate production keys + +It's a good practice to rotate or regenerate the production keys of your application to increase security. + +To do this, go to the `Production Keys` view of the desired application and click on the button `REMOVE KEYS`: + +![Devportal reset keys](../images/services/devportal-regenerate-keys.png) + +To regenerate the keys, refer back to the [Configuring Production Keys](#configuring-production-keys) section. + +!!!info + After resetting the credentials, you will need to reconfigure your software(s) to use the new keys + +### Remove your application + +To delete your application (which means removing all subscriptions and production keys), refer to the Application view and click on the `DELETE` button in the top-right corner. + +[](){#ref-devportal-api-info} +## View information of APIs + +The Developer Portal includes information of the APIs exposed by CSCS on the API view. + +Click on `APIs` button on the top of the Developer Portal, and select on the left menu the option `Overview` + +![Devportal apis overview](../images/services/devportal-api-overview.png) + +The information exposed in the overview view is: + +* Amount of subscriptions to the API (among all clients) +* Documentation of the API (OpenAPI reference, SDK tools, etc) +* Community information (Public repository, slack channel) +* Contact information (developers and business) +* Business plans (amount of requests per minute that are allowed in the API subscription) +* Applications owned by the user that are subscribed to the API + + +### API Documents + +Clicking in the left option `Documents` the user will be able to see the API reference with examples: + +![Devportal apis reference](../images/services/devportal-api-docs.png) + +## Further Information + +* [WSO2 Developer Portal](https://apim.docs.wso2.com/en/4.4.0/consume/consume-api-overview/) +* [Open ID Connect protocol (OIDC)](https://openid.net/developers/how-connect-works/) +* [JSON Web Tokens](https://www.jwt.io/introduction#what-is-json-web-token) +* [OpenAPI initiative](https://www.openapis.org/what-is-openapi) \ No newline at end of file diff --git a/docs/services/index.md b/docs/services/index.md index c94b4708..4b473f42 100644 --- a/docs/services/index.md +++ b/docs/services/index.md @@ -7,11 +7,11 @@ [:octicons-arrow-right-24: CI/CD][ref-cicd] -- :fontawesome-solid-mountain-sun: __FirecREST__ +- :material-garage-variant-lock: __Developer Portal__ - FirecREST is a RESTful API for programmatically accessing High-Performance Computing resources. + Enables CSCS users to create and manage subscriptions to an API. - [:octicons-arrow-right-24: FirecREST][ref-firecrest] + [:octicons-arrow-right-24: Developer Portal][ref-devportal] - :fontawesome-solid-dharmachakra: __Kubernetes__ diff --git a/mkdocs.yml b/mkdocs.yml index 9688f989..dbfa4f6c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -41,6 +41,7 @@ nav: - 'Multi Factor Authentication (MFA)': access/mfa.md - 'Web Services': access/web.md - 'SSH': access/ssh.md + - 'FirecREST': access/firecrest.md - 'JupyterLab': access/jupyterlab.md - 'VSCode': access/vscode.md - 'Software': @@ -105,7 +106,7 @@ nav: - 'WRF': build-install/applications/wrf.md - 'Services': - services/index.md - - 'FirecREST': services/firecrest.md + - 'Developer Portal': services/devportal.md - 'CI/CD': services/cicd.md - 'Kubernetes': - services/kubernetes/index.md