RHIDP-1925 bulk import from GitHub (#616) (#635)

Co-authored-by: Armel Soro <[email protected]>
Co-authored-by: Heena Manwani <[email protected]>
Co-authored-by: Subhash Khileri <[email protected]>

Author: 4 people

artifacts/snip-technology-preview.adoc

@@ -0,0 +1,6 @@
+[IMPORTANT]
+====
+These features are for Technology Preview only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs), might not be functionally complete, and Red Hat does not recommend using them for production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
+
+For more information on Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/[Technology Preview Features Scope].
+====

assemblies/assembly-bulk-importing-from-github.adoc

@@ -0,0 +1,15 @@
+[id="bulk-importing-github-repositories"]
+= Bulk importing GitHub repositories
+
+include::{docdir}/artifacts/snip-technology-preview.adoc[]
+
+{product} can automate GitHub repositories onboarding and track their import status.
+
+include::modules/importing-repositories/procedure-enabling-the-bulk-import-from-github-feature.adoc[leveloffset=+1]
+
+include::modules/importing-repositories/procedure-importing-multiple-repositories-from-github.adoc[leveloffset=+1]
+
+include::modules/importing-repositories/procedure-managing-the-imported-repository-list.adoc[leveloffset=+1]
+
+include::modules/importing-repositories/procedure-understanding-bulk-import-audit-logs.adoc[leveloffset=+1]
+

modules/authorization/ref-rbac-permission-policies.adoc

@@ -78,6 +78,22 @@ Catalog permissions::
 |Allows user or role to delete locations from the catalog
 |===
 
+Bulk import permissions::
+
+[cols="15%,25%,15%,45%", frame="all", options="header"]
+|===
+|Name
+|Resource type
+|Policy
+|Description
+
+|`bulk.import`
+|`bulk-import`
+|
+|Allows the user to access the bulk import endpoints, such as listing all repositories and organizations accessible by all GitHub integrations and managing the import requests.
+
+|===
+
 Scaffolder permissions::
 
 [cols="15%,25%,15%,45%", frame="all", options="header"]

modules/getting-started/con-servicenow-custom-actions.adoc

@@ -1,12 +1,7 @@
 [id='con-servicenow-custom-actions_{context}']
 = ServiceNow Custom actions in {product}
 
-[IMPORTANT]
-====
-These features are for Technology Preview only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs), might not be functionally complete, and Red Hat does not recommend using them for production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
-
-For more information on Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/[Technology Preview Features Scope].
-====
+include::{docdir}/artifacts/snip-technology-preview.adoc[]
 
 In {product}, you can access ServiceNow custom actions (custom actions) for fetching and registering resources in the catalog.
 

modules/importing-repositories/procedure-enabling-the-bulk-import-from-github-feature.adoc

@@ -0,0 +1,40 @@
+[id="enabling-and-giving-access-to-the-bulk-import-feature"]
+= Enabling and giving access to the Bulk Import feature
+You can enable the Bulk Import feature for users and give them the necessary permissions to access it.
+
+.Prerequisites
+* You have link:{authentication-book-url}#enabling-authentication-with-github[configured GitHub authentication and integration].
+
+.Procedure
+
+. The Bulk Import plugins are installed but disabled by default.
+To enable the `./dynamic-plugins/dist/janus-idp-backstage-plugin-bulk-import-backend-dynamic` and `./dynamic-plugins/dist/janus-idp-backstage-plugin-bulk-import` plugins,
+edit your `dynamic-plugins.yaml` with the following content:
++
+.`dynamic-plugins.yaml` fragment
+[source,yaml]
+----
+plugins:
+  - package: ./dynamic-plugins/dist/janus-idp-backstage-plugin-bulk-import-backend-dynamic
+    disabled: false
+  - package: ./dynamic-plugins/dist/janus-idp-backstage-plugin-bulk-import
+    disabled: false
+----
++
+See link:{installing-and-viewing-dynamic-plugins-url}[{installing-and-viewing-dynamic-plugins-title}].
+
+. Configure the required `bulk.import` RBAC permission for the users who are not administrators as follows:
++
+.`rbac-policy.csv` fragment
+[source,csv,subs="+quotes"]
+----
+p, role:default/bulk-import, bulk.import, use, allow
+g, user:default/__<your_user>__, role:default/bulk-import
+----
++
+Note that only {product-short} administrators or users with the `bulk.import` permission can use the Bulk Import feature. See link:{authorization-book-url}#ref-rbac-permission-policies_title-authorization[Permission policies in Red Hat Developer Hub].
+
+.Verification
+* The sidebar displays a *Bulk Import* option.
+* The *Bulk Import* page shows a list of *Added Repositories*.
+

modules/importing-repositories/procedure-importing-multiple-repositories-from-github.adoc

@@ -0,0 +1,37 @@
+[id="importing-multiple-github-repositories]
+= Importing multiple GitHub repositories
+
+In {product}, you can select your GitHub repositories and automate their onboarding to the {product-short} catalog.
+
+.Prerequisites
+* You have link:{authentication-book-url}#enabling-authentication-with-github[configured GitHub authentication and integration].
+* You have xref:enabling-and-giving-access-to-the-bulk-import-feature[enabled the Bulk Import feature and gave access to it].
+
+.Procedure
+. Click *Bulk Import* in the left sidebar.
+. Click the *Add* button in the top-right corner to see the list of all repositories accessible from the configured GitHub integrations.
+.. From the *Repositories* view, you can select any repository, or search for any accessible repositories.
+For each repository selected, a `catalog-info.yaml` is generated.
+.. From the *Organizations* view, you can select any organization by clicking *Select* in the third column.
+This option allows you to select one or more repositories from the selected organization.
+. Click *Preview file* to view or edit the details of the pull request for each repository.
+.. Review the pull request description and the `catalog-info.yaml` file content.
+.. Optional: when the repository has a `.github/CODEOWNERS` file, you can select the *Use CODEOWNERS file as Entity Owner* checkbox to use it, rather than having the `content-info.yaml` contain a specific entity owner.
+.. Click *Save*.
+. Click *Create pull requests*.
+At this point, a set of dry-run checks runs against the selected repositories to ensure they meet the requirements for import, such as:
+.. Verifying that there is no entity in the {product-short} catalog with the name specified in the repository `catalog-info.yaml`
+.. Verifying that the repository is not empty
+.. Verifying that the repository contains a `.github/CODEOWNERS` file if the *Use CODEOWNERS file as Entity Owner* checkbox is selected for that repository
+
+** If any errors occur, the pull requests are not created, and you see a _Failed to create PR_ error message detailing the issues.
+To view more details about the reasons, click *Edit*.
+
+** If there are no errors, the pull requests are created, and you are redirected to the list of added repositories.
+
+. Review and merge each pull request that creates a `catalog-info.yml` file.
+
+.Verification
+* The *Added repositories* list displays the repositories you imported, each with an appropriate status: either _Waiting for approval_ or _Added_.
+* For each _Waiting for approval_ import job listed, there is a corresponding pull request adding the `catalog-info.yaml` file in the corresponding repository.
+

modules/importing-repositories/procedure-managing-the-imported-repository-list.adoc

@@ -0,0 +1,29 @@
+[id="managing-the-added-repositories]
+= Managing the added repositories
+You can oversee and manage the repositories that are imported to the {product-short}.
+
+.Prerequisites
+* You have xref:importing-multiple-github-repositories[imported GitHub repositories].
+
+
+.Procedure
+. Click *Bulk Import* in the left sidebar to display all the current repositories that are being tracked as Import jobs, along with their status.
+
+Added:: The repository is added to the {product-short} catalog after the import pull request is merged or if the repository already contained a `catalog-info.yaml` file during the bulk import. 
+Note that it may take a few minutes for the entities to be available in the catalog.
+
+Waiting for approval:: There is an open pull request adding a `catalog-info.yaml` file to the repository.
+You can:
+* Click the *pencil icon* on the right to see details about the pull request or edit the pull request content right from {product-short}.
+* Delete the Import job, this action closes the import PR as well.
+* To transition the Import job to the _Added_ state, merge the import pull request from the Git repository.
+
+Empty:: {product-short} is unable to determine the import job status because the repository is imported from other sources but does not have a `catalog-info.yaml` file and lacks any import pull request adding it.
+
+[NOTE]
+====
+* After an import pull request is merged, the import status is marked as _Added_ in the list of Added Repositories, but it might take a few seconds for the corresponding entities to appear in the {product-short} Catalog.
+* A location added through other sources (like statically in an `app-config.yaml` file, dynamically when link:{linkgettingstartedguide}#enabling-github-discovery-in-red-hat-developer-hub[enabling GitHub discovery], or registered manually using the "Register an existing component" page) might show up in the Bulk Import list of Added Repositories if the following conditions are met:
+** The target repository is accessible from the configured GitHub integrations.
+** The location URL points to a `catalog-info.yaml` file at the root of the repository default branch.
+====

modules/importing-repositories/procedure-understanding-bulk-import-audit-logs.adoc

@@ -0,0 +1,70 @@
+[id="understanding-bulk-import-audit-logs"]
+= Understanding the Bulk Import audit Logs
+
+The Bulk Import backend plugin adds the following events to the {product-short} audit logs.
+See link:{linkgettingstartedguide}#assembly-audit-log[Audit Logs in {product}] for more information on how to configure and view audit logs.
+
+*Bulk Import Events*:
+
+`BulkImportUnknownEndpoint`::
+Tracks requests to unknown endpoints.
+
+`BulkImportPing`::
+Tracks `GET` requests to the `/ping` endpoint, which allows us to make sure the bulk import backend is up and running.
+
+`BulkImportFindAllOrganizations`::
+Tracks `GET` requests to the `/organizations` endpoint, which returns the list of organizations accessible from all configured GitHub Integrations.
+
+`BulkImportFindRepositoriesByOrganization`::
+Tracks `GET` requests to the `/organizations/:orgName/repositories` endpoint, which returns the list of repositories for the specified organization (accessible from any of the configured GitHub Integrations).
+
+`BulkImportFindAllRepositories`::
+Tracks GET requests to the `/repositories` endpoint, which returns the list of repositories accessible from all configured GitHub Integrations.
+
+`BulkImportFindAllImports`::
+Tracks `GET` requests to the `/imports` endpoint, which returns the list of existing import jobs along with their statuses.
+
+`BulkImportCreateImportJobs`::
+Tracks `POST` requests to the `/imports` endpoint, which allows to submit requests to bulk-import one or many repositories into the {product-short} catalog, by eventually creating import pull requests in the target repositories.
+
+`BulkImportFindImportStatusByRepo`::
+Tracks `GET` requests to the `/import/by-repo` endpoint, which fetches details about the import job for the specified repository.
+
+`BulkImportDeleteImportByRepo`::
+Tracks `DELETE` requests to the `/import/by-repo` endpoint, which deletes any existing import job for the specified repository, by closing any open import pull request that could have been created.
+
+.Example bulk import audit logs
+[code,json]
+----
+{
+  "actor": {
+    "actorId": "user:default/myuser",
+    "hostname": "localhost",
+    "ip": "::1",
+    "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
+  },
+  "eventName": "BulkImportFindAllOrganizations",
+  "isAuditLog": true,
+  "level": "info",
+  "message": "'get /organizations' endpoint hit by user:default/myuser",
+  "meta": {},
+  "plugin": "bulk-import",
+  "request": {
+    "body": {},
+    "method": "GET",
+    "params": {},
+    "query": {
+      "pagePerIntegration": "1",
+      "sizePerIntegration": "5"
+    },
+    "url": "/api/bulk-import/organizations?pagePerIntegration=1&sizePerIntegration=5"
+  },
+  "response": {
+    "status": 200
+  },
+  "service": "backstage",
+  "stage": "completion",
+  "status": "succeeded",
+  "timestamp": "2024-08-26 16:41:02"
+}
+----

titles/getting-started-rhdh/title-getting-started.adoc

@@ -20,6 +20,8 @@ include::modules/getting-started/ref-rhdh-sizing.adoc[leveloffset=+1]
 include::modules/getting-started/ref-rhdh-supported-configs.adoc[leveloffset=+1]
 include::assemblies/assembly-add-custom-app-file-openshift.adoc[leveloffset=+2]
 
+include::assemblies/assembly-bulk-importing-from-github.adoc[leveloffset=+1]
+
 include::modules/getting-started/proc-customize-rhdh-homepage.adoc[leveloffset=+1]
 include::modules/getting-started/proc-customize-rhdh-tech-radar-page.adoc[leveloffset=+1]
 include::modules/getting-started/proc-customize-rhdh-learning-paths.adoc[leveloffset=+1]