RHIDP-1925 bulk import from GitHub

assemblies/assembly-bulk-importing-from-github.adoc

@@ -0,0 +1,18 @@
+[id="bulk-importing-github-repositories"]
+= Bulk importing GitHub repositories
+
+[IMPORTANT]
+====
+This is a https://access.redhat.com/support/offerings/techpreview/[Technology Preview] feature.
+====
+
+{product} can automate GitHub repositories onboarding and track their import status.
+
+include::modules/importing-repositories/procedure-enabling-the-bulk-import-from-github-feature.adoc[leveloffset=+1]
+
+include::modules/importing-repositories/procedure-importing-multiple-repositories-from-github.adoc[leveloffset=+1]
+
+include::modules/importing-repositories/procedure-managing-the-imported-repository-list.adoc[leveloffset=+1]
+
+include::modules/importing-repositories/procedure-understanding-bulk-import-audit-logs.adoc[leveloffset=+1]
+

modules/authorization/ref-rbac-permission-policies.adoc

@@ -78,6 +78,22 @@ Catalog permissions::
 |Allows user or role to delete locations from the catalog
 |===
 
+Bulk import permissions::
+
+[cols="15%,25%,15%,45%", frame="all", options="header"]
+|===
+|Name
+|Resource type
+|Policy
+|Description
+
+|`bulk.import`
+|`bulk-import`
+|
+|Allows the user to access the bulk import endpoints, such as listing all repositories and organizations accessible by all GitHub integrations and managing the import requests.
+
+|===
+
 Scaffolder permissions::
 
 [cols="15%,25%,15%,45%", frame="all", options="header"]

modules/importing-repositories/procedure-enabling-the-bulk-import-from-github-feature.adoc

@@ -0,0 +1,41 @@
+[id="enabling-and-giving-access-to-the-bulk-import-feature"]
+= Enabling and giving access to the Bulk Import feature
+You can enable the Bulk Import feature for users and give them the necessary permissions to access it.
+.Prerequisites
+* You have link:{authentication-book-url}#enabling-authentication-with-github[configured GitHub authentication and integration].
+
+.Procedure
+
+. The Bulk Import plugins are installed but disabled by default.
+To enable the `./dynamic-plugins/dist/janus-idp-backstage-plugin-bulk-import-backend-dynamic` and `./dynamic-plugins/dist/janus-idp-backstage-plugin-bulk-import` plugins,
+edit your `dynamic-plugins.yaml` with the following content:
++
+.`dynamic-plugins.yaml` fragment
+[source,yaml]
+----
+plugins:
+  - package: ./dynamic-plugins/dist/janus-idp-backstage-plugin-bulk-import-backend-dynamic
+    disabled: false
+  - package: ./dynamic-plugins/dist/janus-idp-backstage-plugin-bulk-import
+    disabled: false
+----
++
+See link:{installing-and-viewing-dynamic-plugins-url}[{installing-and-viewing-dynamic-plugins-title}].
+
+. Configure the required `bulk.import` RBAC permission for the users who are not administrators as follows:
++
+.`rbac-policy.csv` fragment
+[source,csv,subs="+quotes"]
+----
+p, role:default/bulk-import, bulk.import, use, allow
+p, role:default/bulk-import, catalog-entity.read, read, allow
+p, role:default/bulk-import, catalog.entity.create, create, allow
+g, user:default/__<your_user>__, role:default/bulk-import
+----
++
+Note that only {product-short} administrators or users with the `bulk.import` permission can use the Bulk Import feature. See link:{authorization-book-url}#ref-rbac-permission-policies_title-authorization[Permission policies in Red Hat Developer Hub].
+
+.Verification
+* The sidebar displays a *Bulk Import* line.
+* The *Bulk Import* page shows a list of *Added Repositories*.
+

modules/importing-repositories/procedure-importing-multiple-repositories-from-github.adoc

@@ -0,0 +1,37 @@
+[id="importing-multiple-github-repositories]
+= Importing multiple GitHub repositories
+
+In {product}, you can select your GitHub repositories and automate their onboarding to the {product-short} catalog.
+
+.Prerequisites
+* You have link:{authentication-book-url}#enabling-authentication-with-github[configured GitHub authentication and integration].
+* You have xref:enabling-and-giving-access-to-the-bulk-import-feature[enabled the Bulk Import feature and gave access to it].
+
+.Procedure
+. Click *Bulk Import* in the left sidebar.
+. Click the *Add* button in the top-right corner to see the list of all repositories accessible from the configured GitHub integrations.
+.. From the *Repositories* view, you can select any repository, or search for any accessible repositories.
+For each repository selected, a `catalog-info.yaml` is generated.
+.. From the *Organizations* view, you can select any organization by clicking *Select* in the third column.
+This option allows you to select one or more repositories from the selected organization.
+. Click *Preview file* to view or edit the details of the pull request for each repository.
+.. Review the pull request description and the `catalog-info.yaml` file content.
+.. Optional: when the repository has a `.github/CODEOWNERS` file, you can select the *Use CODEOWNERS file as Entity Owner* checkbox to use it, rather than having the `content-info.yaml` contain a specific entity owner.
+.. Click *Save*.
+. Click *Create pull requests*.
+At this point, a set of dry-run checks runs against the selected repositories to ensure they meet the requirements for import, such as:
+.. Verifying that there is no entity in the {product-short} catalog with the name specified in the repository `catalog-info.yaml`
+.. Verifying that the repository is not empty
+.. Verifying that the repository contains a `.github/CODEOWNERS` file if the *Use CODEOWNERS file as Entity Owner* checkbox is selected for that repository
+
+** If any errors occur, the pull requests are not created, and you see a _Failed to create PR_ error message detailing the issues.
+To view more details about the reasons, click *Edit*.
+
+** If there are no errors, the pull requests are created, and you are redirected to the list of added repositories.
+
+. Review and merge each pull request that creates a `catalog-info.yml` file.
+
+.Verification
+* The *Added repositories* list displays the repositories you imported, each with an appropriate status: either _Waiting for approval_ or _Added_.
+* For each _Waiting for approval_ import job listed, there is a corresponding pull request adding the `catalog-info.yaml` file in the corresponding repository.
+

modules/importing-repositories/procedure-managing-the-imported-repository-list.adoc

@@ -0,0 +1,30 @@
+[id="managing-the-added-repositories]
+= Managing the added repositories
+You can oversee and manage the repositories that are imported to the {product-short}.
+.Prerequisites
+* You have xref:importing-multiple-github-repositories[imported GitHub repositories].
+
+
+.Procedure
+. Click *Bulk Import* in the left sidebar to display all the current repositories that are being tracked as Import jobs, along with their status.
+
+Added:: The repository is added to the {product-short} catalog after the import pull request is merged or if the repository already contained a `catalog-info.yaml` file during the bulk import. 
+This can be the case after the import PR has been merged, or if the repository already had a `catalog-info.yaml` when it was bulk-imported.
+Note that it may take a few minutes for the entities to be available in the catalog.
+
+Waiting for approval:: There is an open pull request adding a `catalog-info.yaml` file to the repository.
+You can:
+* Click the *pencil icon* on the right to see details about the pull request or edit the pull request content right from {product-short}.
+* Delete the Import job, this action closes the import PR as well.
+* To transition the Import job to the _Added_ state, merge the import pull request from the Git repository.
+
+Empty:: {product-short} is unable to determine the import job status because the repository is imported from other sources but does not have a `catalog-info.yaml` file and lacks any import pull request adding it.
+This can be the case if the repository was imported from other sources but does not have any `catalog-info.yaml` file and does not have any import PR adding it.
+
+[NOTE]
+====
+* After an import pull request is merged, the import status is marked as _Added_ in the list of Added Repositories, but it might take a few seconds for the corresponding entities to appear in the {product-short} Catalog.
+* A location added through other sources (like statically in an `app-config.yaml` file, dynamically when link:{linkgettingstartedguide}#enabling-github-discovery-in-red-hat-developer-hub[enabling GitHub discovery], or registered manually using the "Register an existing component" page) might show up in the Bulk Import list of Added Repositories if the following conditions are met:
+** The target repository is accessible from the configured GitHub integrations.
+** The location URL points to a `catalog-info.yaml` file at the root of the repository default branch.
+====

modules/importing-repositories/procedure-understanding-bulk-import-audit-logs.adoc

@@ -0,0 +1,70 @@
+[id="understanding-bulk-import-audit-logs"]
+= Understanding the Bulk Import audit Logs
+
+The Bulk Import backend plugin adds the following events to the {product-short} audit logs.
+See link:{linkgettingstartedguide}#assembly-audit-log[Audit Logs in {product}] for more information on how to configure and view audit logs.
+
+*Bulk Import Events*:
+
+`BulkImportUnknownEndpoint`::
+Tracks requests to unknown endpoints.
+
+`BulkImportPing`::
+Tracks `GET` requests to the `/ping` endpoint, which allows us to make sure the bulk import backend is up and running.
+
+`BulkImportFindAllOrganizations`::
+Tracks `GET` requests to the `/organizations` endpoint, which returns the list of organizations accessible from all configured GitHub Integrations.
+
+`BulkImportFindRepositoriesByOrganization`::
+Tracks `GET` requests to the `/organizations/:orgName/repositories` endpoint, which returns the list of repositories for the specified organization (accessible from any of the configured GitHub Integrations).
+
+`BulkImportFindAllRepositories`::
+Tracks GET requests to the `/repositories` endpoint, which returns the list of repositories accessible from all configured GitHub Integrations.
+
+`BulkImportFindAllImports`::
+Tracks `GET` requests to the `/imports` endpoint, which returns the list of existing import jobs along with their statuses.
+
+`BulkImportCreateImportJobs`::
+Tracks `POST` requests to the `/imports` endpoint, which allows to submit requests to bulk-import one or many repositories into the {product-short} catalog, by eventually creating import pull requests in the target repositories.
+
+`BulkImportFindImportStatusByRepo`::
+Tracks `GET` requests to the `/import/by-repo` endpoint, which fetches details about the import job for the specified repository.
+
+`BulkImportDeleteImportByRepo`::
+Tracks `DELETE` requests to the `/import/by-repo` endpoint, which deletes any existing import job for the specified repository, by closing any open import pull request that could have been created.
+
+.Example bulk import audit logs
+[code,json]
+----
+{
+  "actor": {
+    "actorId": "user:default/myuser",
+    "hostname": "localhost",
+    "ip": "::1",
+    "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
+  },
+  "eventName": "BulkImportFindAllOrganizations",
+  "isAuditLog": true,
+  "level": "info",
+  "message": "'get /organizations' endpoint hit by user:default/myuser",
+  "meta": {},
+  "plugin": "bulk-import",
+  "request": {
+    "body": {},
+    "method": "GET",
+    "params": {},
+    "query": {
+      "pagePerIntegration": "1",
+      "sizePerIntegration": "5"
+    },
+    "url": "/api/bulk-import/organizations?pagePerIntegration=1&sizePerIntegration=5"
+  },
+  "response": {
+    "status": 200
+  },
+  "service": "backstage",
+  "stage": "completion",
+  "status": "succeeded",
+  "timestamp": "2024-08-26 16:41:02"
+}
+----

titles/getting-started-rhdh/title-getting-started.adoc

@@ -20,6 +20,7 @@ include::modules/installation/ref-rhdh-sizing.adoc[leveloffset=+1]
 include::modules/getting-started/ref-rhdh-supported-configs.adoc[leveloffset=+1]
 include::assemblies/assembly-add-custom-app-file-openshift.adoc[leveloffset=+2]
 include::modules/getting-started/proc-add-source-control-rhdh-catalog.adoc[leveloffset=+2]
+include::assemblies/assembly-bulk-importing-from-github.adoc[leveloffset=+1]
 
 include::modules/getting-started/proc-customize-rhdh-homepage.adoc[leveloffset=+1]
 include::modules/getting-started/proc-customize-rhdh-tech-radar-page.adoc[leveloffset=+1]