RHIDP-1925 bulk import from GitHub

themr0c · themr0c · commit d68fec5f121a · 2024-10-10T17:41:13.000+02:00
Signed-off-by: Fabrice Flore-Thébault &lt;ffloreth@redhat.com&gt;
diff --git a/assemblies/assembly-bulk-importing-from-github.adoc b/assemblies/assembly-bulk-importing-from-github.adoc
@@ -0,0 +1,15 @@
+= Bulk Import GitHub repositories
+
+Bulk Import is a {company-name} *Tech Preview* feature.
+
+It allows you to automate GitHub repositories onboarding and track their import status.
+
+include::modules/importing-repositories/procedure-enabling-the-bulk-import-from-github-feature.adoc[leveloffset=+1]
+
+include::modules/importing-repositories/procedure-importing-multiple-repositories-from-github.adoc[leveloffset=+1]
+
+include::modules/importing-repositories/procedure-managing-the-imported-repository-list.adoc[leveloffset=+1]
+
+include::modules/importing-repositories/procedure-understanding-bulk-import-audit-logs.adoc[leveloffset=+1]
+
+include::modules/importing-repositories/procedure-understanding-bulk-import-limitations.adoc[leveloffset=+1]
diff --git a/modules/importing-repositories/procedure-enabling-the-bulk-import-from-github-feature.adoc b/modules/importing-repositories/procedure-enabling-the-bulk-import-from-github-feature.adoc
@@ -0,0 +1,16 @@
+= Enabling and giving access to the Bulk Import feature
+
+.Procedure
+
+. Set up at least one GitHub integration.
+  For more information, see link:{}#configuring-github-integration[Configuring GitHub integration].
+. Enable the **`./dynamic-plugins/dist/janus-idp-backstage-plugin-bulk-import-backend-dynamic`</strong> and *`./dynamic-plugins/dist/janus-idp-backstage-plugin-bulk-import`* preinstalled plugins. For more information, see https://docs.redhat.com/en/documentation/red_hat_developer_hub/1.2/html-single/configuring_plugins_in_red_hat_developer_hub/index[Configuring Plugins in Red Hat Developer Hub].
+. Configure the required `bulk.import` RBAC permission. See https://docs.redhat.com/en/documentation/red_hat_developer_hub/1.2/html-single/authorization/index#ref-rbac-permission-policies_title-authorization[Permission policies in Red Hat Developer Hub]. Only Developer Hub administrators or users with the `bulk.import` permission will be able to use the Bulk Import feature.
+
+.Verification
+
+* The sidebar displays a *Bulk Import* option and clicking on it shows a list of *Added Repositories*.
+
+==== Troubleshooting
+
+* If clicking on the *Bulk Import* option displays the following error message (_To view the added repositories, contact your administrator to give you the `bulk.import` permission_), it means that your user does not have the necessary permission to use the Bulk Import feature. Refer back to https://docs.redhat.com/en/documentation/red_hat_developer_hub/1.2/html-single/authorization/index#ref-rbac-permission-policies_title-authorization[Permission policies in Red Hat Developer Hub] for more information on how an administrator can add that permission to your user.
diff --git a/modules/importing-repositories/procedure-importing-multiple-repositories-from-github.adoc b/modules/importing-repositories/procedure-importing-multiple-repositories-from-github.adoc
@@ -0,0 +1,38 @@
+= Importing multiple repositories
+
+The Bulk Import feature allows you to view all the GitHub repositories and organizations accessible from the configured GitHub Integrations.
+From there, you can pick any number of repositories and onboard them.
+
+.Prerequisites
+* You enabled the Bulk Import feature.
+* Your RHDH user has access to the Bulk Import feature.
+
+.Procedure
+. Click on *Bulk Import* in the left sidebar.
+. Click on the *Add* button in the top-right corner.
+You will see a list of all repositories accessible from the configured GitHub integrations.
+. From the *Repositories* view, you can select any repository, or search for any accessible repositories.
+For each repository selected, a `catalog-info.yaml` will be generated.
+. From the *Organizations* view, you can select any organization by clicking on *Select* in the third column.
+This will allow you to pick one or more repositories that are part of the selected organization.
+. Click on *Preview file* to view or edit the details of the pull request for each repository.
+The pull request details form will be preloaded with some data, and you can also preview the content of the `catalog-info.yaml`.
+If the repository already has a `.github/CODEOWNERS` file, you can select the *Use CODEOWNERS file as Entity Owner* checkbox to use it, rather than having the `content-info.yaml` contain a specific entity owner.
+. Click on *Save*.
+. Click on *Create pull requests*.
+At this point, a set of dry-run checks will be run against the selected repositories to make sure they meet the requirements to be imported, that is:
+. Checking that there is no entity in the Developer Hub catalog with the name specified in the repository `catalog-info.yaml`
+. Checking that the repository is not empty
+. Checking that the repository already has a `.github/CODEOWNERS` file if the *Use CODEOWNERS file as Entity Owner* checkbox was selected for that repository.
+
+If there are any errors, the pull requests will not be created, but you will see a "``Failed to create PR``" error message indicating the failures. Clicking on *Edit* will show more details about the reasons.
+
+If there are no errors, the pull requests will be created, and you will be redirected to the list of Added repositories.
+
+. Review and merge each pull request that creates a `catalog-info.yml` file.
+
+.Verification
+* The list of Added repositories displays the repositories you just imported, each with an appropriate status: either "Waiting for approval" or "Added"
+* For each "Waiting for approval" Import job listed, there is a corresponding PR adding the `catalog-info.yaml` file in the corresponding repository
+
+
diff --git a/modules/importing-repositories/procedure-managing-the-imported-repository-list.adoc b/modules/importing-repositories/procedure-managing-the-imported-repository-list.adoc
@@ -0,0 +1,15 @@
+= Managing the list of Added repositories
+
+.Procedure
+
+. Click on *Bulk Import* in the left sidebar. It will display all the current repositories that are being tracked as Import jobs, along with their status.
+. If an Import is in the "**`Added`</strong>" state, it means that the repository has been added to the Developer Hub catalog. This can be the case after the import PR has been merged, or if the repository already had a `catalog-info.yaml` when it was bulk-imported. Please note that it might take a few minutes until the entities are available in the catalog.
+. If an Import is in the "*`Waiting for approval`*" state, it means that there is an open Pull Request adding a `catalog-info.yaml`. You can click on the *pencil icon* on the right to see details about this PR or edit the PR content right from Developer Hub. You can also delete the Import job, in which case the import PR will be closed as well. For the Import job to transition to the "*Added*" state, you (or someone with the appropriate write permissions on the repository) will need to merge the import PR from the Git repository.
+. If an Import does not have any state, it might mean that Developer Hub was not able to determine its status. This can be the case if the repository was imported from other sources but does not have any `catalog-info.yaml` file and does not have any import PR adding it.
+
+*NOTES*
+
+. After an import PR is merged, the Import status will be "Added" in the list of Added Repositories, but it might take a few seconds until the corresponding entities show up in the Developer Hub Catalog.
+. A Location added through other sources (like statically in an app-config file, dynamically when https://docs.redhat.com/en/documentation/red_hat_developer_hub/1.2/html/getting_started_with_red_hat_developer_hub/ref-rhdh-supported-configs_rhdh-getting-started#enabling-github-discovery-in-red-hat-developer-hub[enabling GitHub discovery], or registered manually via the "Register an existing component" page) may show up in the Bulk Import list of Added Repositories if:
+. the target repository is accessible from the configured GitHub integrations, and
+. the Location URL points to a catalog-info.yaml file at the root of the repository default branch
diff --git a/modules/importing-repositories/procedure-understanding-bulk-import-audit-logs.adoc b/modules/importing-repositories/procedure-understanding-bulk-import-audit-logs.adoc
@@ -0,0 +1,51 @@
+= Bulk Import Audit Logs
+
+The Bulk Import Backend plugin adds the events below to the Developer Hub audit logs. See https://docs.redhat.com/en/documentation/red_hat_developer_hub/1.2/html/getting_started_with_red_hat_developer_hub/assembly-audit-log[Audit Logs in Red Hat Developer Hub] for more information on how to configure and view such audit logs.
+
+*Bulk Import Events*:
+
+* **`BulkImportUnknownEndpoint`</strong>: tracks requests to unknown endpoints.
+* *`BulkImportPing`*: tracks `GET` requests to the `/ping` endpoint, which allows us to make sure the bulk import backend is up and running.
+* *`BulkImportFindAllOrganizations`*: tracks `GET` requests to the /organizations endpoint, which returns the list of organizations accessible from all configured GitHub Integrations.
+* *`BulkImportFindRepositoriesByOrganization`*: tracks `GET` requests to the `/organizations/:orgName/repositories` endpoint, which returns the list of repositories for the specified organization (accessible from any of the configured GitHub Integrations).
+* *`BulkImportFindAllRepositories`*: tracks GET requests to the `/repositories` endpoint, which returns the list of repositories accessible from all configured GitHub Integrations.
+* *`BulkImportFindAllImports`*: tracks `GET` requests to the `/imports` endpoint, which returns the list of existing import jobs along with their statuses.
+* *`BulkImportCreateImportJobs`*: tracks `POST` requests to the `/imports` endpoint, which allows to submit requests to bulk-import one or many repositories into the Developer Hub Catalog, by eventually creating import Pull Requests in the target repositories.
+* *`BulkImportFindImportStatusByRepo`*: tracks `GET` requests to the `/import/by-repo` endpoint, which fetches details about the import job for the specified repository.
+* *`BulkImportDeleteImportByRepo`*: tracks `DELETE` requests to the `/import/by-repo` endpoint, which deletes any existing import job for the specified repository, by closing any open import Pull Request that could have been created.
+
+.Example
+[code,javascript]
+----
+{
+  "actor": {
+    "actorId": "user:default/myuser",
+    "hostname": "localhost",
+    "ip": "::1",
+    "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
+  },
+  "eventName": "BulkImportFindAllOrganizations",
+  "isAuditLog": true,
+  "level": "info",
+  "message": "'get /organizations' endpoint hit by user:default/myuser",
+  "meta": {},
+  "plugin": "bulk-import",
+  "request": {
+    "body": {},
+    "method": "GET",
+    "params": {},
+    "query": {
+      "pagePerIntegration": "1",
+      "sizePerIntegration": "5"
+    },
+    "url": "/api/bulk-import/organizations?pagePerIntegration=1&sizePerIntegration=5"
+  },
+  "response": {
+    "status": 200
+  },
+  "service": "backstage",
+  "stage": "completion",
+  "status": "succeeded",
+  "timestamp": "2024-08-26 16:41:02"
+}
+----
diff --git a/modules/importing-repositories/procedure-understanding-bulk-import-limitations.adoc b/modules/importing-repositories/procedure-understanding-bulk-import-limitations.adoc
@@ -0,0 +1,4 @@
+= Bulk Import limitations
+
+* Only the first 20 repositories (in alphabetical order) can be displayed at most on the Added Repositories page. Also, the count of Added Repositories displayed might be wrong. In future releases, we plan to address this with proper pagination. Meanwhile, as a workaround, searching would still work against all Added Repositories. So you can still search any Added Repository and get it listed on the table. See https://issues.redhat.com/browse/RHIDP-4067[RHIDP-4067].
+* Repositories might be added to Developer Hub from various sources (like statically in an app-config file or dynamically when https://docs.redhat.com/en/documentation/red_hat_developer_hub/1.2/html/getting_started_with_red_hat_developer_hub/ref-rhdh-supported-configs_rhdh-getting-started#enabling-github-discovery-in-red-hat-developer-hub[enabling GitHub discovery]). By design, the bulk import plugin will only track repositories that are accessible from the configured GitHub integrations. However, deleting a repository from the Bulk Imports listing page may not work if the repository was discovered by the GitHub discovery plugin. See https://issues.redhat.com/browse/RHIDP-3931[RHIDP-3931].
