RHIDP-1925 bulk import from GitHub

Signed-off-by: Fabrice Flore-Thébault <[email protected]>

Author: themr0c

assemblies/assembly-bulk-importing-from-github.adoc

@@ -0,0 +1,15 @@
+= Bulk Import GitHub repositories
+
+Bulk Import is a {company-name} *Tech Preview* feature.
+
+It allows you to automate GitHub repositories onboarding and track their import status.
+
+include::modules/importing-repositories/procedure-enabling-the-bulk-import-from-github-feature.adoc[leveloffset=+1]
+
+include::modules/importing-repositories/procedure-importing-multiple-repositories-from-github.adoc[leveloffset=+1]
+
+include::modules/importing-repositories/procedure-managing-the-imported-repository-list.adoc[leveloffset=+1]
+
+include::modules/importing-repositories/procedure-understanding-bulk-import-audit-logs.adoc[leveloffset=+1]
+
+include::modules/importing-repositories/procedure-understanding-bulk-import-limitations.adoc[leveloffset=+1]

modules/importing-repositories/procedure-enabling-the-bulk-import-from-github-feature.adoc

@@ -0,0 +1,22 @@
+[id="enabling-ang-giving-access-to-the-bulk-import-feature"]
+= Enabling and giving access to the Bulk Import feature
+
+.Procedure
+. To give {product-short} access to GitHub, link:{linkgettingstartedguide}#configuring-github-integration[configure GitHub integration].
+
+. The Bulk Import plugin is installed but disabled by default.
+Enable the `./dynamic-plugins/dist/janus-idp-backstage-plugin-bulk-import-backend-dynamic` and `./dynamic-plugins/dist/janus-idp-backstage-plugin-bulk-import` preinstalled plugins.
+See link:{plugins-configure-book-url}[{plugins-configure-book-title}].
+
+. Only {product-short} administrators or users with the `bulk.import` permission can use the Bulk Import feature.
+Configure the required `bulk.import` RBAC permission for the users that are not administrators.
+See link:{authorization-book-url}#ref-rbac-permission-policies_title-authorization[Permission policies in Red Hat Developer Hub].
+
+
+.Verification
+* The sidebar displays a *Bulk Import* line.
+* The *Bulk Import* page shows a list of *Added Repositories*.
+
+.Troubleshooting
+* If clicking on the *Bulk Import* option displays the following error message: _To view the added repositories, contact your administrator to give you the `bulk.import` permission_, it means that your user does not have the necessary permission to use the Bulk Import feature.
+Refer back to link:{authorization-book-url}#ref-rbac-permission-policies_title-authorization[Permission policies in Red Hat Developer Hub] for more information on how an administrator can add that permission to your user.

modules/importing-repositories/procedure-importing-multiple-repositories-from-github.adoc

@@ -0,0 +1,37 @@
+= Importing multiple repositories
+
+The Bulk Import feature allows you to view all the GitHub repositories and organizations accessible from the configured GitHub Integrations.
+From there, you can pick any number of repositories and onboard them.
+
+.Prerequisites
+* xref:enabling-ang-giving-access-to-the-bulk-import-feature[You enabled the Bulk Import feature].
+* xref:enabling-ang-giving-access-to-the-bulk-import-feature[Your {product-short} user has access to the Bulk Import feature].
+
+.Procedure
+. Click on *Bulk Import* in the left sidebar.
+. Click on the *Add* button in the top-right corner to see the list of all repositories accessible from the configured GitHub integrations.
+.. From the *Repositories* view, you can select any repository, or search for any accessible repositories.
+For each repository selected, a `catalog-info.yaml` will be generated.
+.. From the *Organizations* view, you can select any organization by clicking on *Select* in the third column.
+This will allow you to pick one or more repositories that are part of the selected organization.
+. Click on *Preview file* to view or edit the details of the pull request for each repository.
+The pull request details form will be preloaded with some data, and you can also preview the content of the `catalog-info.yaml`.
+If the repository already has a `.github/CODEOWNERS` file, you can select the *Use CODEOWNERS file as Entity Owner* checkbox to use it, rather than having the `content-info.yaml` contain a specific entity owner.
+. Click on *Save*.
+. Click on *Create pull requests*.
+At this point, a set of dry-run checks will be run against the selected repositories to make sure they meet the requirements to be imported, that is:
+.. Checking that there is no entity in the Developer Hub catalog with the name specified in the repository `catalog-info.yaml`
+.. Checking that the repository is not empty
+.. Checking that the repository already has a `.github/CODEOWNERS` file if the *Use CODEOWNERS file as Entity Owner* checkbox was selected for that repository.
+
+** If there are any errors, the pull requests will not be created, but you will see a _Failed to create PR_ error message indicating the failures.
+Clicking on *Edit* will show more details about the reasons.
+
+** If there are no errors, the pull requests will be created, and you will be redirected to the list of Added repositories.
+
+. Review and merge each pull request that creates a `catalog-info.yml` file.
+
+.Verification
+* The *Added repositories* list displays the repositories you just imported, each with an appropriate status: either _Waiting for approval_ or _Added_.
+* For each _Waiting for approval_ Import job listed, there is a corresponding pull request adding the `catalog-info.yaml` file in the corresponding repository
+

modules/importing-repositories/procedure-managing-the-imported-repository-list.adoc

@@ -0,0 +1,25 @@
+= Managing the list of Added repositories
+
+.Procedure
+. Click on *Bulk Import* in the left sidebar to display all the current repositories that are being tracked as Import jobs, along with their status.
+
+Added:: The repository has been added to the Developer Hub catalog.
+This can be the case after the import PR has been merged, or if the repository already had a `catalog-info.yaml` when it was bulk-imported.
+Please note that it might take a few minutes until the entities are available in the catalog.
+
+Waiting for approval:: There is an open pull request adding a `catalog-info.yaml`.
+You can:
+* Click on the *pencil icon* on the right to see details about this pull request or edit the pull request content right from {product-short}.
+* Delete the Import job, in which case the import PR will be closed as well.
+* To transition the Import job to the _Added_ state, merge the import pull request from the Git repository.
+
+Empty:: {product-short} was not able to determine the Import job status.
+This can be the case if the repository was imported from other sources but does not have any `catalog-info.yaml` file and does not have any import PR adding it.
+
+[NOTE]
+====
+* After an import pull request is merged, the Import status will be _Added_ in the list of Added Repositories, but it might take a few seconds until the corresponding entities show up in the Developer Hub Catalog.
+* A Location added through other sources (like statically in an `app-config.yaml` file, dynamically when link:{linkgettingstartedguide}#enabling-github-discovery-in-red-hat-developer-hub[enabling GitHub discovery], or registered manually via the "Register an existing component" page) may show up in the Bulk Import list of Added Repositories if both conditions are met:
+** The target repository is accessible from the configured GitHub integrations.
+** The Location URL points to a `catalog-info.yaml` file at the root of the repository default branch.
+====

modules/importing-repositories/procedure-understanding-bulk-import-audit-logs.adoc

@@ -0,0 +1,69 @@
+= Bulk Import Audit Logs
+
+The Bulk Import Backend plugin adds the events below to the Developer Hub audit logs.
+See link:{linkgettingstartedguide}#assembly-audit-log[Audit Logs in {product}] for more information on how to configure and view such audit logs.
+
+*Bulk Import Events*:
+
+`BulkImportUnknownEndpoint`::
+Tracks requests to unknown endpoints.
+
+`BulkImportPing`::
+Tracks `GET` requests to the `/ping` endpoint, which allows us to make sure the bulk import backend is up and running.
+
+`BulkImportFindAllOrganizations`::
+Tracks `GET` requests to the /organizations endpoint, which returns the list of organizations accessible from all configured GitHub Integrations.
+
+`BulkImportFindRepositoriesByOrganization`::
+Tracks `GET` requests to the `/organizations/:orgName/repositories` endpoint, which returns the list of repositories for the specified organization (accessible from any of the configured GitHub Integrations).
+
+`BulkImportFindAllRepositories`::
+Tracks GET requests to the `/repositories` endpoint, which returns the list of repositories accessible from all configured GitHub Integrations.
+
+`BulkImportFindAllImports`::
+Tracks `GET` requests to the `/imports` endpoint, which returns the list of existing import jobs along with their statuses.
+
+`BulkImportCreateImportJobs`::
+Tracks `POST` requests to the `/imports` endpoint, which allows to submit requests to bulk-import one or many repositories into the Developer Hub Catalog, by eventually creating import Pull Requests in the target repositories.
+
+`BulkImportFindImportStatusByRepo`::
+Tracks `GET` requests to the `/import/by-repo` endpoint, which fetches details about the import job for the specified repository.
+
+`BulkImportDeleteImportByRepo`::
+Tracks `DELETE` requests to the `/import/by-repo` endpoint, which deletes any existing import job for the specified repository, by closing any open import Pull Request that could have been created.
+
+.Example
+[code,json]
+----
+{
+  "actor": {
+    "actorId": "user:default/myuser",
+    "hostname": "localhost",
+    "ip": "::1",
+    "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
+  },
+  "eventName": "BulkImportFindAllOrganizations",
+  "isAuditLog": true,
+  "level": "info",
+  "message": "'get /organizations' endpoint hit by user:default/myuser",
+  "meta": {},
+  "plugin": "bulk-import",
+  "request": {
+    "body": {},
+    "method": "GET",
+    "params": {},
+    "query": {
+      "pagePerIntegration": "1",
+      "sizePerIntegration": "5"
+    },
+    "url": "/api/bulk-import/organizations?pagePerIntegration=1&sizePerIntegration=5"
+  },
+  "response": {
+    "status": 200
+  },
+  "service": "backstage",
+  "stage": "completion",
+  "status": "succeeded",
+  "timestamp": "2024-08-26 16:41:02"
+}
+----

modules/importing-repositories/procedure-understanding-bulk-import-limitations.adoc

@@ -0,0 +1,13 @@
+= Bulk Import limitations
+
+* Only the first 20 repositories (in alphabetical order) can be displayed at most on the Added Repositories page.
+Also, the count of Added Repositories displayed might be wrong.
+In future releases, we plan to address this with proper pagination.
+Meanwhile, as a workaround, searching would still work against all Added Repositories.
+So you can still search any Added Repository and get it listed on the table.
+See https://issues.redhat.com/browse/RHIDP-4067[RHIDP-4067].
+
+* Repositories might be added to {product-short} from various sources (like statically in an app-config file or dynamically when link:{linkgettingstartedguide}#enabling-github-discovery-in-red-hat-developer-hub[enabling GitHub discovery]).
+By design, the bulk import plugin will only track repositories that are accessible from the configured GitHub integrations.
+However, deleting a repository from the Bulk Imports listing page may not work if the repository was discovered by the GitHub discovery plugin.
+See https://issues.redhat.com/browse/RHIDP-3931[RHIDP-3931].