Add support of remote task on remote Pipeline

We now support remote tasks on remote Pipeline, allowing to share
a remote Pipeline across multiple repositories.

User can override tasks from the remote pipeline by adding a task with
the same name.

Signed-off-by: Chmouel Boudjnah <[email protected]>

Author: chmouel

docs/content/docs/guide/resolver.md

@@ -46,7 +46,7 @@ command to learn on how to use it.
 location with annotations on PipelineRun.
 
 If the resolver sees a PipelineRun referencing a remote task or a Pipeline in
-a PipelineRun or a PipelineSpec it will automatically inlines them.
+a PipelineRun or a PipelineSpec it will automatically inline them.
 
 If multiple annotations reference the same task name the resolver will pick the
 first one fetched from the annotations.
@@ -128,18 +128,19 @@ will fetch the task directly from that remote URL :
 ### Remote HTTP URL from a private GitHub repository
 
 If you are using `GitHub` and If the remote task URL uses the same host as where
-the repo CRD is, PAC will use the  GitHub token and fetch the URL using the
+the repository CRD is, PAC will use the  GitHub token and fetch the URL using the
 GitHub API.
 
-For example if you have a repo URL looking like this :
+For example if you have a repository URL looking like this :
 
 <https://github.com/organization/repository>
 
 and the remote HTTP URLs is a referenced GitHub "blob" URL:
 
 <https://github.com/organization/repository/blob/mainbranch/path/file>
 
-if the remote HTTP url has a slash (/) in the branch name you will need to html encode with the `%2F` character, eg:
+if the remote HTTP URL has a slash (/) in the branch name you will need to HTML
+encode with the `%2F` character, example:
 
 <https://github.com/organization/repository/blob/feature%2Fmainbranch/path/file>
 
@@ -150,14 +151,14 @@ GitHub app token are scoped to the owner or organization where the repository is
 If you are using the GitHub webhook method you are able to fetch any private or
 public repositories on any organization where the personal token is allowed.
 
-There is settings you can set in the pac `Configmap` to control that behavior, see the
+There is settings you can set in the pac `Configmap` to control that behaviour, see the
 `secret-github-app-token-scoped` and `secret-github-app-scope-extra-repos` settings in the
 [settings documentation](/docs/install/settings).
 
 ### Tasks or Pipelines inside the repository
 
 Additionally, you can as well have a reference to a task or pipeline from a YAML file inside
-your repo if you specify the relative path to it, for example :
+your repository if you specify the relative path to it, for example :
 
 ```yaml
 pipelinesascode.tekton.dev/task: "[share/tasks/git-clone.yaml]"
@@ -174,18 +175,77 @@ If the object fetched cannot be parsed as a Tekton `Task` it will error out.
 
 ## Remote Pipeline annotations
 
-Remote Pipeline can be referenced by annotation, this allows you to share your Pipeline definition across.
+Remote Pipeline can be referenced by annotation, allowing you to share a Pipeline across multiple repositories.
 
-Only one Pipeline is allowed in annotation.
+Only one Pipeline is allowed on the `PipelineRun` annotation.
 
-An annotation to a remote pipeline looks like this :
+An annotation to a remote pipeline looks like this, using a remote URL:
 
 ```yaml
 pipelinesascode.tekton.dev/pipeline: "https://git.provider/raw/pipeline.yaml
 ```
 
-It supports remote URL and files inside the same Git repository.
+or from a relative path inside the repository:
+
+```yaml
+pipelinesascode.tekton.dev/pipeline: "./tasks/pipeline.yaml
+```
+
+Fetching `Pipelines` from the [Tekton Hub](https://hub.tekton.dev) is not currently supported.
+
+### Overriding tasks from a remote pipeline on a PipelineRun
+
+Remote task annotations on the remote pipeline are supported. No other
+annotations like `on-target-branch`, `on-event` or `on-cel-expression` are
+supported.
+
+If a user wants to override one of the tasks from the remote pipeline, they can do
+so by adding a task in the annotations that has the same name In their `PipelineRun` annotations.
+
+For example if the user PipelineRun contains those annotations:
+
+```yaml
+kind: PipelineRun
+metadata:
+  annotations:
+    pipelinesascode.tekton.dev/pipeline: "https://git.provider/raw/pipeline.yaml
+    pipelinesascode.tekton.dev/task: "./my-git-clone-task.yaml
+```
+
+and the Pipeline referenced by the `pipelinesascode.tekton.dev/pipeline` annotation
+in "<https://git.provider/raw/pipeline.yaml>"  contains those annotations:
+
+```yaml
+kind: Pipeline
+metadata:
+  annotations:
+    pipelinesascode.tekton.dev/task: "git-clone"
+```
+
+In this case if the `my-git-clone-task.yaml` file in the root directory is a
+task named `git-clone` it will be used instead of the `git-clone` on the remote
+pipeline that is coming from the Tekon Hub.
 
 {{< hint info >}}
-[Tekton Hub](https://hub.tekton.dev) doesn't currently have support for `Pipeline`.
+Task overriding is only supported for tasks that are referenced by a `taskRef`
+to a `Name`, no override is done on `Tasks` embedded with a `taskSpec`. See
+[Tekton documentation](https://tekton.dev/docs/pipelines/pipelines/#adding-tasks-to-the-pipeline) for the differences between `taskRef` and `taskSpec`:
 {{< /hint >}}
+
+### Tasks or Pipelines Precedence
+
+From where tasks or pipelines of the same name takes precedence?
+
+for remote Tasks, when you have a `taskRef` on a task name, pac will try to find the task in this order:
+
+1. A task matched from the PipelineRun annotations
+2. A task matched from the remote Pipeline annotations
+3. A task matched fetched from the Tekton directory
+   (the tasks from the `.tekton` directory and its subdirs are automatically included)
+
+for remote Pipelines referenced on a `pipelineRef`, pac will try to match a
+pipeline in this order:
+
+1. Pipeline from the PipelineRun annotations
+2. Pipeline from the Tekton directory (pipelines are automatically fetched from
+   the `.tekton` directory and subdirs)

pkg/action/patch_test.go

@@ -22,7 +22,7 @@ func TestPatchPipelineRun(t *testing.T) {
 	observer, _ := zapobserver.New(zap.InfoLevel)
 	logger := zap.New(observer).Sugar()
 
-	testPR := tektontest.MakePR("namespace", "force-me", []pipelinev1.ChildStatusReference{
+	testPR := tektontest.MakePRStatus("namespace", "force-me", []pipelinev1.ChildStatusReference{
 		tektontest.MakeChildStatusReference("first"),
 		tektontest.MakeChildStatusReference("last"),
 		tektontest.MakeChildStatusReference("middle"),

pkg/matcher/annotation_tasks_install.go

@@ -216,7 +216,7 @@ func (rt RemoteTasks) GetTaskFromAnnotations(ctx context.Context, annotations ma
 
 // GetPipelineFromAnnotations Get pipeline remotely if they are on Annotations
 // TODO: merge in a generic between the two
-func (rt RemoteTasks) GetPipelineFromAnnotations(ctx context.Context, annotations map[string]string) ([]*tektonv1.Pipeline, error) {
+func (rt RemoteTasks) GetPipelineFromAnnotations(ctx context.Context, annotations map[string]string) (*tektonv1.Pipeline, error) {
 	ret := []*tektonv1.Pipeline{}
 	pipelinesAnnotation, err := grabValuesFromAnnotations(annotations, pipelineAnnotationsRegexp)
 	if err != nil {
@@ -225,6 +225,9 @@ func (rt RemoteTasks) GetPipelineFromAnnotations(ctx context.Context, annotation
 	if len(pipelinesAnnotation) > 1 {
 		return nil, fmt.Errorf("only one pipeline is allowed on remote resolution, we have received multiple of them: %+v", pipelinesAnnotation)
 	}
+	if len(pipelinesAnnotation) == 0 {
+		return nil, nil
+	}
 	for _, v := range pipelinesAnnotation {
 		data, err := rt.getRemote(ctx, v, false)
 		if err != nil {
@@ -239,7 +242,7 @@ func (rt RemoteTasks) GetPipelineFromAnnotations(ctx context.Context, annotation
 		}
 		ret = append(ret, pipeline)
 	}
-	return ret, nil
+	return ret[0], nil
 }
 
 // getTaskFromLocalFS get task locally if file exist

pkg/matcher/annotation_tasks_install_test.go

@@ -463,10 +463,10 @@ func TestGetPipelineFromAnnotations(t *testing.T) {
 				assert.Assert(t, len(fakelog.FilterMessageSnippet(tt.wantLog).TakeAll()) > 0, "could not find log message: got ", fakelog)
 			}
 			assert.NilError(t, err)
-			assert.Assert(t, len(got) > 0, "GetPipelineFromAnnotations() error no pipelines has been processed")
+			assert.Assert(t, got != nil, "GetPipelineFromAnnotations() error no pipelines has been processed")
 
 			if tt.gotPipelineName != "" {
-				assert.Equal(t, tt.gotPipelineName, got[0].GetName())
+				assert.Equal(t, tt.gotPipelineName, got.GetName())
 			}
 		})
 	}

pkg/pipelineascode/pipelineascode_test.go

@@ -517,7 +517,7 @@ func TestRun(t *testing.T) {
 				},
 				Repositories: tt.repositories,
 				PipelineRuns: []*pipelinev1.PipelineRun{
-					tektontest.MakePR("namespace", "force-me", []pipelinev1.ChildStatusReference{
+					tektontest.MakePRStatus("namespace", "force-me", []pipelinev1.ChildStatusReference{
 						tektontest.MakeChildStatusReference("first"),
 						tektontest.MakeChildStatusReference("last"),
 						tektontest.MakeChildStatusReference("middle"),

pkg/reconciler/controller_test.go

@@ -33,7 +33,7 @@ func TestCheckStateAndEnqueue(t *testing.T) {
 	})
 
 	// Create a new PipelineRun object with the "started" state label.
-	testPR := tektontest.MakePR("namespace", "force-me", []pipelinev1.ChildStatusReference{
+	testPR := tektontest.MakePRStatus("namespace", "force-me", []pipelinev1.ChildStatusReference{
 		tektontest.MakeChildStatusReference("first"),
 		tektontest.MakeChildStatusReference("last"),
 		tektontest.MakeChildStatusReference("middle"),

pkg/resolve/remote.go

@@ -0,0 +1,105 @@
+package resolve
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/openshift-pipelines/pipelines-as-code/pkg/matcher"
+)
+
+type NamedItem interface {
+	GetName() string
+}
+
+func alreadySeen[T NamedItem](items []T, item T) bool {
+	for _, value := range items {
+		if value.GetName() == item.GetName() {
+			return true
+		}
+	}
+	return false
+}
+
+// getRemotes will get remote tasks or Pipelines from annotations.
+//
+// It already has some tasks or pipeline coming from the tekton directory stored in [types]
+//
+// The precedence logic for tasks is in this order:
+//
+// * Tasks from the PipelineRun annotations
+// * Tasks from the Pipeline annotations
+// * Tasks from the Tekton directory
+//
+// The precedence logic for Pipeline is first from PipelineRun annotations and
+// then from Tekton directory
+func getRemotes(ctx context.Context, rt *matcher.RemoteTasks, types TektonTypes) (TektonTypes, error) {
+	remoteType := &TektonTypes{}
+	for _, pipelinerun := range types.PipelineRuns {
+		if len(pipelinerun.GetObjectMeta().GetAnnotations()) == 0 {
+			continue
+		}
+
+		// get first all the tasks from the pipelinerun annotations
+		remoteTasks, err := rt.GetTaskFromAnnotations(ctx, pipelinerun.GetObjectMeta().GetAnnotations())
+		if err != nil {
+			return TektonTypes{}, fmt.Errorf("error getting remote task from pipelinerun annotations: %w", err)
+		}
+
+		for _, task := range remoteTasks {
+			if alreadySeen(remoteType.Tasks, task) {
+				rt.Logger.Infof("skipping duplicated task %s in annotations on pipelinerun %s", task.GetName(), pipelinerun.GetName())
+				continue
+			}
+			remoteType.Tasks = append(remoteType.Tasks, task)
+		}
+
+		// get the pipeline from the remote annotation if any
+		remotePipeline, err := rt.GetPipelineFromAnnotations(ctx, pipelinerun.GetObjectMeta().GetAnnotations())
+		if err != nil {
+			return TektonTypes{}, fmt.Errorf("error getting remote pipeline from pipelinerun annotation: %w", err)
+		}
+
+		if remotePipeline != nil {
+			remoteType.Pipelines = append(remoteType.Pipelines, remotePipeline)
+		}
+	}
+
+	// grab the tasks from the remote pipeline
+	for _, pipeline := range remoteType.Pipelines {
+		if pipeline.GetObjectMeta().GetAnnotations() == nil {
+			continue
+		}
+		remoteTasks, err := rt.GetTaskFromAnnotations(ctx, pipeline.GetObjectMeta().GetAnnotations())
+		if err != nil {
+			return TektonTypes{}, fmt.Errorf("error getting remote tasks from remote pipeline %s: %w", pipeline.GetName(), err)
+		}
+
+		for _, remoteTask := range remoteTasks {
+			if alreadySeen(remoteType.Tasks, remoteTask) {
+				rt.Logger.Infof("skipping remote task %s from remote pipeline %s as already defined in pipelinerun", remoteTask.GetName(), pipeline.GetName())
+				continue
+			}
+			remoteType.Tasks = append(remoteType.Tasks, remoteTask)
+		}
+	}
+
+	ret := TektonTypes{
+		PipelineRuns: types.PipelineRuns,
+	}
+	// first get the remote types and then the local ones so remote takes precedence
+	for _, task := range append(remoteType.Tasks, types.Tasks...) {
+		if alreadySeen(ret.Tasks, task) {
+			rt.Logger.Infof("overriding task %s coming from tekton directory by an annotation task on the pipeline or pipelinerun", task.GetName())
+			continue
+		}
+		ret.Tasks = append(ret.Tasks, task)
+	}
+	for _, remotePipeline := range append(remoteType.Pipelines, types.Pipelines...) {
+		if alreadySeen(ret.Pipelines, remotePipeline) {
+			rt.Logger.Infof("overriding pipeline %s coming from tekton directory by the annotation pipelinerun", remotePipeline.GetName())
+			continue
+		}
+		ret.Pipelines = append(ret.Pipelines, remotePipeline)
+	}
+	return ret, nil
+}