Skip to content

Commit ef96027

Browse files
github-actions[bot]JohnTitorkj455t-kikuc
authored
Cherry-pick #5523 #5540 #5558 #5571 (#5573)
* Correct notification routing for `DEPLOYMENT_STARTED` (#5523) * Correct notification routing for `DEPLOYMENT_STARTED` Signed-off-by: Yuki Okushi <okushi@canary-inc.jp> * Harden test case Signed-off-by: Yuki Okushi <okushi@canary-inc.jp> --------- Signed-off-by: Yuki Okushi <okushi@canary-inc.jp> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * Sort results of plan-preview (#5540) * Sort results of plan-preview Signed-off-by: kj455 <kaji.ibuki45@gmail.com> * Ensure the order of list piped Signed-off-by: kj455 <kaji.ibuki45@gmail.com> * fix: lint Signed-off-by: kj455 <kaji.ibuki45@gmail.com> * fix: move sorting to pipectl Signed-off-by: kj455 <kaji.ibuki45@gmail.com> * fix: add testcase Signed-off-by: kj455 <kaji.ibuki45@gmail.com> * fix: dev docs Signed-off-by: kj455 <kaji.ibuki45@gmail.com> * add docs Signed-off-by: kj455 <kaji.ibuki45@gmail.com> --------- Signed-off-by: kj455 <kaji.ibuki45@gmail.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * Enhanced EventWatcher logs (#5558) * Show push error log earlier than reporting Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> * Use WarnLog in retry Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> * clarify log messages Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> * clarify log messages Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> * add TestDoCalls for asserting counts Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> * add eventIDs in log Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> * enrich logs in updateValues Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> * nits Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> * Revert "add TestDoCalls for asserting counts" This reverts commit de3f112. Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> --------- Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * update RELEASE to v0.50.2 with doc update (#5571) Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> --------- Signed-off-by: Yuki Okushi <okushi@canary-inc.jp> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> Signed-off-by: kj455 <kaji.ibuki45@gmail.com> Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> Co-authored-by: Yuki Okushi <okushi@canary-inc.jp> Co-authored-by: Ibuki Kaji <38521709+kj455@users.noreply.github.com> Co-authored-by: Tetsuya KIKUCHI <97105818+t-kikuc@users.noreply.github.com>
1 parent 8ff25b1 commit ef96027

File tree

8 files changed

+244
-13
lines changed

8 files changed

+244
-13
lines changed

‎RELEASE‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Generated by `make release` command.
22
# DO NOT EDIT.
3-
tag: v0.50.1
3+
tag: v0.50.2
44

55
releaseNoteGenerator:
66
showCommitter: false

‎docs/content/en/docs-dev/user-guide/plan-preview.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,8 @@ pipectl plan-preview \
3838
--repo-remote-url={ REPO_REMOTE_GIT_SSH_URL } \
3939
--head-branch={ HEAD_BRANCH } \
4040
--head-commit={ HEAD_COMMIT } \
41-
--base-branch={ BASE_BRANCH }
41+
--base-branch={ BASE_BRANCH } \
42+
--sort-label-keys={ SORT_LABEL_KEYS }
4243
```
4344

4445
You can run it locally or integrate it to your CI system to run automatically when a new pull request is opened/updated. Use `--help` to see more options.
@@ -47,6 +48,13 @@ You can run it locally or integrate it to your CI system to run automatically wh
4748
pipectl plan-preview --help
4849
```
4950

51+
### Order of the results
52+
53+
By default, the results are sorted by PipedID and Application Name.
54+
55+
If you want to sort the results by labels, add `--sort-label-keys` option. For example, when you run with `--sort-label-keys=env,team`, the results will be sorted by PipedID, `env` label, `team` label, and then Application Name.
56+
57+
5058
## GitHub Actions
5159

5260
If you are using GitHub Actions, you can seamlessly integrate our prepared [actions-plan-preview](https://github.com/pipe-cd/actions-plan-preview) to your workflows. This automatically comments the plan-preview result on the pull request when it is opened or updated. You can also trigger to run plan-preview manually by leave a comment `/pipecd plan-preview` on the pull request.

‎docs/content/en/docs-v0.50.x/user-guide/plan-preview.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,8 @@ pipectl plan-preview \
3838
--repo-remote-url={ REPO_REMOTE_GIT_SSH_URL } \
3939
--head-branch={ HEAD_BRANCH } \
4040
--head-commit={ HEAD_COMMIT } \
41-
--base-branch={ BASE_BRANCH }
41+
--base-branch={ BASE_BRANCH } \
42+
--sort-label-keys={ SORT_LABEL_KEYS }
4243
```
4344

4445
You can run it locally or integrate it to your CI system to run automatically when a new pull request is opened/updated. Use `--help` to see more options.
@@ -47,6 +48,13 @@ You can run it locally or integrate it to your CI system to run automatically wh
4748
pipectl plan-preview --help
4849
```
4950

51+
### Order of the results
52+
53+
By default, the results are sorted by PipedID and Application Name.
54+
55+
If you want to sort the results by labels, add `--sort-label-keys` option. For example, when you run with `--sort-label-keys=env,team`, the results will be sorted by PipedID, `env` label, `team` label, and then Application Name.
56+
57+
5058
## GitHub Actions
5159

5260
If you are using GitHub Actions, you can seamlessly integrate our prepared [actions-plan-preview](https://github.com/pipe-cd/actions-plan-preview) to your workflows. This automatically comments the plan-preview result on the pull request when it is opened or updated. You can also trigger to run plan-preview manually by leave a comment `/pipecd plan-preview` on the pull request.

‎pkg/app/pipectl/cmd/planpreview/planpreview.go‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ import (
2020
"fmt"
2121
"io"
2222
"os"
23+
"sort"
2324
"strings"
2425
"time"
2526

@@ -49,6 +50,7 @@ type command struct {
4950
timeout time.Duration
5051
pipedHandleTimeout time.Duration
5152
checkInterval time.Duration
53+
sortLabelKeys []string
5254

5355
clientOptions *client.Options
5456
}
@@ -75,6 +77,7 @@ func NewCommand() *cobra.Command {
7577
cmd.Flags().StringVar(&c.out, "out", c.out, "Write planpreview result to the given path.")
7678
cmd.Flags().DurationVar(&c.timeout, "timeout", c.timeout, "Maximum amount of time this command has to complete. Default is 10m.")
7779
cmd.Flags().DurationVar(&c.pipedHandleTimeout, "piped-handle-timeout", c.pipedHandleTimeout, "Maximum amount of time that a Piped can take to handle. Default is 5m.")
80+
cmd.Flags().StringSliceVar(&c.sortLabelKeys, "sort-label-keys", c.sortLabelKeys, "The application label keys to sort the results by. If not specified, the results will be sorted by only PipedID and ApplicationName.")
7881

7982
cmd.MarkFlagRequired("repo-remote-url")
8083
cmd.MarkFlagRequired("head-branch")
@@ -147,11 +150,32 @@ func (c *command) run(ctx context.Context, _ cli.Input) error {
147150
fmt.Printf("Failed to retrieve plan-preview results: %v\n", err)
148151
return err
149152
}
153+
sortResults(results, c.sortLabelKeys)
150154
return printResults(results, os.Stdout, c.out)
151155
}
152156
}
153157
}
154158

159+
// sortResults sorts the given results by pipedID and the given sortLabelKeys.
160+
// If sortLabelKeys is not specified or the all values of sortLabelKeys are the same, it sorts by pipedID and ApplicationName.
161+
func sortResults(allResults []*model.PlanPreviewCommandResult, sortLabelKeys []string) {
162+
sort.SliceStable(allResults, func(i, j int) bool {
163+
return allResults[i].PipedId < allResults[j].PipedId
164+
})
165+
for _, resultsPerPiped := range allResults {
166+
results := resultsPerPiped.Results
167+
sort.SliceStable(results, func(i, j int) bool {
168+
a, b := results[i], results[j]
169+
for _, key := range sortLabelKeys {
170+
if a.Labels[key] != b.Labels[key] {
171+
return a.Labels[key] < b.Labels[key]
172+
}
173+
}
174+
return a.ApplicationName < b.ApplicationName
175+
})
176+
}
177+
}
178+
155179
func printResults(results []*model.PlanPreviewCommandResult, stdout io.Writer, outFile string) error {
156180
r := convert(results)
157181

‎pkg/app/pipectl/cmd/planpreview/planpreview_test.go‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -244,3 +244,123 @@ NOTE: An error occurred while building plan-preview for applications of the foll
244244
})
245245
}
246246
}
247+
func TestSortResults(t *testing.T) {
248+
t.Parallel()
249+
testcases := []struct {
250+
name string
251+
results []*model.PlanPreviewCommandResult
252+
sortLabelKeys []string
253+
expected []*model.PlanPreviewCommandResult
254+
}{
255+
{
256+
name: "sort by pipedID and application name",
257+
results: []*model.PlanPreviewCommandResult{
258+
{
259+
PipedId: "piped-2",
260+
Results: []*model.ApplicationPlanPreviewResult{
261+
{ApplicationName: "app-2"},
262+
{ApplicationName: "app-1"},
263+
},
264+
},
265+
{
266+
PipedId: "piped-1",
267+
Results: []*model.ApplicationPlanPreviewResult{
268+
{ApplicationName: "app-2"},
269+
{ApplicationName: "app-1"},
270+
},
271+
},
272+
},
273+
sortLabelKeys: []string{},
274+
expected: []*model.PlanPreviewCommandResult{
275+
{
276+
PipedId: "piped-1",
277+
Results: []*model.ApplicationPlanPreviewResult{
278+
{ApplicationName: "app-1"},
279+
{ApplicationName: "app-2"},
280+
},
281+
},
282+
{
283+
PipedId: "piped-2",
284+
Results: []*model.ApplicationPlanPreviewResult{
285+
{ApplicationName: "app-1"},
286+
{ApplicationName: "app-2"},
287+
},
288+
},
289+
},
290+
},
291+
{
292+
name: "sort by label keys",
293+
results: []*model.PlanPreviewCommandResult{
294+
{
295+
PipedId: "piped-1",
296+
Results: []*model.ApplicationPlanPreviewResult{
297+
{ApplicationName: "app-1", Labels: map[string]string{"env": "staging"}},
298+
{ApplicationName: "app-1", Labels: map[string]string{"env": "prod"}},
299+
},
300+
},
301+
{
302+
PipedId: "piped-2",
303+
Results: []*model.ApplicationPlanPreviewResult{
304+
{ApplicationName: "app-3", Labels: map[string]string{"env": "staging"}},
305+
{ApplicationName: "app-3", Labels: map[string]string{"env": "prod"}},
306+
{ApplicationName: "app-2", Labels: map[string]string{"env": "staging"}},
307+
{ApplicationName: "app-2", Labels: map[string]string{"env": "prod"}},
308+
},
309+
},
310+
},
311+
sortLabelKeys: []string{"env"},
312+
expected: []*model.PlanPreviewCommandResult{
313+
{
314+
PipedId: "piped-1",
315+
Results: []*model.ApplicationPlanPreviewResult{
316+
{ApplicationName: "app-1", Labels: map[string]string{"env": "prod"}},
317+
{ApplicationName: "app-1", Labels: map[string]string{"env": "staging"}},
318+
},
319+
},
320+
{
321+
PipedId: "piped-2",
322+
Results: []*model.ApplicationPlanPreviewResult{
323+
{ApplicationName: "app-2", Labels: map[string]string{"env": "prod"}},
324+
{ApplicationName: "app-3", Labels: map[string]string{"env": "prod"}},
325+
{ApplicationName: "app-2", Labels: map[string]string{"env": "staging"}},
326+
{ApplicationName: "app-3", Labels: map[string]string{"env": "staging"}},
327+
},
328+
},
329+
},
330+
},
331+
{
332+
name: "sort by multiple label keys",
333+
results: []*model.PlanPreviewCommandResult{
334+
{
335+
PipedId: "piped-1",
336+
Results: []*model.ApplicationPlanPreviewResult{
337+
{ApplicationName: "app-1", Labels: map[string]string{"env": "prod", "team": "team-2"}},
338+
{ApplicationName: "app-1", Labels: map[string]string{"env": "staging", "team": "team-1"}},
339+
{ApplicationName: "app-1", Labels: map[string]string{"env": "prod", "team": "team-1"}},
340+
{ApplicationName: "app-2", Labels: map[string]string{"env": "prod", "team": "team-2"}},
341+
},
342+
},
343+
},
344+
sortLabelKeys: []string{"env", "team"},
345+
expected: []*model.PlanPreviewCommandResult{
346+
{
347+
PipedId: "piped-1",
348+
Results: []*model.ApplicationPlanPreviewResult{
349+
{ApplicationName: "app-1", Labels: map[string]string{"env": "prod", "team": "team-1"}},
350+
{ApplicationName: "app-1", Labels: map[string]string{"env": "prod", "team": "team-2"}},
351+
{ApplicationName: "app-2", Labels: map[string]string{"env": "prod", "team": "team-2"}},
352+
{ApplicationName: "app-1", Labels: map[string]string{"env": "staging", "team": "team-1"}},
353+
},
354+
},
355+
},
356+
},
357+
}
358+
359+
for _, tc := range testcases {
360+
t.Run(tc.name, func(t *testing.T) {
361+
t.Parallel()
362+
sortResults(tc.results, tc.sortLabelKeys)
363+
assert.Equal(t, tc.expected, tc.results)
364+
})
365+
}
366+
}

‎pkg/app/piped/eventwatcher/eventwatcher.go‎

Lines changed: 35 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -183,7 +183,7 @@ func (w *watcher) run(ctx context.Context, repo git.Repo, repoCfg config.PipedRe
183183
case <-ticker.C:
184184
err := repo.Pull(ctx, repo.GetClonedBranch())
185185
if err != nil {
186-
w.logger.Error("failed to perform git pull",
186+
w.logger.Error("failed to perform git pull. will retry in the next loop",
187187
zap.String("repo-id", repoCfg.RepoID),
188188
zap.String("branch", repo.GetClonedBranch()),
189189
zap.Error(err),
@@ -233,6 +233,7 @@ func (w *watcher) run(ctx context.Context, repo git.Repo, repoCfg config.PipedRe
233233
if err := w.updateValues(ctx, repo, repoCfg.RepoID, cfg.Events, commitMsg); err != nil {
234234
w.logger.Error("failed to update the values",
235235
zap.String("repo-id", repoCfg.RepoID),
236+
zap.String("branch", repo.GetClonedBranch()),
236237
zap.Error(err),
237238
)
238239
}
@@ -294,6 +295,7 @@ func (w *watcher) run(ctx context.Context, repo git.Repo, repoCfg config.PipedRe
294295
if err := w.execute(ctx, repo, repoCfg.RepoID, cfgs); err != nil {
295296
w.logger.Error("failed to execute the event from application configuration",
296297
zap.String("repo-id", repoCfg.RepoID),
298+
zap.String("branch", repo.GetClonedBranch()),
297299
zap.Error(err),
298300
)
299301
}
@@ -456,28 +458,40 @@ func (w *watcher) execute(ctx context.Context, repo git.Repo, repoID string, eve
456458
var responseError error
457459
retry := backoff.NewRetry(retryPushNum, backoff.NewConstant(retryPushInterval))
458460
for branch, events := range branchHandledEvents {
461+
eventIDs := make([]string, 0, len(events))
462+
for _, e := range events {
463+
eventIDs = append(eventIDs, e.Id)
464+
}
465+
zlogger := w.logger.With(
466+
zap.String("repo-id", repoID),
467+
zap.String("branch", tmpRepo.GetClonedBranch()),
468+
zap.Strings("event-ids", eventIDs),
469+
)
470+
459471
_, err = retry.Do(ctx, func() (interface{}, error) {
460472
if err := tmpRepo.Push(ctx, branch); err != nil {
461-
w.logger.Error("failed to push commits", zap.String("repo-id", repoID), zap.String("branch", branch), zap.Error(err))
473+
zlogger.Warn(fmt.Sprintf("failed to push commits. retry attempt %d/%d", retry.Calls(), retryPushNum), zap.Error(err))
462474
return nil, err
463475
}
464476
return nil, nil
465477
})
466478

467479
if err == nil {
468480
if _, err := w.apiClient.ReportEventStatuses(ctx, &pipedservice.ReportEventStatusesRequest{Events: events}); err != nil {
469-
w.logger.Error("failed to report event statuses", zap.Error(err))
481+
zlogger.Error("failed to report event statuses", zap.Error(err))
470482
}
471483
w.executionMilestoneMap.Store(repoID, maxTimestamp)
472484
continue
473485
}
474486

475487
// If push fails because the local branch was not fresh, exit to retry again in the next interval.
476488
if err == git.ErrBranchNotFresh {
477-
w.logger.Warn("failed to push commits", zap.Error(err))
489+
zlogger.Warn("failed to push commits. local branch was not up-to-date. will retry in the next loop", zap.Error(err))
478490
continue
479491
}
480492

493+
zlogger.Error("failed to push commits", zap.Error(err))
494+
481495
// If push fails because of the other reason, re-set all statuses to FAILURE.
482496
for i := range events {
483497
if events[i].Status == model.EventStatus_EVENT_FAILURE {
@@ -487,7 +501,7 @@ func (w *watcher) execute(ctx context.Context, repo git.Repo, repoID string, eve
487501
events[i].StatusDescription = fmt.Sprintf("Failed to push changed files: %v", err)
488502
}
489503
if _, err := w.apiClient.ReportEventStatuses(ctx, &pipedservice.ReportEventStatusesRequest{Events: events}); err != nil {
490-
w.logger.Error("failed to report event statuses", zap.Error(err))
504+
zlogger.Error("failed to report event statuses", zap.Error(err))
491505
}
492506
w.executionMilestoneMap.Store(repoID, maxTimestamp)
493507
responseError = errors.Join(responseError, err)
@@ -600,17 +614,27 @@ func (w *watcher) updateValues(ctx context.Context, repo git.Repo, repoID string
600614
return nil
601615
}
602616

617+
eventIDs := make([]string, 0, len(handledEvents))
618+
for _, e := range handledEvents {
619+
eventIDs = append(eventIDs, e.Id)
620+
}
621+
zlogger := w.logger.With(
622+
zap.String("repo-id", repoID),
623+
zap.String("branch", tmpRepo.GetClonedBranch()),
624+
zap.Strings("event-ids", eventIDs),
625+
)
626+
603627
retry := backoff.NewRetry(retryPushNum, backoff.NewConstant(retryPushInterval))
604628
_, err = retry.Do(ctx, func() (interface{}, error) {
605629
if err := tmpRepo.Push(ctx, tmpRepo.GetClonedBranch()); err != nil {
606-
w.logger.Error("failed to push commits", zap.String("repo-id", repoID), zap.String("branch", tmpRepo.GetClonedBranch()), zap.Error(err))
630+
zlogger.Warn(fmt.Sprintf("failed to push commits. retry attempt %d/%d", retry.Calls(), retryPushNum), zap.Error(err))
607631
return nil, err
608632
}
609633
return nil, nil
610634
})
611635
if err == nil {
612636
if _, err := w.apiClient.ReportEventStatuses(ctx, &pipedservice.ReportEventStatusesRequest{Events: handledEvents}); err != nil {
613-
w.logger.Error("failed to report event statuses", zap.Error(err))
637+
zlogger.Error("failed to report event statuses", zap.Error(err))
614638
return err
615639
}
616640
w.milestoneMap.Store(repoID, maxTimestamp)
@@ -619,10 +643,12 @@ func (w *watcher) updateValues(ctx context.Context, repo git.Repo, repoID string
619643

620644
// If push fails because the local branch was not fresh, exit to retry again in the next interval.
621645
if err == git.ErrBranchNotFresh {
622-
w.logger.Warn("failed to push commits", zap.Error(err))
646+
zlogger.Warn("failed to push commits. local branch was not up-to-date. will retry in the next loop", zap.Error(err))
623647
return nil
624648
}
625649

650+
zlogger.Error("failed to push commits", zap.Error(err))
651+
626652
// If push fails because of the other reason, re-set all statuses to FAILURE.
627653
for i := range handledEvents {
628654
if handledEvents[i].Status == model.EventStatus_EVENT_FAILURE {
@@ -632,11 +658,10 @@ func (w *watcher) updateValues(ctx context.Context, repo git.Repo, repoID string
632658
handledEvents[i].StatusDescription = fmt.Sprintf("Failed to push changed files: %v", err)
633659
}
634660
if _, err := w.apiClient.ReportEventStatuses(ctx, &pipedservice.ReportEventStatusesRequest{Events: handledEvents}); err != nil {
635-
w.logger.Error("failed to report event statuses: %w", zap.Error(err))
661+
zlogger.Error("failed to report event statuses: %w", zap.Error(err))
636662
return err
637663
}
638664
w.milestoneMap.Store(repoID, maxTimestamp)
639-
w.logger.Error("failed to push commits", zap.Error(err))
640665
return err
641666
}
642667

0 commit comments

Comments
 (0)