-
Notifications
You must be signed in to change notification settings - Fork 109
Feature: Moving to pure Dagger based CI/CD Pipeline from Goreleaser #547
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
5bd89b3
refactor: refactoring .dagger pipeline to allow splitting of code and…
NucleoFusion b7254b5
add: adding build command that builds to the ./dist directory
NucleoFusion a04cea2
add: adding archive function that compresses and archives the built b…
NucleoFusion 619742c
refactor: splitting code to allow a singular pipeline function withou…
NucleoFusion c47652a
feat: adding deb & rpm build using nfpm
NucleoFusion 26a4ee2
feat: Adding Homebrew Formula creation
NucleoFusion e7654d3
feat: refactor/adding Lint/Test/Coverage/Doc dagger commands
NucleoFusion 3bba7f4
feat: adding initial release logic
NucleoFusion 678ff6b
refactor: modifying github actions for dagger support
NucleoFusion e6e182f
feat: adding brew formula to release
NucleoFusion c942db3
feat: adding apt-repo support
NucleoFusion f57677f
feat: Adding checksum.txt to publish
NucleoFusion 33045ff
feat: version mismatch fix
NucleoFusion 4a21417
Delete Lintreport.json
bupd 4611878
refactor: Adding back the build-dev function and some refactoring
NucleoFusion 8b87267
feat: adding back publish-image-and-sign support
NucleoFusion 52cbc9f
fix: removing redundant code(test-release) and unsafe echo cmd
NucleoFusion ca90220
Update .dagger/buildDev.go
NucleoFusion af56c0a
feat: fixes and automatic release notes
NucleoFusion 1cb3f75
refactor: apt-repo and docker-publish (to reuse bins)
NucleoFusion 469403b
feat: adding apk support
NucleoFusion 8bdfe6f
fix: removing 'v' in version to follow standards
NucleoFusion 55fefe2
feat: modifying archive directory
NucleoFusion 19bbdb7
fix: re-adding the vuln fix
NucleoFusion File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,101 +1,103 @@ | ||
| # 🛠️ Harbor CLI Dagger Pipeline | ||
| # 🛠️ Harbor CLI — Dagger Pipeline | ||
|
|
||
| We use [Dagger](https://dagger.io) to define a CI/CD pipeline for building, linting, and publishing the [Harbor CLI](https://github.com/goharbor/harbor-cli). | ||
| This README will help beginners understand how to use Dagger in local development and CI workflows. | ||
| We use [Dagger](https://dagger.io) to define a **modular and reproducible CI/CD pipeline** for building, linting, testing, and publishing the [Harbor CLI](https://github.com/goharbor/harbor-cli). | ||
| This README provides a clear reference for contributors and maintainers to understand, run, and extend the pipeline locally or in CI. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| Before you start, ensure you have the following: | ||
|
|
||
| 1. Dagger: Install the latest version of Dagger. You can check the official documentation for installation steps: [Dagger Installation Guide](https://docs.dagger.io/install). | ||
|
|
||
| ## Dagger Setup and Development Mode | ||
| --- | ||
|
|
||
| ### Run Dagger Develop | ||
| ## 🚧 Prerequisites | ||
|
|
||
| ```bash | ||
| dagger develop | ||
| ``` | ||
| Before using the pipeline, make sure you have: | ||
|
|
||
| This command will generate the necessary files and configuration for building and running Dagger. | ||
| 1. **Dagger CLI** — Install the latest version from the official docs: | ||
| 👉 [Dagger Installation Guide](https://docs.dagger.io/install) | ||
| 2. **Go** — Installed according to the version specified in the project’s `go.mod`. | ||
| 3. **Docker** — Required if you’re publishing images. | ||
|
|
||
| --- | ||
|
|
||
| ## 📦 Dagger Functions Explained | ||
| ## ⚙️ Setup and Development Mode | ||
|
|
||
| ### 🔧 `BuildDev(platform)` | ||
| ### Run Dagger in Development Mode | ||
|
|
||
| Builds a development binary for your target platform. | ||
| To start the Dagger session and enable live code reloads: | ||
|
|
||
| ```bash | ||
| dagger call build-dev --platform="linux/amd64" export --path=bin/harbor-dev | ||
| dagger develop | ||
NucleoFusion marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ``` | ||
|
|
||
| ### 🧼 `LintReport()` | ||
| This command prepares the environment for pipeline development and local testing. | ||
|
|
||
| Runs `golangci-lint` on your code and saves the report to a file. | ||
| ## 📦 Dagger Functions Overview | ||
|
|
||
| ```bash | ||
| dagger call lint-report export --path=./LintReport.json | ||
| ``` | ||
| | **Name** | **Description** | | ||
| |--------------------------------|-------------------------------------------------------------------------------------------------| | ||
| | `lint` | Runs `golangci-lint` and prints the report as a string to stdout. | | ||
| | `lint-report` | Runs `golangci-lint` and writes the lint report to a file. | | ||
| | `pipeline` | Executes the **full CI/CD pipeline** including build, test, lint, and publish stages. | | ||
| | `run-doc` | Generates CLI documentation and returns the directory containing generated files. | | ||
| | `test` | Runs all Go tests in the repository. | | ||
| | `test-report` | Executes Go tests and outputs a structured JSON test report. | | ||
| | `test-coverage` | Runs Go tests with coverage tracking. | | ||
| | `test-coverage-report` | Processes coverage data and returns a formatted Markdown report. | | ||
| | `vulnerability-check` | Runs `govulncheck` to detect known vulnerabilities in dependencies. | | ||
| | `vulnerability-check-report` | Runs `govulncheck` and saves results to a file (`vulnerability-check.report`). | | ||
| | `build-dev` | Create build of Harbor CLI for local testing and development| | ||
|
|
||
| ### 📝 `TestCoverageReport()` | ||
| --- | ||
|
|
||
| Runs go test coverage tools and creates a report. | ||
| ```bash | ||
| dagger call test-coverage-report export --path=coverage-report.md | ||
| ``` | ||
| ## 🧩 Example Usage | ||
|
|
||
| ### ✅ `CheckCoverageThreshold(context, threshold)` | ||
| Below are some common commands to run specific Dagger functions locally: | ||
|
|
||
| Runs go test coverage tools and creates a report. The total coverage is compared to a threshold that can be set to e.g. 80%. | ||
| ```bash | ||
| dagger call check-coverage-threshold --threshold 80.0 | ||
| ``` | ||
| # Development build for binaries | ||
|
|
||
| ### 🚀 `PublishImage(registry, imageTags)` | ||
| dagger call build-dev --source=. --platform="linux/amd64" export --path=bin/harbor-dev | ||
|
|
||
| Builds and publishes the Harbor CLI image to the given container registry with proper OCI metadata labels. | ||
| # Print report to stdout | ||
| dagger call lint | ||
|
|
||
| Before running the command you have to export you registry password | ||
| # Save report to a file | ||
| dagger call lint-report export --path=LintReport.json | ||
|
|
||
| ```shell | ||
| export REGPASS=Harbor12345 | ||
| ``` | ||
| # Run Tests | ||
| dagger call test | ||
|
|
||
| ```bash | ||
| dagger call publish-image \ | ||
| --registry=demo.goharbor.io \ | ||
| --registry-username=harbor-cli \ | ||
| --registry-password=env:REGPASS \ | ||
| --imageTags=v0.1.0,latest | ||
| ``` | ||
| # Generate a JSON Report | ||
| dagger call test-report export --path=TestReport.json | ||
|
|
||
| --- | ||
| # Test Coverage | ||
| dagger call test-coverage | ||
|
|
||
| ## ⚙️ Configuration Constants | ||
| # Generate a Markdown Report | ||
| dagger call test-coverage-report export --path=coverage-report.md | ||
|
|
||
| Dagger uses these constant versions (you can modify them as needed): | ||
| # Vulnerability Check | ||
| dagger call vulnerability-check | ||
|
|
||
| ```go | ||
| const ( | ||
| GO_VERSION = "1.24.2" | ||
| GOLANGCILINT_VERSION = "v2.1.2" | ||
| SYFT_VERSION = "v1.9.0" | ||
| GORELEASER_VERSION = "v2.3.2" | ||
| ) | ||
| # Generate a Report | ||
| dagger call vulnerability-check-report export --path=vuln.report | ||
|
|
||
| # Generate CLI docs | ||
| dagger call run-doc export --path=docs/cli | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## 💡 Tips for Beginners | ||
| ## 💡 Tips for Contributors | ||
|
|
||
| - Every container step is **reproducible** you can build locally or in GitHub Actions without changes. | ||
| - Use Dagger to cache Go builds and lint output, speeding up re-runs. | ||
| - Every step in Dagger is **deterministic and reproducible** — what you run locally is identical to CI. | ||
| - Use Dagger’s built-in caching to accelerate Go builds, lint runs, and dependency installs. | ||
| - Modular functions let you run only what you need, improving iteration speed and debugging efficiency. | ||
| - Prefer using `dagger develop` for fast iteration and testing new steps before committing. | ||
| - Store output reports (lint, test, coverage) under a consistent `/reports` directory for easier CI integration. | ||
| - If you modify or add new pipeline steps, document them under **Dagger Functions Overview** to maintain clarity. | ||
| - Always validate pipelines with `dagger call pipeline` locally before merging into main. | ||
|
|
||
| --- | ||
|
|
||
| ## 📚 References | ||
|
|
||
| - [Dagger Go SDK Docs](https://pkg.go.dev/dagger.io/dagger) | ||
| - [golangci-lint](https://golangci-lint.run/) | ||
| - [Goreleaser](https://goreleaser.com/) | ||
| - [Dagger Go SDK](https://pkg.go.dev/dagger.io/dagger) | ||
| - [golangci-lint Docs](https://golangci-lint.run/) | ||
| - [govulncheck](https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,107 @@ | ||
| package main | ||
|
|
||
| import ( | ||
| "context" | ||
| "fmt" | ||
|
|
||
| "dagger/harbor-cli/internal/dagger" | ||
| ) | ||
|
|
||
| func (m *HarborCli) Apk(ctx context.Context, | ||
| buildDir *dagger.Directory, | ||
| // +ignore=[".gitignore"] | ||
| // +defaultPath="." | ||
| source *dagger.Directory, | ||
| ) (*dagger.Directory, error) { | ||
| if !m.IsInitialized { | ||
| err := m.init(ctx, source) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| } | ||
|
|
||
| buildfile := dag.File("APKBUILD", apkbuild(m.AppVersion)) | ||
|
|
||
| archs := []struct { | ||
| Arch string | ||
| ApkArch string | ||
| }{ | ||
| {"arm64", "aarch64"}, | ||
| {"amd64", "x86_64"}, | ||
| } | ||
|
|
||
| for _, arch := range archs { | ||
| filename := fmt.Sprintf("bin/harbor-cli_%s_linux_%s", m.AppVersion, arch.Arch) | ||
| binary := buildDir.File(filename) | ||
|
|
||
| apk := dag.Container(dagger.ContainerOpts{ | ||
| Platform: dagger.Platform(fmt.Sprintf("linux/%s", arch.Arch)), | ||
| }). | ||
| From("alpine:3.19"). | ||
| WithExec([]string{"apk", "add", "--no-cache", "alpine-sdk", "abuild"}). | ||
| WithWorkdir("/build"). | ||
| WithFile("/build/harbor-cli", binary). | ||
| WithFile("/build/APKBUILD", buildfile). | ||
|
|
||
| // create builder user + abuild group | ||
| WithExec([]string{"adduser", "-D", "builder"}). | ||
| WithExec([]string{"addgroup", "builder", "abuild"}). | ||
| WithExec([]string{"chown", "-R", "builder:builder", "/build"}). | ||
|
|
||
| // switch to builder FIRST | ||
| WithUser("builder"). | ||
| WithEnvVariable("HOME", "/home/builder"). | ||
|
|
||
| // generate signing key AS BUILDER (this is critical) | ||
| WithExec([]string{"abuild-keygen", "-a", "-n"}). | ||
|
|
||
| // switch back to root ONLY to trust the public key | ||
| WithUser("root"). | ||
| WithExec([]string{ | ||
| "sh", "-c", | ||
| "mkdir -p /etc/apk/keys && cp /home/builder/.abuild/*.rsa.pub /etc/apk/keys/", | ||
| }). | ||
|
|
||
| // back to builder for the build | ||
| WithUser("builder"). | ||
| WithEnvVariable("HOME", "/home/builder"). | ||
|
|
||
| // sanity check (keep this until stable) | ||
| WithExec([]string{"sh", "-c", "ls -l ~/.abuild && echo HOME=$HOME"}). | ||
|
|
||
| // run abuild | ||
| WithExec([]string{"abuild", "-rd"}) | ||
|
|
||
| apkFile := apk. | ||
| Directory("/home/builder/packages"). | ||
| Directory(arch.ApkArch). | ||
| File(fmt.Sprintf("harbor-cli-%s-r0.apk", m.AppVersion)) | ||
|
|
||
| apkFileName := fmt.Sprintf("apk/harbor-cli_%s_%s.apk", m.AppVersion, arch.Arch) | ||
| buildDir = buildDir.WithFile(apkFileName, apkFile) | ||
| } | ||
|
|
||
| return buildDir, nil | ||
| } | ||
|
|
||
| func apkbuild(ver string) string { | ||
| return fmt.Sprintf(`# APKBUILD | ||
| pkgname=harbor-cli | ||
| pkgver=%s | ||
| pkgrel=0 | ||
| pkgdesc="Harbor CLI — a command-line interface for interacting with your Harbor container registry." | ||
| url="https://github.com/goharbor/harbor-cli" | ||
| arch="x86_64 aarch64" | ||
| license="Apache-2.0" | ||
| depends="" | ||
| makedepends="" | ||
| source="" | ||
| maintainer="NucleoFusion <[email protected]>" | ||
| builddir="/build" | ||
|
|
||
| package() { | ||
| install -Dm755 "$builddir/harbor-cli" \ | ||
| "$pkgdir/usr/bin/harbor-cli" | ||
| } | ||
| `, ver) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| package main | ||
|
|
||
| import ( | ||
| "context" | ||
| "fmt" | ||
|
|
||
| "dagger/harbor-cli/internal/dagger" | ||
| ) | ||
|
|
||
| func (m *HarborCli) AptBuild(ctx context.Context, | ||
| buildDir *dagger.Directory, | ||
| // +ignore=[".gitignore"] | ||
| // +defaultPath="." | ||
| source *dagger.Directory, | ||
| token *dagger.Secret, | ||
| ) error { | ||
| if !m.IsInitialized { | ||
| err := m.init(ctx, source) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| } | ||
|
|
||
| archs := []string{"amd64", "arm64"} | ||
| root := dag.Directory() | ||
| root = root.WithDirectory("pool/main/m", buildDir.Directory("deb")) | ||
| githubToken, err := token.Plaintext(ctx) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| // Base container | ||
| container := dag.Container(). | ||
| From("debian:bookworm-slim"). | ||
| WithExec([]string{"apt-get", "update"}). | ||
| WithExec([]string{"apt-get", "install", "-y", "dpkg-dev", "gzip", "git"}). | ||
| WithEnvVariable("GH_TOKEN", githubToken). | ||
| WithMountedDirectory("/repo", root). | ||
| WithWorkdir("/repo") | ||
|
|
||
| // Building `Package` file for each arch | ||
| for _, arch := range archs { | ||
| pkgDir := fmt.Sprintf("buildDirs/stable/main/binary-%s", arch) | ||
| poolDir := "pool/main/m" | ||
|
|
||
| container = container.WithExec([]string{ | ||
| "bash", "-c", | ||
| fmt.Sprintf("mkdir -p %s && dpkg-scanpackages -a %s %s /dev/null > %s/Packages && gzip -9c %s/Packages > %s/Packages.gz && rm -rf %s/Packages", | ||
| pkgDir, arch, poolDir, pkgDir, pkgDir, pkgDir, pkgDir), | ||
| }) | ||
| } | ||
|
|
||
| // Release File | ||
| container = container.WithExec([]string{ | ||
| "bash", "-c", | ||
| `cat <<EOF > /repo/buildDirs/stable/Release | ||
| Origin: https://github.com/goharbor/harbor-cli | ||
| Label: HarborCLI | ||
| Suite: stable | ||
| Codename: stable | ||
| Architectures: amd64 arm64 | ||
| Components: main | ||
| Description: Harbor CLI — a command-line interface for interacting with your Harbor container registry. | ||
| EOF`, | ||
| }) | ||
|
|
||
| container = container. | ||
| WithWorkdir("/repo"). | ||
| WithExec([]string{ | ||
| "bash", "-c", | ||
| fmt.Sprintf(` | ||
| set -e | ||
| cd /repo | ||
|
|
||
| git init | ||
| git remote add origin https://x-access-token:[email protected]/nucleofusion/harbor-cli.git | ||
| git checkout -B gh-pages || git checkout --orphan gh-pages | ||
|
|
||
| git config user.name "github-actions[bot]" | ||
| git config user.email "github-actions[bot]@users.noreply.github.com" | ||
|
|
||
| git add buildDirs pool | ||
|
|
||
| git commit -m "Update APT repo for %s" || echo "No changes to commit" | ||
| git push origin gh-pages -f | ||
| `, m.AppVersion), | ||
| }) | ||
|
|
||
| _, err = container.Sync(ctx) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to run container: %w", err) | ||
| } | ||
|
|
||
| return nil | ||
| } | ||
|
|
||
| // GH-PAGES Structure | ||
| // | ||
| // / | ||
| // ├── dist/ | ||
| // │ └── stable/ | ||
| // │ ├── Release | ||
| // │ └── main/ | ||
| // │ ├── binary-amd64/ | ||
| // │ │ └── Packages.gz | ||
| // │ └── binary-arm64/ | ||
| // │ └── Packages.gz | ||
| // └── pool/ | ||
| // └── main/ | ||
| // └── m/ | ||
| // ├── myapp_1.0.0_amd64.deb | ||
| // └── myapp_1.0.0_arm64.deb | ||
| // |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.