Skip to content

feat: add git clone with thinpack option to support self hosted azure devops #446

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Mar 13, 2025
+39 −2
Merged

feat: add git clone with thinpack option to support self hosted azure devops #446

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
2a22162
feat: add git clone with thinkpack option to support self hosted azur…
CH-Chang Jan 11, 2025
e4fb15a
feat: adjust thinpack logging and option definition
CH-Chang Mar 13, 2025
4653010
docs: update git clone thin pack option description
CH-Chang Mar 13, 2025
98985a2
feat: sync command line option description with docs
CH-Chang Mar 13, 2025
cbd387e
docs: update docs with make docs/env-variables.md
CH-Chang Mar 13, 2025
af75d61
refactor: adjust if statement
CH-Chang Mar 13, 2025
9e9bdde
Merge branch 'main' into dev/chchang/thinkpack
CH-Chang Mar 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/env-variables.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
| `--git-url` | `ENVBUILDER_GIT_URL` | | The URL of a Git repository containing a Devcontainer or Docker image to clone. This is optional. |
| `--git-clone-depth` | `ENVBUILDER_GIT_CLONE_DEPTH` | | The depth to use when cloning the Git repository. |
| `--git-clone-single-branch` | `ENVBUILDER_GIT_CLONE_SINGLE_BRANCH` | | Clone only a single branch of the Git repository. |
| `--git-clone-thinpack` | `ENVBUILDER_GIT_CLONE_THINPACK` | | Git clone with thin pack compatibility enabled, ensuring that even when thin pack compatibility is activated, it will not be turned on for the domain dev.zaure.com. |
| `--git-username` | `ENVBUILDER_GIT_USERNAME` | | The username to use for Git authentication. This is optional. |
| `--git-password` | `ENVBUILDER_GIT_PASSWORD` | | The password to use for Git authentication. This is optional. |
| `--git-ssh-private-key-path` | `ENVBUILDER_GIT_SSH_PRIVATE_KEY_PATH` | | Path to an SSH private key to be used for Git authentication. If this is set, then GIT_SSH_PRIVATE_KEY_BASE64 cannot be set. |
Expand Down
52 changes: 33 additions & 19 deletions git/git.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ type CloneRepoOptions struct {
Progress sideband.Progress
Insecure bool
SingleBranch bool
ThinPack bool
Depth int
CABundle []byte
ProxyOptions transport.ProxyOptions
Expand All @@ -53,28 +54,40 @@ func CloneRepo(ctx context.Context, logf func(string, ...any), opts CloneRepoOpt
return false, fmt.Errorf("parse url %q: %w", opts.RepoURL, err)
}
logf("Parsed Git URL as %q", parsed.Redacted())
if parsed.Hostname() == "dev.azure.com" {
// Azure DevOps requires capabilities multi_ack / multi_ack_detailed,
// which are not fully implemented and by default are included in
// transport.UnsupportedCapabilities.
//
// The initial clone operations require a full download of the repository,
// and therefore those unsupported capabilities are not as crucial, so
// by removing them from that list allows for the first clone to work
// successfully.
//
// Additional fetches will yield issues, therefore work always from a clean
// clone until those capabilities are fully supported.
//
// New commits and pushes against a remote worked without any issues.
// See: https://github.com/go-git/go-git/issues/64
//
// This is knowingly not safe to call in parallel, but it seemed
// like the least-janky place to add a super janky hack.

thinPack := true

if !opts.ThinPack {
thinPack = false
logf("ThinPack options is false, Marking thin-pack as unsupported")
} else {
if parsed.Hostname() == "dev.azure.com" {
// Azure DevOps requires capabilities multi_ack / multi_ack_detailed,
// which are not fully implemented and by default are included in
// transport.UnsupportedCapabilities.
//
// The initial clone operations require a full download of the repository,
// and therefore those unsupported capabilities are not as crucial, so
// by removing them from that list allows for the first clone to work
// successfully.
//
// Additional fetches will yield issues, therefore work always from a clean
// clone until those capabilities are fully supported.
//
// New commits and pushes against a remote worked without any issues.
// See: https://github.com/go-git/go-git/issues/64
//
// This is knowingly not safe to call in parallel, but it seemed
// like the least-janky place to add a super janky hack.
thinPack = false
logf("Workaround for Azure DevOps: marking thin-pack as unsupported")
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: else if { ... }

Copy link
Contributor Author

@CH-Chang CH-Chang Mar 13, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I have updated the code to use else if { ... } and have run make docs/env-variables.md to update the documentation.


if !thinPack {
transport.UnsupportedCapabilities = []capability.Capability{
capability.ThinPack,
}
logf("Workaround for Azure DevOps: marking thin-pack as unsupported")
}

err = opts.Storage.MkdirAll(opts.Path, 0o755)
Expand Down Expand Up @@ -347,6 +360,7 @@ func CloneOptionsFromOptions(logf func(string, ...any), options options.Options)
Storage: options.Filesystem,
Insecure: options.Insecure,
SingleBranch: options.GitCloneSingleBranch,
ThinPack: options.GitCloneThinPack,
Depth: int(options.GitCloneDepth),
CABundle: caBundle,
}
Expand Down
11 changes: 11 additions & 0 deletions options/options.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,8 @@ type Options struct {
GitCloneDepth int64
// GitCloneSingleBranch clone only a single branch of the Git repository.
GitCloneSingleBranch bool
// GitCloneThinPack clone with thin pack compabilities. This is optional.
GitCloneThinPack bool
// GitUsername is the username to use for Git authentication. This is
// optional.
GitUsername string
Expand Down Expand Up @@ -375,6 +377,15 @@ func (o *Options) CLI() serpent.OptionSet {
Value: serpent.BoolOf(&o.GitCloneSingleBranch),
Description: "Clone only a single branch of the Git repository.",
},
{
Flag: "git-clone-thinpack",
Env: WithEnvPrefix("GIT_CLONE_THINPACK"),
Value: serpent.BoolOf(&o.GitCloneThinPack),
Default: "true",
Description: "Git clone with thin pack compatibility enabled, " +
"ensuring that even when thin pack compatibility is activated," +
"it will not be turned on for the domain dev.zaure.com.",
},
{
Flag: "git-username",
Env: WithEnvPrefix("GIT_USERNAME"),
Expand Down
8 changes: 8 additions & 0 deletions options/options_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,31 +38,39 @@ func TestEnvOptionParsing(t *testing.T) {
t.Run("lowercase", func(t *testing.T) {
t.Setenv(options.WithEnvPrefix("SKIP_REBUILD"), "true")
t.Setenv(options.WithEnvPrefix("GIT_CLONE_SINGLE_BRANCH"), "false")
t.Setenv(options.WithEnvPrefix("GIT_CLONE_THINPACK"), "false")
o := runCLI()
require.True(t, o.SkipRebuild)
require.False(t, o.GitCloneSingleBranch)
require.False(t, o.GitCloneThinPack)
})

t.Run("uppercase", func(t *testing.T) {
t.Setenv(options.WithEnvPrefix("SKIP_REBUILD"), "TRUE")
t.Setenv(options.WithEnvPrefix("GIT_CLONE_SINGLE_BRANCH"), "FALSE")
t.Setenv(options.WithEnvPrefix("GIT_CLONE_THINPACK"), "FALSE")
o := runCLI()
require.True(t, o.SkipRebuild)
require.False(t, o.GitCloneSingleBranch)
require.False(t, o.GitCloneThinPack)
})

t.Run("numeric", func(t *testing.T) {
t.Setenv(options.WithEnvPrefix("SKIP_REBUILD"), "1")
t.Setenv(options.WithEnvPrefix("GIT_CLONE_SINGLE_BRANCH"), "0")
t.Setenv(options.WithEnvPrefix("GIT_CLONE_THINPACK"), "0")
o := runCLI()
require.True(t, o.SkipRebuild)
require.False(t, o.GitCloneSingleBranch)
require.False(t, o.GitCloneThinPack)
})

t.Run("empty", func(t *testing.T) {
t.Setenv(options.WithEnvPrefix("GIT_CLONE_SINGLE_BRANCH"), "")
t.Setenv(options.WithEnvPrefix("GIT_CLONE_THINPACK"), "")
o := runCLI()
require.False(t, o.GitCloneSingleBranch)
require.True(t, o.GitCloneThinPack)
})
})
}
Expand Down
5 changes: 5 additions & 0 deletions options/testdata/options.golden
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,11 @@ OPTIONS:
--git-clone-single-branch bool, $ENVBUILDER_GIT_CLONE_SINGLE_BRANCH
Clone only a single branch of the Git repository.

--git-clone-thinpack bool, $ENVBUILDER_GIT_CLONE_THINPACK (default: true)
Git clone with thin pack compatibility enabled, ensuring that even
when thin pack compatibility is activated,it will not be turned on for
the domain dev.zaure.com.

--git-http-proxy-url string, $ENVBUILDER_GIT_HTTP_PROXY_URL
The URL for the HTTP proxy. This is optional.

Expand Down
Loading