Skip to content

Commit 74f5ed7

Browse files
openshift-merge-bot[bot]openshift-merge-bot[bot]
authored
Merge pull request #26713 from Luap99/deps-script
contrib: add script and docs to analyze dependency usage
2 parents ecd5dfe + 512e049 commit 74f5ed7

File tree

6 files changed

+266
-125
lines changed

6 files changed

+266
-125
lines changed

‎contrib/dependencies/README.md‎

Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
# Dependency analysis
2+
3+
## Requirements
4+
5+
The script assumes you have goda installed and in $PATH. You can install it with
6+
```
7+
go install github.com/loov/goda@latest
8+
```
9+
10+
Check https://github.com/loov/goda for more details.
11+
12+
## Basic Usage
13+
14+
The dependencies.sh script is mainly a small wrapper script around goda to make certain operations easier to use as the goda expression syntax is quite special.
15+
16+
### List all the packages used by podman
17+
```
18+
$ ./contrib/dependencies/dependencies.sh list
19+
```
20+
If you like to use another command, i.e. podman-remote or quadlet the script supports the `--command`/`-c` option:
21+
```
22+
$ ./contrib/dependencies/dependencies.sh -c quadlet list
23+
```
24+
25+
### Show dependency tree
26+
```
27+
$ ./contrib/dependencies/dependencies.sh tree
28+
```
29+
Note the output is general not very readable so it is unlikely that it helps much.
30+
31+
### Show what imports a specific package
32+
33+
To know what import a given package use
34+
35+
```
36+
$ ./contrib/dependencies/dependencies.sh why ./pkg/ps/
37+
github.com/containers/podman/v5/pkg/api/handlers/compat
38+
github.com/containers/podman/v5/pkg/domain/infra/abi
39+
github.com/containers/podman/v5/pkg/ps
40+
```
41+
42+
This is useful to know from where the package is being used.
43+
44+
## Bloat check analysis
45+
46+
Golang binaries are big, often a single import can cause a large increase in binary size.
47+
This section describes how to find such imports.
48+
49+
First, if you hit the bloat check on a PR it may be useful to run a diff between two branches.
50+
The diff subcommand runs the list command on two branches and then shows you the diff. This is
51+
useful to see all the new imports that might cause the bloat. It may be easy to spot something
52+
that should not get dragged in.
53+
54+
```
55+
$ ./contrib/dependencies/dependencies.sh diff main pr/26577
56+
Switched to branch 'main'
57+
Your branch is ahead of 'origin/master' by 17550 commits.
58+
(use "git push" to publish your local commits)
59+
Switched to branch 'pr/26577'
60+
285d284
61+
< github.com/containers/podman/v5/cmd/podman/quadlet
62+
318a318
63+
> github.com/containers/podman/v5/pkg/bindings/artifacts
64+
362d361
65+
< github.com/containers/podman/v5/pkg/logiface
66+
404d402
67+
< github.com/containers/podman/v5/pkg/systemd/quadlet
68+
```
69+
70+
If the diff from the imports is not clear there is the experimental
71+
weight-diff command.
72+
73+
```
74+
$ ./contrib/dependencies/dependencies.sh weight-diff main pr/26577
75+
...
76+
name diff bin/podman.1 bin/podman
77+
github.com/containers/podman/v5/pkg/domain/infra/abi.(*ContainerEngine).QuadletRemove 11025 11025 -
78+
github.com/containers/podman/v5/pkg/domain/infra/abi.(*ContainerEngine).QuadletInstall 6233 6233 -
79+
github.com/containers/podman/v5/pkg/bindings/images.Build 5760 38418 32658
80+
github.com/containers/buildah.(*Builder).createMountTargets 5684 5684 -
81+
github.com/containers/podman/v5/pkg/domain/infra/abi.(*ContainerEngine).QuadletList 5485 5485 -
82+
github.com/containers/buildah.(*Builder).Add 5231 21172 15941
83+
github.com/containers/podman/v5/pkg/api/handlers/compat.handleBuildContexts 4254 4254 -
84+
github.com/containers/podman/v5/pkg/bindings/artifacts.Extract 2958 - 2958
85+
github.com/containers/podman/v5/pkg/bindings/images.Build.func2 2799 2799 -
86+
github.com/containers/podman/v5/pkg/domain/infra/abi.(*ContainerEngine).installQuadlet 2743 2743 -
87+
github.com/containers/buildah.(*Builder).makeContainerImageRef 2208 8293 6085
88+
github.com/containers/buildah.(*Builder).setupMounts 2087 4869 6956
89+
github.com/containers/podman/v5/cmd/podman/quadlet.listFlags.AutocompleteFormat.func1 2074 2074 -
90+
init 2011 2016 5
91+
github.com/containers/podman/v5/cmd/podman/quadlet.rm 1342 1342 -
92+
github.com/containers/podman/v5/pkg/systemd/quadlet.AppendSubPaths 1285 1285 -
93+
github.com/containers/podman/v5/pkg/systemd/parser.(*UnitFile).Parse 1263 1263 -
94+
github.com/containers/buildah/imagebuildah.(*StageExecutor).intermediateImageExists 1248 3829 2581
95+
github.com/containers/podman/v5/pkg/systemd/parser.(*UnitFileParser).parseGroup 1241 1241 -
96+
github.com/containers/podman/v5/pkg/systemd/parser.(*UnitFileParser).parseKeyValuePair 1241 1241 -
97+
github.com/containers/podman/v5/pkg/bindings/artifacts.Pull 1221 - 1221
98+
github.com/containers/podman/v5/pkg/bindings/artifacts.Add 1209 - 1209
99+
github.com/containers/podman/v5/pkg/systemd/quadlet.getRootlessDirs 1199 1199 -
100+
github.com/containers/podman/v5/pkg/domain/infra/tunnel.(*ImageEngine).ArtifactAdd 1198 59 1257
101+
github.com/containers/podman/v5/pkg/domain/infra/abi.getAllQuadletPaths 1189 1189 -
102+
github.com/containers/podman/v5/cmd/podman/quadlet.outputTemplate 1189 1189 -
103+
github.com/containers/podman/v5/pkg/bindings/artifacts.Push 1076 - 1076
104+
```
105+
106+
That gives insight into the symbol sizes. There is likely not much that can be done to avoid it
107+
though unless you manage to de-duplicate a lot of code somehow. Or avoid some of the new imports/symbols.
108+
109+
The `diff` and `weight-diff` commands accept two git references as argument to diff between,
110+
i.e. branch names, git tags, or commit sha's should all work. Basically anything that git checkout accepts.
111+
112+
## Debloat exercise
113+
114+
When actively working to reduce binary size it is easiest to look into big dependencies that have not
115+
many users which can be replaced with something else or maybe are not needed at all.
116+
117+
To find such dependencies use the cut command
118+
```
119+
$ ./contrib/dependencies/dependencies.sh cut
120+
ID InDegree Cut.PackageCount Cut.AllFiles.Size Cut.Go.Lines
121+
github.com/containers/podman/v5/cmd/podman 0 905 42.5MB 964250
122+
vendor/golang.org/x/crypto/internal/poly1305 0 1 16.4KB 395
123+
vendor/golang.org/x/net/http2/hpack 0 1 43.9KB 1471
124+
...
125+
github.com/containers/ocicrypt/keywrap/keyprovider 1 75 1.9MB 56453
126+
github.com/docker/docker/client 1 37 1.5MB 39437
127+
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp 1 22 0.7MB 19647
128+
github.com/containers/buildah/imagebuildah 1 14 1.2MB 36431
129+
github.com/containers/image/v5/pkg/cli/sigstore 1 14 229.8KB 6343
130+
github.com/containers/storage/drivers/register 1 12 229.0KB 7109
131+
go/types 1 12 1.0MB 31977
132+
```
133+
134+
Here you see a list of all imported packages:
135+
- InDegree means how many times it is imported.
136+
- Cut.PackageCount shows how many packages (including transitive imports) are getting drop by removing this package.
137+
- Cut.AllFiles.Size and Cut.Go.Lines show the info about the go source files of that package and its transitive imports.
138+
Note this is not the actual binary size. File size/Line count is not directly related to the resulting binary size.
139+
It is likely that if we only use a single function out of a big package we do not bloat the binary size by much.
140+
141+
Look for packages with a low InDegree count and a high PackageCount for best gains. A low important count makes it of
142+
course easier to remove the dep somehow. A high PackageCount count means the binary size gain will likely be a lot.
143+
144+
Of course there is a log of basic functionality that we can just remove/replace so it requires best judgement to
145+
actually find packages that we can remove.
146+
147+
A special case are the `vendor/golang.org/x/...` packages, these are actually vendored by the standard library directly
148+
and not shared with our `golang.org/x/...` imports. As such they cause bloat but since they are used by the std lib we
149+
cannot really get rid of them easily or change them as such we have to simple accept them.
150+
151+
The `go/types` package is a good example of finding something to be cut. It is only used once for one small function so
152+
we can replace it easily. That is done in https://github.com/containers/buildah/pull/6253.
153+
154+
Because file size != binary size it may be worth to look into the actual symbol size of each package first before deciding
155+
if we really gain much by removing it to spend
156+
157+
This is done with the weight command:
158+
```
159+
$ ./contrib/dependencies/dependencies.sh weight
160+
1213750 1125084 /github.com/containers/podman/v5/libpod [syms 1699]
161+
28143 t (*Container).generateSpec
162+
18790 t (*Runtime).setupContainer
163+
...
164+
476690 476690 /go/types [syms 1100]
165+
21630 t (*Checker).builtin
166+
...
167+
```
168+
169+
The output lists each packages total size and then the individual symbol sizes sorted by the biggest packages first.
170+
171+
Here we can see is that `go/types` is recorded with `476690` bytes so it is a good candidate to cut.
172+
173+
Overall there is no silver bullet to this, it mostly relies on experience which parts of the code would be good
174+
candidates for removal/reworks to reduce the number of dependencies.

‎contrib/dependencies/dependencies.sh‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
#!/usr/bin/env bash
2+
3+
set -e -o pipefail
4+
5+
GODA=${GODA:-goda}
6+
7+
8+
# default to podman
9+
command=podman
10+
declare -a args=()
11+
declare -a build_tags=()
12+
13+
# Parse Arguments
14+
while [[ "$#" -gt 0 ]]; do
15+
case $1 in
16+
-c | --command)
17+
if [[ "$2" == "podman-remote" ]]; then
18+
command=podman
19+
build_tags+=("remote")
20+
else
21+
command="$2"
22+
fi
23+
shift
24+
;;
25+
-t | --tags)
26+
build_tags+=("$2")
27+
shift
28+
;;
29+
*) args+=("$1") ;;
30+
esac
31+
shift
32+
done
33+
34+
function create_goda_expr() {
35+
local expr="./cmd/$command:all"
36+
for tag in ${build_tags[*]}; do
37+
expr="$tag=1($expr)"
38+
done
39+
echo $expr
40+
}
41+
42+
function run_list() {
43+
expr=$(create_goda_expr)
44+
$GODA list -std -h - "$expr"
45+
}
46+
47+
function run_tree() {
48+
expr=$(create_goda_expr)
49+
$GODA tree -std "$expr"
50+
}
51+
52+
function run_why() {
53+
expr=$(create_goda_expr)
54+
$GODA list -std -h - "incoming($expr, $1)"
55+
}
56+
57+
function run_diff() {
58+
expr=$(create_goda_expr)
59+
cb=$(git branch --show-current)
60+
tmpdir=$(mktemp -d)
61+
git checkout $1
62+
$GODA list -std -h - "$expr" | sort >$tmpdir/one
63+
git checkout ${2:-$cb}
64+
$GODA list -std -h - "$expr" | sort >$tmpdir/two
65+
diff --color $tmpdir/one $tmpdir/two || true
66+
rm -rf "$tmpdir"
67+
git checkout "$cb"
68+
}
69+
70+
function run_cut() {
71+
expr=$(create_goda_expr)
72+
$GODA cut -std $expr
73+
}
74+
75+
function run_weight() {
76+
$GODA weight bin/$command
77+
}
78+
79+
function run_weight-diff() {
80+
cb=$(git branch --show-current)
81+
local file="bin/$command"
82+
git checkout $1
83+
make $file
84+
mv $file $file.1
85+
git checkout ${2:-$cb}
86+
make $file
87+
$GODA weight-diff -miss $file.1 $file
88+
rm $file.1
89+
git checkout "$cb"
90+
}
91+
92+
run_${args[0]} "${args[@]:1}"

‎dependencies/analyses/README.md‎

Lines changed: 0 additions & 87 deletions
This file was deleted.

‎dependencies/analyses/dependency-tree.sh‎

Lines changed: 0 additions & 17 deletions
This file was deleted.

‎dependencies/analyses/go-archive-analysis.sh‎

Lines changed: 0 additions & 12 deletions
This file was deleted.

‎dependencies/analyses/nm-symbols-analysis.sh‎

Lines changed: 0 additions & 9 deletions
This file was deleted.

0 commit comments

Comments
 (0)