Skip to content

Commit e2bf9b6

Browse files
egibsegibs
authored
Migrate from go-yara to yara-x; improve performance and readability (#734)
* Swap over to yara-x; improve performance and readability Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Refresh test data Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Small concurrency tweaks Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Tweak third-party.yaml Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Re-use c.Concurrency Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Refresh test data Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Fix stats JSON test Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Add UPX to Workflows Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Replace empty match strings with the matched patterns Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Revert str variable change Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Use code highlighting to avoid escaping patterns Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Small comment tweaks Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Move scanner pool assignment in refresh functions Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * One last go get -u ./... Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Split up test jobs Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Another round of go get -u Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Container hardening Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Remove --security-opt Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Only run container workloads in the parent repository Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Update golangci-lint job Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Use 4-core CPU shares Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Clean up API installation instructions Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Less redundant wording Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> * Simplify further Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> --------- Signed-off-by: egibs <20933572+egibs@users.noreply.github.com> Signed-off-by: Evan Gibler <20933572+egibs@users.noreply.github.com>
1 parent 98bb996 commit e2bf9b6

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

64 files changed

+2609
-2125
lines changed

‎.github/workflows/go-tests.yaml‎

Lines changed: 50 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -16,31 +16,67 @@ permissions:
1616

1717
jobs:
1818
test:
19+
if: ${{ github.repository }} == 'chainguard-dev/malcontent'
1920
runs-on: mal-ubuntu-latest-8-core
20-
21+
container:
22+
image: cgr.dev/chainguard/wolfi-base@sha256:eeb70e74e2ac07d3c80a30150bf473970c8b51a57f06daef3e4d065ac52489bc
23+
options: >-
24+
--cap-add DAC_OVERRIDE
25+
--cap-add SETGID
26+
--cap-add SETUID
27+
--cap-drop ALL
28+
--cgroupns private
29+
--cpu-shares=8192
30+
--memory-swappiness=0
31+
--security-opt no-new-privileges
32+
--ulimit core=0
33+
--ulimit nofile=1024:1024
34+
--ulimit nproc=4096:4096
2135
steps:
22-
- uses: step-security/harden-runner@0080882f6c36860b6ba35c610c98ce87d4e2f26f # v2.10.2
23-
with:
24-
egress-policy: audit
36+
- name: Install dependencies
37+
run: |
38+
apk update
39+
apk add curl findutils git go nodejs upx xz yara-x-compat
2540
2641
- name: Checkout code
2742
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
2843

29-
- uses: actions/setup-go@3041bf56c941b39c61721a86cd11f3bb1338122a # v5.2.0
30-
with:
31-
go-version-file: go.mod
32-
check-latest: true
33-
34-
- uses: chainguard-dev/actions/goimports@9d943fc9889a0c0795e3c2bd4b949a9b610ac02e # main
35-
36-
- name: install dependencies
37-
run: |
38-
sudo apt update && sudo apt install libyara-dev xz-utils -y
44+
- name: Trust repository
45+
run: git config --global --add safe.directory $GITHUB_WORKSPACE
3946

4047
- name: Unit tests
4148
run: |
4249
make test
4350
51+
integration:
52+
if: ${{ github.repository }} == 'chainguard-dev/malcontent'
53+
runs-on: mal-ubuntu-latest-8-core
54+
container:
55+
image: cgr.dev/chainguard/wolfi-base@sha256:eeb70e74e2ac07d3c80a30150bf473970c8b51a57f06daef3e4d065ac52489bc
56+
options: >-
57+
--cap-add DAC_OVERRIDE
58+
--cap-add SETGID
59+
--cap-add SETUID
60+
--cap-drop ALL
61+
--cgroupns private
62+
--cpu-shares=8192
63+
--memory-swappiness=0
64+
--security-opt no-new-privileges
65+
--ulimit core=0
66+
--ulimit nofile=1024:1024
67+
--ulimit nproc=4096:4096
68+
steps:
69+
- name: Install dependencies
70+
run: |
71+
apk update
72+
apk add curl findutils git go nodejs upx xz yara-x-compat
73+
74+
- name: Checkout code
75+
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
76+
77+
- name: Trust repository
78+
run: git config --global --add safe.directory $GITHUB_WORKSPACE
79+
4480
- name: Integration tests
4581
run: |
4682
make integration

‎.github/workflows/style.yaml‎

Lines changed: 25 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -114,29 +114,33 @@ jobs:
114114

115115
golangci-lint:
116116
name: golangci-lint
117-
runs-on: ubuntu-24.04
118-
117+
runs-on: ubuntu-latest
118+
container:
119+
image: cgr.dev/chainguard/wolfi-base@sha256:eeb70e74e2ac07d3c80a30150bf473970c8b51a57f06daef3e4d065ac52489bc
120+
options: >-
121+
--cap-add DAC_OVERRIDE
122+
--cap-add SETGID
123+
--cap-add SETUID
124+
--cap-drop ALL
125+
--cgroupns private
126+
--cpu-shares=4096
127+
--memory-swappiness=0
128+
--security-opt no-new-privileges
129+
--ulimit core=0
130+
--ulimit nofile=1024:1024
131+
--ulimit nproc=4096:4096
119132
steps:
120-
- name: Harden Runner
121-
uses: step-security/harden-runner@0080882f6c36860b6ba35c610c98ce87d4e2f26f # v2.10.2
122-
with:
123-
egress-policy: audit
133+
- name: Install dependencies
134+
run: |
135+
apk update
136+
apk add curl findutils git go nodejs yara-x-compat
124137
125-
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
138+
- name: Checkout code
139+
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
126140

127-
- name: Set up Go
128-
uses: actions/setup-go@3041bf56c941b39c61721a86cd11f3bb1338122a # v5.2.0
129-
with:
130-
go-version-file: go.mod
131-
check-latest: true
141+
- name: Trust repository
142+
run: git config --global --add safe.directory $GITHUB_WORKSPACE
132143

133-
- name: install libyara-dev
144+
- name: Run golangci-lint
134145
run: |
135-
sudo apt update && sudo apt install libyara-dev -y
136-
137-
- name: golangci-lint
138-
uses: golangci/golangci-lint-action@971e284b6050e8a5849b72094c50ab08da042db8 # v6.1.1
139-
with:
140-
version: v1.62.0
141-
args: --timeout=5m
142-
146+
make golangci-lint-lint

‎.github/workflows/third-party.yaml‎

Lines changed: 19 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -12,14 +12,29 @@ jobs:
1212
update:
1313
if: ${{ github.repository }} == 'chainguard-dev/malcontent'
1414
runs-on: mal-ubuntu-latest-8-core
15+
container:
16+
image: cgr.dev/chainguard/wolfi-base@sha256:eeb70e74e2ac07d3c80a30150bf473970c8b51a57f06daef3e4d065ac52489bc
17+
options: >-
18+
--cap-add DAC_OVERRIDE
19+
--cap-add SETGID
20+
--cap-add SETUID
21+
--cap-drop ALL
22+
--cgroupns private
23+
--cpu-shares=8192
24+
--memory-swappiness=0
25+
--security-opt no-new-privileges
26+
--ulimit core=0
27+
--ulimit nofile=1024:1024
28+
--ulimit nproc=4096:4096
1529
permissions:
1630
contents: write
1731
id-token: write
1832
pull-requests: write
1933
steps:
20-
- uses: step-security/harden-runner@0080882f6c36860b6ba35c610c98ce87d4e2f26f
21-
with:
22-
egress-policy: audit
34+
- name: Install dependencies
35+
run: |
36+
apk update
37+
apk add bash curl findutils gh git go nodejs upx xz yara-x-compat
2338
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
2439
- uses: chainguard-dev/actions/setup-gitsign@e82b4e5ae10182af72972addcb3fedf7454621c8
2540
- name: Set up Octo-STS
@@ -28,9 +43,7 @@ jobs:
2843
with:
2944
scope: chainguard-dev/malcontent
3045
identity: third-party
31-
- name: Install yara and libyara-dev
32-
run: |
33-
sudo apt update && sudo apt install yara libyara-dev -y
46+
3447
- name: Run make update-third-party
3548
run: |
3649
make update-third-party

‎Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ $(GOLANGCI_LINT_BIN):
5050
curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(LINT_ROOT)/out/linters $(GOLANGCI_LINT_VERSION)
5151
mv $(LINT_ROOT)/out/linters/golangci-lint $@
5252

53-
YARA_X_VERSION ?= v0.10.0
53+
YARA_X_VERSION ?= v0.12.0
5454
YARA_X_BIN := $(LINT_ROOT)/out/linters/yr-$(YARA_X_VERSION)-$(LINT_ARCH)
5555
$(YARA_X_BIN):
5656
mkdir -p $(LINT_ROOT)/out/linters

‎README.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -118,20 +118,20 @@ The analyze mode emits a list of capabilities often seen in malware, categorized
118118
Requirements:
119119

120120
* [go](https://go.dev/) - the programming language
121-
* [yara](https://virustotal.github.io/yara/) - the rule language
121+
* [yara-x](https://virustotal.github.io/yara-x/) - the rule language
122122
* [pkgconf](http://pkgconf.org/) - required by Go to find C dependencies, included in many UNIX distributions
123123

124-
Linux or macOS users can run the following command to install the necessary dependencies, other than Go:
124+
`yara-x` requires an underlying C API to function. To build and install the API, reference the documentation here: https://virustotal.github.io/yara-x/docs/api/c/c-/#building-the-c-library.
125125

126-
```shell
127-
brew install yara || sudo apt install libyara-dev \
128-
|| sudo dnf install yara-devel || sudo pacman -S yara \
129-
|| sudo zypper install yara
126+
Running `cargo cinstall -p yara-x-capi --release` without `sudo` may encounter permission denied errors.
127+
128+
If this is the case, run the following:
129+
```sh
130+
sudo -E env "PATH=$PATH" cargo cinstall -p yara-x-capi --release
130131
```
131132

132133
Install malcontent:
133-
134-
```shell
134+
```sh
135135
go install github.com/chainguard-dev/malcontent/cmd/mal@latest
136136
```
137137

‎cmd/mal/mal.go‎

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -239,8 +239,26 @@ func main() {
239239
rfs = append(rfs, thirdparty.FS)
240240
}
241241

242+
yrs, err := action.CachedRules(ctx, rfs)
243+
if err != nil {
244+
returnCode = ExitInvalidRules
245+
}
246+
247+
concurrency := concurrencyFlag
248+
if concurrency < 1 {
249+
concurrency = 1
250+
}
251+
252+
var pool *malcontent.ScannerPool
253+
if mc.ScannerPool == nil {
254+
pool, err = malcontent.NewScannerPool(yrs, concurrency)
255+
if err != nil {
256+
returnCode = ExitInvalidRules
257+
}
258+
}
259+
242260
mc = malcontent.Config{
243-
Concurrency: concurrencyFlag,
261+
Concurrency: concurrency,
244262
ExitFirstHit: exitFirstHitFlag,
245263
ExitFirstMiss: exitFirstMissFlag,
246264
IgnoreSelf: ignoreSelfFlag,
@@ -251,8 +269,9 @@ func main() {
251269
OCI: ociFlag,
252270
QuantityIncreasesRisk: quantityIncreasesRiskFlag,
253271
Renderer: renderer,
254-
RuleFS: rfs,
272+
Rules: yrs,
255273
ScanPaths: scanPaths,
274+
ScannerPool: pool,
256275
Stats: statsFlag,
257276
}
258277

‎go.mod‎

Lines changed: 12 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
module github.com/chainguard-dev/malcontent
22

3-
go 1.23.3
3+
go 1.23.4
44

55
require (
6+
github.com/VirusTotal/yara-x/go v0.12.0
67
github.com/agext/levenshtein v1.2.3
78
github.com/cavaliergopher/cpio v1.0.1
89
github.com/cavaliergopher/rpm v1.2.0
@@ -15,7 +16,6 @@ require (
1516
github.com/gabriel-vasile/mimetype v1.4.8
1617
github.com/google/go-cmp v0.6.0
1718
github.com/google/go-containerregistry v0.20.2
18-
github.com/hillu/go-yara/v4 v4.3.3
1919
github.com/klauspost/compress v1.17.11
2020
github.com/olekukonko/tablewriter v0.0.5
2121
github.com/shirou/gopsutil/v4 v4.24.12
@@ -31,21 +31,21 @@ require (
3131
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
3232
github.com/bahlo/generic-list-go v0.2.0 // indirect
3333
github.com/buger/jsonparser v1.1.1 // indirect
34-
github.com/charmbracelet/x/ansi v0.4.5 // indirect
34+
github.com/charmbracelet/x/ansi v0.6.0 // indirect
3535
github.com/charmbracelet/x/term v0.2.1 // indirect
36-
github.com/containerd/stargz-snapshotter/estargz v0.15.1 // indirect
37-
github.com/cpuguy83/go-md2man/v2 v2.0.5 // indirect
38-
github.com/docker/cli v27.3.1+incompatible // indirect
36+
github.com/containerd/stargz-snapshotter/estargz v0.16.3 // indirect
37+
github.com/cpuguy83/go-md2man/v2 v2.0.6 // indirect
38+
github.com/docker/cli v27.4.1+incompatible // indirect
3939
github.com/docker/distribution v2.8.3+incompatible // indirect
4040
github.com/docker/docker-credential-helpers v0.8.2 // indirect
41-
github.com/ebitengine/purego v0.8.1 // indirect
41+
github.com/ebitengine/purego v0.8.2 // indirect
4242
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
4343
github.com/go-ole/go-ole v1.3.0 // indirect
4444
github.com/kr/pretty v0.2.1 // indirect
4545
github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
4646
github.com/lufia/plan9stats v0.0.0-20240909124753-873cd0166683 // indirect
47-
github.com/mailru/easyjson v0.7.7 // indirect
48-
github.com/mattn/go-colorable v0.1.13 // indirect
47+
github.com/mailru/easyjson v0.9.0 // indirect
48+
github.com/mattn/go-colorable v0.1.14 // indirect
4949
github.com/mattn/go-isatty v0.0.20 // indirect
5050
github.com/mattn/go-localereader v0.0.1 // indirect
5151
github.com/mattn/go-runewidth v0.0.16 // indirect
@@ -65,9 +65,10 @@ require (
6565
github.com/vbatts/tar-split v0.11.6 // indirect
6666
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1 // indirect
6767
github.com/yusufpapurcu/wmi v1.2.4 // indirect
68-
golang.org/x/crypto v0.31.0 // indirect
69-
golang.org/x/net v0.33.0 // indirect
68+
golang.org/x/crypto v0.32.0 // indirect
69+
golang.org/x/net v0.34.0 // indirect
7070
golang.org/x/sys v0.29.0 // indirect
7171
golang.org/x/text v0.21.0 // indirect
72+
google.golang.org/protobuf v1.36.2 // indirect
7273
pault.ag/go/topsort v0.1.1 // indirect
7374
)

0 commit comments

Comments
 (0)