Skip to content

Commit cf0618b

Browse files
michaelkedarmichaelkedar
authored
feat(staging): record checker (#3957)
Adds a cron job to validate the consistency of records between GCS and datastore with the new database format (#3850). This sends messages to the recoverer (#3821) to attempt to repair. The record-checker stores information from its previous runs in datastore in a new JobData entity - where the datastore Key is the name of the metadata, and the value is stored in a `value` property. I've written this in Go to because it's fairly simple and to set a precedent for writing / migrating other components away from python. I made a copy of the `logging` submodule from `vulnfeeds` into the new toplevel `go` directory - we'll want to consolidate these soon. There's also room for refactoring the record checker code have some reusable components for e.g. setting up datastore, gcs, or pub/sub. I added a validation test script to make sure datastore entities written in Python are compatible with the go definitions, and vice versa.
1 parent 93a55dd commit cf0618b

File tree

15 files changed

+1164
-1
lines changed

15 files changed

+1164
-1
lines changed

‎deployment/build-and-stage.yaml‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,20 @@ steps:
107107
args: ['push', '--all-tags', 'gcr.io/oss-vdb/recoverer']
108108
waitFor: ['build-recoverer', 'cloud-build-queue']
109109

110+
- name: 'gcr.io/cloud-builders/docker'
111+
entrypoint: 'bash'
112+
args: ['-c', 'docker pull gcr.io/oss-vdb/record-checker:latest || exit 0']
113+
id: 'pull-record-checker'
114+
waitFor: ['setup']
115+
- name: gcr.io/cloud-builders/docker
116+
args: ['build', '-t', 'gcr.io/oss-vdb/record-checker:latest', '-t', 'gcr.io/oss-vdb/record-checker:$COMMIT_SHA', '.']
117+
dir: 'go'
118+
id: 'build-record-checker'
119+
waitFor: ['pull-record-checker']
120+
- name: gcr.io/cloud-builders/docker
121+
args: ['push', '--all-tags', 'gcr.io/oss-vdb/record-checker']
122+
waitFor: ['build-record-checker', 'cloud-build-queue']
123+
110124
# Build/push staging-api-test images to gcr.io/oss-vdb-test.
111125
- name: gcr.io/cloud-builders/docker
112126
args: ['build', '-t', 'gcr.io/oss-vdb-test/staging-api-test:latest', '-t', 'gcr.io/oss-vdb-test/staging-api-test:$COMMIT_SHA', '.']
@@ -316,7 +330,8 @@ steps:
316330
nvd-cve-osv=gcr.io/oss-vdb/nvd-cve-osv:$COMMIT_SHA,\
317331
nvd-mirror=gcr.io/oss-vdb/nvd-mirror:$COMMIT_SHA,\
318332
recoverer=gcr.io/oss-vdb/recoverer:$COMMIT_SHA,\
319-
cve5-to-osv=gcr.io/oss-vdb/cve5-to-osv:$COMMIT_SHA"
333+
cve5-to-osv=gcr.io/oss-vdb/cve5-to-osv:$COMMIT_SHA,\
334+
record-checker=gcr.io/oss-vdb/record-checker:$COMMIT_SHA"
320335
]
321336
dir: deployment/clouddeploy/gke-workers
322337

@@ -374,3 +389,4 @@ images:
374389
- 'gcr.io/oss-vdb-test/osv-linter:$COMMIT_SHA'
375390
- 'gcr.io/oss-vdb/recoverer:$COMMIT_SHA'
376391
- 'gcr.io/oss-vdb/cve5-to-osv:$COMMIT_SHA'
392+
- 'gcr.io/oss-vdb/record-checker:$COMMIT_SHA'

‎deployment/clouddeploy/gke-workers/environments/oss-vdb-test/kustomization.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ resources:
33
- staging-api-test.yaml
44
- osv-linter.yaml
55
- cve5-to-osv.yaml
6+
- record-checker.yaml
67
patches:
78
- path: workers.yaml
89
- path: scaler.yaml

‎deployment/clouddeploy/gke-workers/environments/oss-vdb-test/record-checker.yaml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
apiVersion: batch/v1
2+
kind: CronJob
3+
metadata:
4+
name: record-checker
5+
labels:
6+
cronLastSuccessfulTimeMins: "90"
7+
spec:
8+
schedule: "10/15 * * * *"
9+
concurrencyPolicy: Forbid
10+
jobTemplate:
11+
spec:
12+
template:
13+
spec:
14+
containers:
15+
- name: record-checker
16+
image: record-checker
17+
env:
18+
- name: GOOGLE_CLOUD_PROJECT
19+
value: oss-vdb-test
20+
- name: OSV_VULNERABILITIES_BUCKET
21+
value: osv-test-vulnerabilities
22+
imagePullPolicy: Always
23+
resources:
24+
requests:
25+
cpu: "1"
26+
memory: "1G"
27+
limits:
28+
cpu: "1"
29+
memory: "2G"
30+
restartPolicy: Never

‎go/.golangci.yaml‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
version: "2"
2+
3+
linters:
4+
default: all
5+
# prettier-ignore
6+
disable:
7+
- forbidigo
8+
- paralleltest
9+
- tparallel
10+
- cyclop #
11+
- depguard # Too annoying
12+
- err113 # will re-add later (another-rex)
13+
- exhaustruct # overkill (g-rath)
14+
- forcetypeassert # too hard (g-rath)
15+
- funlen #
16+
- funcorder #
17+
- gochecknoglobals # disagree with, for non changing variables (another-rex)
18+
- gocognit #
19+
- goconst # not everything should be a constant
20+
- gocyclo #
21+
- godot # comments are fine without full stops (g-rath)
22+
- godox # to-do comments are fine (g-rath)
23+
- ireturn # disagree with, sort of (g-rath)
24+
- lll # line length is hard (g-rath)
25+
- maintidx #
26+
- mnd # not every number is magic (g-rath)
27+
- nestif #
28+
- noctx # Most of these don't need a context
29+
- noinlineerr #
30+
- nonamedreturns # disagree with, for now (another-rex)
31+
- tagliatelle # we're parsing data from external sources (g-rath)
32+
- testpackage # will re-add later (another-rex)
33+
- varnamelen # maybe later (g-rath)
34+
- wrapcheck # too difficult, will re-add later (another-rex)
35+
- wsl # disagree with, for now (g-rath)
36+
- wsl_v5 # disagree with, for now (g-rath)
37+
settings:
38+
exhaustive:
39+
default-signifies-exhaustive: true
40+
gocritic:
41+
disabled-checks:
42+
- ifElseChain
43+
nlreturn:
44+
block-size: 2
45+
revive:
46+
rules:
47+
- name: increment-decrement
48+
disabled: true
49+
- name: blank-imports
50+
disabled: false
51+
- name: context-as-argument
52+
disabled: false
53+
- name: context-keys-type
54+
disabled: false
55+
- name: dot-imports
56+
disabled: false
57+
- name: empty-block
58+
disabled: false
59+
- name: error-naming
60+
disabled: false
61+
- name: error-return
62+
disabled: false
63+
- name: error-strings
64+
disabled: false
65+
- name: errorf
66+
disabled: false
67+
- name: exported
68+
disabled: false
69+
arguments:
70+
# TODO: get these all enabled
71+
- "check-private-receivers"
72+
# - "check-public-interface"
73+
- "disable-checks-on-constants"
74+
- "disable-checks-on-functions"
75+
- "disable-checks-on-methods"
76+
- "disable-checks-on-types"
77+
- "disable-checks-on-variables"
78+
- name: import-alias-naming
79+
disabled: false
80+
- name: import-shadowing
81+
disabled: false
82+
- name: indent-error-flow
83+
disabled: false
84+
- name: package-comments
85+
disabled: false
86+
- name: range
87+
disabled: false
88+
- name: receiver-naming
89+
disabled: false
90+
- name: redefines-builtin-id
91+
disabled: false
92+
- name: redundant-test-main-exit
93+
disabled: false
94+
- name: superfluous-else
95+
disabled: false
96+
- name: time-naming
97+
disabled: false
98+
- name: unexported-return
99+
disabled: false
100+
- name: unreachable-code
101+
disabled: false
102+
- name: unused-parameter
103+
disabled: false
104+
- name: use-any
105+
disabled: false
106+
- name: var-declaration
107+
disabled: false
108+
- name: var-naming
109+
disabled: false
110+
arguments:
111+
- [] # AllowList
112+
- [] # DenyList
113+
- - skip-package-name-checks: true
114+
exclusions:
115+
generated: lax
116+
presets:
117+
- common-false-positives
118+
- legacy
119+
- std-error-handling
120+
rules:
121+
- path: _test\.go
122+
linters:
123+
- dupl
124+
- path-except: _test\.go
125+
text: use `testutility.GetCurrentWorkingDirectory`
126+
paths:
127+
- third_party$
128+
- builtin$
129+
- examples$
130+
131+
formatters:
132+
enable:
133+
- gofmt
134+
- goimports
135+
exclusions:
136+
generated: lax
137+
paths:
138+
- third_party$
139+
- builtin$
140+
- examples$
141+
142+
issues:
143+
max-issues-per-linter: 0
144+
max-same-issues: 0

‎go/cmd/recordchecker/Dockerfile‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Copyright 2025 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
FROM golang:1.25.0-alpine@sha256:f18a072054848d87a8077455f0ac8a25886f2397f88bfdd222d6fafbb5bba440 AS build
16+
17+
WORKDIR /src
18+
19+
COPY ./go.mod /src/go.mod
20+
COPY ./go.sum /src/go.sum
21+
RUN go mod download && go mod verify
22+
23+
24+
COPY ./ /src/
25+
RUN CGO_ENABLED=0 go build -o recordchecker ./cmd/recordchecker/
26+
27+
FROM gcr.io/distroless/static-debian12@sha256:87bce11be0af225e4ca761c40babb06d6d559f5767fbf7dc3c47f0f1a466b92c
28+
29+
COPY --from=build /src/recordchecker /
30+
31+
ENTRYPOINT ["/recordchecker"]

0 commit comments

Comments
 (0)