You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -12,103 +12,178 @@ Overall, `fastcve` is a useful tool for anyone who is interested in keeping up-t
12
12
Technical details
13
13
-----------------
14
14
15
+
Some key technical characteristics of `fastcve`:
15
16
16
-
Some key technical characteristics of `fastcve`tool:
17
+
1.**Docker containerization**: Runs as a Docker Compose stack with two services: `fastcve`(API + CLI) and `fastcve-db` (PostgreSQL).
17
18
18
-
1.**Docker containerization**: This tool would run as a Docker container, meaning it is packaged and runs as a standalone executable unit that includes all its dependencies. This makes it easy to deploy and run the tool on any system that supports Docker.
19
+
2.**Automatically creates the DB on first start**: PostgreSQL creates the initial database (`POSTGRES_DB`) when the DB volume is empty.
19
20
20
-
2.**Automatically creates the DB on first start**: The tool would automatically create the database when it is started for the first time, eliminating the need for manual setup.
21
+
3.**Automatically creates and upgrades the DB schema if needed**: The schema is created/updated automatically on app start and before CLI commands run.
21
22
22
-
3.**Automatically creates and upgrades the DB schema if needed**: The tool would automatically manage the database schema, creating it when necessary and upgrading it as needed to support new features or changes in the data structure.
23
+
4.**Capability to populate the DB using external sources**: The `load` command can populate the DB from NVD (CVE/CPE via NVD 2.0 APIs) and additional sources (MITRE CWE/CAPEC, EPSS, CISA KEV).
23
24
24
-
3.**Capability to populate the DB using NVD APIs (2.0)**: The tool would have the capability to populate the database using APIs from the National Vulnerability Database (NVD), a comprehensive database of vulnerabilities that is maintained by the US National Institute of Standards and Technology.
25
+
5.**Incremental updates**: Re-running `load` fetches only changes since the last successful update (NVD increments are limited to `fetch.max.days.period`, default 120 days).
25
26
26
-
4.**Automatically updates the DB schema for the latest changes from the previous update**: The tool would automatically update the database with the latest changes, new vulnerabilities, and new Common Platform Enumeration (CPE) entries, ensuring that the database remains up-to-date and relevant.
27
+
The tool is a comprehensive solution for managing a local vulnerability database and querying it via CLI and HTTP API.
27
28
28
-
The tool is a comprehensive solution for managing a database of vulnerabilities, Common Platform Enumeration (CPE) entries, designed to make it easy to keep the database up-to-date and relevant, and to simplify the process of querying the database for vulnerabilities and information about them.
29
-
30
-
It was designed to be able to handle multiple query entries and maintain a high level of performance. It is optimized to be able to handle a large number of queries efficiently and quickly. The goal is to provide a fast and reliable solution for searching, maintaining and updating information about vulnerabilities and their corresponding CPE entries. The tool is also able to provide information about CWE (Common Weakness Enumerations) and CAPEC (Common Attack Pattern Enumerations and Classification)
29
+
It is optimized to handle many queries efficiently, and to keep vulnerability data up to date via incremental updates.
31
30
32
31
33
32
Build
34
33
----------
35
34
36
35
37
-
In order to build the docker image you need to trigger:
38
-
```
39
-
docker compose build
36
+
To build the application image:
37
+
```bash
38
+
docker compose build fastcve
40
39
```
41
40
42
-
this would create a new image named as `fastcve:latest`. In case a special tag is needed then export before hand ENV var `DOCKER_TAG=<your_tag>` to generate `fastcve:<your_tag>`
41
+
This builds `${FASTCVE_DOCKER_IMG}:${FASTCVE_DOCKER_TAG}` (application / API / CLI). The default values are in `.env`.
42
+
43
+
The DB container uses an upstream PostgreSQL image (defaults to `postgres:16-alpine3.19`, configurable via `FASTCVE_DB_IMAGE`) and is not built from this repo.
44
+
45
+
If you need a custom tag, set `FASTCVE_DOCKER_TAG=<your_tag>` to generate `${FASTCVE_DOCKER_IMG}:<your_tag>`.
46
+
47
+
48
+
Migration (single-container -> split DB/app)
49
+
--------------------------------------------
50
+
51
+
Previous versions (-> v1.2.3) ran the PostgreSQL DB and the FastCVE API/CLI in the same `fastcve` container. The current `docker-compose.yml` splits this into two services: `fastcve-db` (PostgreSQL) and `fastcve` (API/CLI).
52
+
53
+
To migrate an existing deployment:
54
+
55
+
1. Stop the old container/stack **without deleting the DB volume**:
56
+
```
57
+
docker compose down
58
+
```
59
+
60
+
2. Update your environment variables / `.env` for the new compose file:
61
+
- Keep `FCDB_USER` and `FCDB_PASS` the same as before (so the existing DB user credentials still match).
62
+
-`FCDB_USER`/`FCDB_PASS` are used for both:
63
+
- the app connection (`postgresql://${FCDB_USER}:${FCDB_PASS}@...`), and
64
+
- PostgreSQL initialization (`POSTGRES_USER`/`POSTGRES_PASSWORD`) when the DB volume is empty.
65
+
- For Docker Compose deployments, set `INP_ENV_NAME=dev` (the default). (`setenv_dev.ini` points to the in-stack DB hostname `fastcve-db`.)
66
+
- (Optional) Set `FASTCVE_DB_IMAGE` if you need a different PostgreSQL image/tag.
67
+
68
+
3. If you previously bind-mounted `/fastcve/config/setenv` from the host, ensure the *mounted*`setenv_${INP_ENV_NAME}.ini` matches the new split:
69
+
-`FCDB_HOST=fastcve-db`
70
+
-`FCDB_PORT=5432`
71
+
72
+
4. Start the new split stack:
73
+
```
74
+
docker compose up -d
75
+
```
76
+
77
+
Notes:
78
+
- The DB data is still kept in the Docker volume `vol_fastcve_db`. Do not use `docker compose down -v` unless you intentionally want to wipe the DB.
79
+
- If your existing `vol_fastcve_db` was created by an older PostgreSQL *major* version, you must do a `pg_dump`/restore (PostgreSQL data directories are not forward-compatible across major versions).
80
+
- DB admin commands that used to run in `fastcve` now run in `fastcve-db` (example: `docker compose exec fastcve-db psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"`).
81
+
- If you change `FCDB_PASS` for an already-initialized `vol_fastcve_db`, PostgreSQL will *not* automatically update the user password; you must run `ALTER USER ... PASSWORD ...` (or recreate the volume).
43
82
44
83
45
84
First Run
46
85
---------
47
86
48
-
Before starting the container for the first time several env variables has to be exported first:
49
-
```
50
-
export INP_ENV_NAME=<the env name from where settings to be read . i.e.:dev>
51
-
export POSTGRES_PASSWORD=<password for postgre DB user>
52
-
export FCDB_USER=<fastcve DB User Name>
53
-
export FCDB_PASS=<fastcve DB User Password>
87
+
Docker Compose reads `.env` automatically. The defaults in this repo are enough to start a local stack.
88
+
89
+
Create a local folder for the `/backup` bind-mount used by `fastcve-db` (Linux):
90
+
```bash
91
+
mkdir -p backup && chmod 1777 backup
54
92
```
55
93
56
-
To run the container:
94
+
If you want to override them, set variables in your shell or edit `.env`:
95
+
```bash
96
+
export INP_ENV_NAME=dev
97
+
export FCDB_USER=fastcve_db_user
98
+
export FCDB_PASS=fastcve_db_pass
99
+
export FASTCVE_DOCKER_IMG=binare/fastcve
100
+
export FASTCVE_DOCKER_TAG=latest
57
101
```
58
-
docker compose up
102
+
103
+
To start the stack:
104
+
```bash
105
+
docker compose up -d --build
59
106
```
60
107
108
+
Troubleshooting
109
+
---------------
110
+
111
+
-`password authentication failed for user ...`: the DB is stored in `vol_fastcve_db`, so if the volume was initialized with different credentials you must reset it with `docker compose down -v` (or manually `ALTER USER` inside Postgres).
112
+
61
113
Configuration parameters
62
114
------------------------
63
115
64
-
The docker container holds the relevant configuration files under `/fastcve/config/setenv`
65
-
This map would contain the following files:
66
-
-`config.ini` - contains app related config parameters
67
-
-`setenv_<${INP_ENV_NAME}>.ini` - contains parameters that are env dependent as per the `INP_ENV_NAME` env variable value.
116
+
The application container contains the configuration under `/fastcve/config`:
-`/fastcve/config/setenv/setenv_${INP_ENV_NAME}.ini` - env-specific settings loaded by `/fastcve/config/setenv.sh`
68
119
69
-
For an easy access and modification of the config files, mapping between HOST and container's folder `/fastcve/config/setenv` is recomended that can be specified in the `docker-compose.yml` file.
120
+
In this repo, those files are located under `src/config` and are copied into the image during build.
121
+
122
+
For easier local edits, you can bind-mount config into the `fastcve` container (example):
Note: `INP_ENV_NAME=prod` is not configured for Docker Compose out-of-the-box (`setenv_prod.ini` uses `FCDB_HOST=localhost`). For the split stack, use `dev` or update `setenv_prod.ini` to point to `fastcve-db:5432`.
70
130
71
131
How To
72
132
------
73
133
74
134
75
-
-**Populate the CVE, CPE, CWE, and CAPEC data for the first time**:
135
+
- **Backup the DB (pg_dump, compact)**:
136
+
> Important:
137
+
> The dump is written inside the DB container to `/backup` (bind-mounted from the host via `docker-compose.yml`).
138
+
> Ensure the host `./backup` directory exists and is writable (Linux):
this will fetch the new/modified data for the period from the last data fetch (for a max of `n` days: parameter set in the config. NVD is allowing max 120 days period)
160
+
This fetches changes since the last successful update (for CVE/CPE), with an upper limit of `fetch.max.days.period` (default 120 days) enforced by the loader.
86
161
87
-
If there is a need to repopulate the DB for the CWE/CAPEC info, then `--full` and `--drop` options are available for the CWE/CAPEC info load command. `--full`will cause to ignore the fact the CWE/CAPEC data is already present and `--drop`will cause to drop any exiting CWE/CAPEC related data before processing the new downloaded data. When using `--data epss` in combination with `--epss-now`, the load command explicitly fetches the EPSS data for the current date. If `--epss-now` is not specified, the script defaults to retrieve EPSS data from the previous day.
162
+
If there is a need to repopulate the DB for the CWE/CAPEC info, then `--full` and `--drop` options are available for the load command. `--full` ignores the fact the data is already present and `--drop` drops existing data before loading. When using `--data epss` in combination with `--epss-now`, the loader fetches EPSS data for the current date; otherwise it defaults to the previous day.
88
163
89
164
- search for the data: **get the CVEs details (JSON) for a list of CVE-IDs**
Above will search in the CVE text and return the details of those CVEs that would match the summary/description text with the specified keyword. It is possible to specify more than one keywords and each keyword can be specified in form of regexp pattern. If multiple keywords are specified, it would consider as AND condition between the keywords.
173
+
Above will search CVE text and return details for CVEs that match the given keyword(s). Multiple `--keyword` values are treated as an AND condition. Each keyword can be a regular expression.
99
174
100
175
- search for the data: **get the CVEs IDs for a specific CPE**
101
176
```
102
-
docker exec fastcve search --search-info cve --cpe23 cpe:2.3:*:*:linux_kernel:2.6.32: --output id
above will return the list of CVE-IDs that are related to `linux_kernel` product for version 2.6.32.
180
+
Above will return the list of CVE-IDs that are related to the `linux_kernel` product for version 2.6.32.
106
181
107
-
To get the CVE details, request the output in JSON format: `--output json`.
182
+
To get the CVE details, request the output in JSON format: `--output json`.
108
183
109
184
To get only those CVEs that were modified in the last `n` days, add the option `--days-back n` i.e. `--days-back 10` - only created/modified in the last **10** days
110
185
111
-
additional filters are available for the search in CVE DB:
186
+
Additional filters are available for CVE search:
112
187
```
113
188
--cvss-severity-v2 {low, medium, high} # retrieve only those CVEs that has the severity as per CVSS score Version 2
114
189
--cvss-severity-v3 {low, medium, high, critical} # retrieve only those CVEs that has the severity as per CVSS score Version 3.x
@@ -123,34 +198,35 @@ additional filters are available for the search in CVE DB:
123
198
- search for the data: **get the valid list of CPE names for a query on part/vendor/product/version etc**.
124
199
125
200
```
126
-
docker exec fastcve search --search-info cpe --cpe23 cpe:2.3:h:*:dl*:*: --output id
Above will search for all valid existing CPE 2.3 names that are of hardware type, for any vendor, product starts with `dl`*, and version is any
130
205
131
206
To see for the other options available for both `load` and `search` commands run these with `-h` option
132
207
133
208
```
134
-
docker exec fastcve search -h
135
-
docker exec fastcve load -h
209
+
docker compose exec fastcve search -h
210
+
docker compose exec fastcve load -h
136
211
```
137
212
138
-
The same search capabilities are exposed through the web interface as well. The web interface is exposed through port 8000 by default. Can be changed in the `docker-compose.yml` file.
213
+
The same search capabilities are exposed through the API (FastAPI). The API is exposed through port 8000 by default and can be changed in `docker-compose.yml`.
139
214
140
215
The following endpoints are exposed through HTTP requests
141
216
```
217
+
/status - DB status
142
218
/api/search/cve - search for CVE data
143
219
/api/search/cpe - search for CPE data
144
220
/api/search/cwe - search for CWE data
145
221
/api/search/capec - search for CAPEC data
146
222
```
147
223
148
-
For the api definitions and execution instructions you can access on
0 commit comments