Skip to content

Commit 63ef7bb

Browse files
plasticuprojectplasticuproject
committed
Initial commit
0 parents  commit 63ef7bb

File tree

5 files changed

+532
-0
lines changed

5 files changed

+532
-0
lines changed

‎.github/workflows/rust.yml‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
name: Lint Build Release
2+
3+
on:
4+
push:
5+
branches:
6+
- '**'
7+
tags:
8+
- 'v*'
9+
pull_request:
10+
branches:
11+
- '**'
12+
13+
jobs:
14+
test:
15+
runs-on: ubuntu-latest
16+
17+
steps:
18+
- name: Checkout code
19+
uses: actions/checkout@v4
20+
21+
- name: Run tests
22+
run: cargo test --verbose
23+
24+
- name: Run Clippy
25+
run: cargo clippy --all-targets --all-features -- -D warnings
26+
27+
build:
28+
runs-on: ubuntu-latest
29+
if: startsWith(github.ref, 'refs/tags/')
30+
31+
steps:
32+
- name: Checkout code
33+
uses: actions/checkout@v4
34+
35+
- name: Build the project
36+
run: cargo build --release
37+
38+
- name: Package build
39+
run: |
40+
mkdir -p release
41+
cp target/release/dashboard_datatable_builder release/
42+
cd release
43+
tar -czf dashboard_datatable_builder-${{ github.sha }}-${{ github.ref_name }}.tar.gz dashboard_datatable_builder
44+
45+
- name: List Release Directory
46+
run: ls -la release/
47+
48+
- name: Upload Release Artifact
49+
uses: actions/upload-artifact@v4
50+
with:
51+
name: dashboard_datatable_builder-${{ github.sha }}-${{ github.ref_name }}
52+
path: release/dashboard_datatable_builder-${{ github.sha }}-${{ github.ref_name }}.tar.gz
53+
54+
release:
55+
runs-on: ubuntu-latest
56+
needs: build
57+
if: startsWith(github.ref, 'refs/tags/')
58+
59+
permissions:
60+
contents: write
61+
62+
steps:
63+
- name: Checkout code
64+
uses: actions/checkout@v4
65+
66+
- name: Download artifact
67+
uses: actions/download-artifact@v4
68+
with:
69+
name: dashboard_datatable_builder-${{ github.sha }}-${{ github.ref_name }}
70+
71+
- name: Create GitHub Release
72+
uses: softprops/action-gh-release@v2
73+
with:
74+
files: dashboard_datatable_builder-${{ github.sha }}-${{ github.ref_name }}.tar.gz
75+
env:
76+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎.gitignore‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
/# Generated by Cargo
2+
# will have compiled files and executables
3+
debug/
4+
target/
5+
6+
# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
7+
# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
8+
Cargo.lock
9+
10+
# These are backup files generated by rustfmt
11+
**/*.rs.bk
12+
13+
# MSVC Windows builds of rustc generate these, which store debugging information
14+
*.pdbtarget

‎Cargo.toml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
[package]
2+
name = "dashboard_datatable_builder"
3+
version = "0.1.0"
4+
edition = "2021"
5+
rust-version = "1.80"
6+
authors = ["plasticuproject <[email protected]>"]
7+
description = "CC/B1 Minimial dashboard datatable builder."
8+
readme = "README.md"
9+
repository = "https://github.com/plasticuproject/dashboard_datatable_builder"
10+
11+
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
12+
13+
[dependencies]
14+
regex = "1"
15+
chrono = "0.4"
16+
csv = "1.1"
17+
serde_json = "1.0"
18+
serde = { version = "1.0", features = ["derive"] }
19+
20+
[badges.maintenance]
21+
status = "actively-developed"
22+
23+
[lints.rust]
24+
unsafe_code = "forbid"
25+
26+
[lints.clippy]
27+
enum_glob_use = "deny"
28+
pedantic = { level = "deny", priority = -1 }
29+
nursery = { level = "deny", priority = -1 }
30+
unwrap_used = "deny"
31+
32+
[profile.release]
33+
opt-level = 'z' # Optimize for size.
34+
lto = true # Enable Link Time Optimisation
35+
codegen-units = 1 # Reduced to increase optimisations.
36+
panic = 'abort' # Abort on panic
37+
strip = "symbols" # Strip symbols from binary

‎README.md‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
[![Rust 1.80](https://img.shields.io/badge/rust-1.80+-red.svg)](https://www.rust-lang.org/tools/install)
2+
[![Lint Build Release](https://github.com/plasticuproject/dashboard_datatable_builder/actions/workflows/rust.yml/badge.svg)](https://github.com/plasticuproject/dashboard_datatable_builder/actions/workflows/rust.yml)
3+
![Maintenance](https://img.shields.io/badge/maintenance-actively--developed-brightgreen.svg)
4+
5+
# Dashboard DataTable Builder
6+
7+
The `dashboard_datatable_builder` is a Rust-based tool designed to process large volumes of CSV log data, specifically targeting structured log files with threat indicators, priorities, and other critical information. It extracts, cleans, consolidates, and sorts data into a succinct CSV format, facilitating the generation of data tables for dashboard integrations or further analysis.
8+
9+
## Features
10+
11+
- **Pattern-based File Selection**: Chooses log files for processing based on their names and modification dates, focusing on recent data.
12+
- **Data Cleaning and Deduplication**: Utilizes regular expressions to clean 'Event Description' fields and ensures that only unique data entries are included in the output.
13+
- **Date Filtering**: Filters log entries to include only those within a specified range of recent days, allowing for targeted analysis. Entries are first filtered by user-specified days and then by a default retention period of 15 days.
14+
- **Condition-based Filtering**: Further filters entries to include only those that are flagged as 'Blocked' in the dataset.
15+
- **Efficient Data Handling**: Processes files in a memory-efficient manner, suitable for handling large datasets without overwhelming system resources.
16+
- **Sorted CSV Output**: Produces an `events.csv` file that combines filtered, cleaned, and sorted (by date) entries for easy use in data tables or dashboards.
17+
18+
## Getting Started
19+
20+
### Prerequisites
21+
22+
- Rust 1.8.0 or later
23+
- Cargo for managing Rust packages
24+
25+
### Installation
26+
27+
1. Clone the repository:
28+
29+
```sh
30+
git clone https://github.com/plasticuproject/dashboard_datatable_builder.git
31+
```
32+
33+
2. Navigate to the project directory:
34+
35+
```sh
36+
cd dashboard_datatable_builder
37+
```
38+
39+
3. Build the project:
40+
41+
```sh
42+
cargo build --release
43+
```
44+
45+
### Usage
46+
47+
Ensure your designated directory (e.g., `/var/log/fwd/db/`) is populated with compatible log files.
48+
To run the aggregator, provide the path to your log files directory and the number of days back to filter the files based on their modification date as arguments:
49+
50+
```sh
51+
cargo run --release <path_to_log_files> <days_back>
52+
```
53+
54+
For example, to process logs from the last 15 days in the `/var/log/fwd/db` directory:
55+
56+
```sh
57+
cargo run --release /var/log/fwd/db 15
58+
```
59+
60+
You can also execute the pre-built binary in the directory where you want your output files to reside:
61+
62+
```sh
63+
./dashboard_datatable_builder /var/log/fwd/db 15
64+
```
65+
66+
The program will generate an `events.csv` file in the current working directory, containing the processed, aggregated, and sorted data.
67+
68+
### Post-Execution
69+
70+
After running the program, check the `events.csv` file for your aggregated data. This file is overwritten each time the program is run to ensure it contains only the most recent and relevant entries.
71+
72+
### Note
73+
74+
The default retention period for the final filtering step is set to 15 days. This means after initial processing based on the `days_back` parameter, entries older than 7 days from the current date are further filtered out before the final write operation.

0 commit comments

Comments
 (0)