Skip to content

Commit a4b80cf

Browse files
userjack6880userjack6880
authored
Merge pull request #6 from userjack6880/dev
Dev to Main 0-α2
2 parents 60f6246 + ece6930 commit a4b80cf

File tree

10 files changed

+922
-284
lines changed

10 files changed

+922
-284
lines changed

‎.github/ISSUE_TEMPLATE/bug_report.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,8 @@ A clear and concise description of what you expected to happen.
2424
If applicable, add screenshots to help explain your problem.
2525

2626
**Software Version**
27-
[ ] Version 0 Alpha 1
27+
[] Version 0 Alpha 2
28+
[] Version 0 Alpha 1
2829

2930
**Server (please complete the following information):**
3031
- Server OS [e.g. CentOS 7.3]

‎.github/ISSUE_TEMPLATE/critical_report.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,8 @@ A clear and concise description of what you expected to happen.
2424
If applicable, add screenshots to help explain your problem.
2525

2626
**Software Version**
27-
[ ] Version 0 Alpha 1
27+
[] Version 0 Alpha 2
28+
[] Version 0 Alpha 1
2829

2930
**Server (please complete the following information):**
3031
- Server OS [e.g. CentOS 7.3]

‎README.md‎

Lines changed: 53 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ A Perl based tool to parse DMARC reports, based on John Levine's [rddmarc](http:
88

99
Open Report Parser is a fork of [techsneeze's dmarcts-report-parser](https://github.com/techsneeze/dmarcts-report-parser), and was forked to more closely match the needs of [Open DMARC Analyzer](https://github.com/userjack6880/Open-DMARC-Analyzer).
1010

11-
Open Report Parser Version 0 Alpha 1 (0-α1) is an [Anomaly \<Codebase\>](https://systemanomaly.com/codebase) project by John Bradley ([email protected]).
11+
Open Report Parser Version 0 Alpha 1 (0-α2) is an [Anomaly \<Codebase\>](https://systemanomaly.com/codebase) project by John Bradley ([email protected]).
1212

1313
# Minimum Requirements
1414

@@ -93,21 +93,26 @@ innodb_file_per_table = true
9393

9494
# Configuration Options
9595

96-
**Debug Options**
96+
**Script Options**
9797

9898
```perl
9999
$debug = 0;
100100
$delete_reports = 0;
101+
#$dmarc_only = 0;
102+
# if set to 1, do not process tls reports, if set to -1 do not process
103+
# dmarc reports - defaults to 1 for legacy support
104+
# this is ignored for all methods except IMAP, use --tls to process
105+
# TLS reports for other methods
101106
```
102107

103108
**Database Options**
104109

105110
```perl
106-
#$dbtype = 'mysql'; # Supported types - mysql, postgres - defaults to mysql if unset
111+
#$dbtype = 'mysql'; # Supported types - mysql, postgres - defaults to mysql if unset
107112
$dbname = 'dmarc';
108113
$dbuser = 'dmarc';
109114
$dbpass = 'password';
110-
$dbhost = 'dbhost'; # Set the hostname if we can't connect to the local socket.
115+
#$dbhost = 'dbhost'; # Set the hostname if we can't connect to the local socket.
111116
$dbport = '3306';
112117
```
113118

@@ -118,33 +123,40 @@ $imapserver = 'imap.server';
118123
$imapuser = 'username';
119124
$imappass = 'password';
120125
$imapport = '143';
121-
$imapssl = '0'; # If set to 1, remember to change server port to 993 and disable imaptls.
122-
$imaptls = '0'; # Enabled as the default and best-practice.
123-
$tlsverify = '0'; # Enable verify server cert as the default and best-practice.
124-
$imapignoreerror = '0'; # set it to 1 if you see an "ERROR: message_string()
125-
# expected 119613 bytes but received 81873 you may
126-
# need the IgnoreSizeErrors option" because of malfunction
127-
# imap server as MS Exchange 2007, ...
128-
$imapreadfolder = 'dmarc';
129-
130-
# If $imapmovefolder is set, processed IMAP messages will be moved (overruled by
126+
$imapssl = '0'; # If set to 1, remember to change server port to 993 and disable imaptls.
127+
$imaptls = '0';
128+
$tlsverify = '0';
129+
$imapignoreerror = '0'; # recommended if you use MS Exchange 2007, ...
130+
131+
$imapdmarcfolder = 'dmarc';
132+
$imaptlsfolder = 'tls';
133+
134+
# If $imapxxxproc is set, processed IMAP messages will be moved (overruled by
131135
# the --delete option!)
132-
$imapmovefolder = 'dmarc/processed';
136+
# $imapdmarcproc = 'dmarc.Processed';
137+
# $imaptlsproc = 'tls.Processed';
138+
139+
# If $imapxxxerr is set, IMAP messages that fail will be moved. If unset, failed messages
140+
# will move to $imapdmarcproc (if it is set). Overruled by the --delete option!
141+
# $imapdmarcerr = 'dmarc.notProcessed';
142+
# $imaptlserr = 'tls.notProcessed';
133143
```
134144

135145
These settings are ignored when using the -m flag.
136146

137147
**XML Storage Options**
138148

139149
```perl
140-
# maximum size of XML files to store in database, long files can cause transaction aborts
150+
# maximum size of XML/JSON files to store in database, long files can cause transaction aborts
141151
$maxsize_xml = 50000;
152+
$maxsize_json = 50000;
142153

143-
# store XML as base64 encopded gzip in database (save space, harder usable)
154+
# store XML/JSON as base64 encopded gzip in database (save space, harder usable)
144155
$compress_xml = 0;
156+
$compress_json = 0;
145157
```
146158

147-
**Processing Failure Action***
159+
**Processing Failure Action**
148160

149161
```perl
150162
# if there was an error during file processing (message does not contain XML or ZIP parts,
@@ -161,7 +173,7 @@ Note: Be sure to use the proper hierarchy separator for your server in all folde
161173
# Usage
162174

163175
```
164-
./dmarcts-report-parser.pl [OPTIONS] [PATH]
176+
./report-parser.pl [OPTIONS] [PATH]
165177
```
166178

167179
PATH can be the filename of a single file or a list of files - wildcard expression are allowed.
@@ -171,27 +183,35 @@ PATH can be the filename of a single file or a list of files - wildcard expressi
171183
One of the following source options must be provided:
172184

173185
```
174-
-i : Read reports from messages on IMAP server as defined in the config file.
175-
-m : Read reports from mbox file(s) provided in PATH.
176-
-e : Read reports from MIME email file(s) provided in PATH.
177-
-x : Read reports from xml file(s) provided in PATH.
178-
-z : Read reports from zip file(s) provided in PATH.
186+
-i : Read reports from messages on IMAP server as defined in the
187+
config file.
188+
-m : Read reports from mbox file(s) provided in PATH.
189+
-e : Read reports from MIME email file(s) provided in PATH.
190+
-x : Read reports from xml file(s) provided in PATH.
191+
-j : Read reports from json files(s) provided in PATH.
192+
-z : Read reports from zip file(s) provided in PATH.
179193
```
180194

181195
The following options are always allowed:
182196

183197
```
184-
-d : Print debug info.
185-
-r : Replace existing reports rather than failing.
186-
--delete : Delete processed message files (the XML is stored in the database for later reference).
187-
--info | Print out number of XML files or emails processed.
198+
-d : Print debug info.
199+
-r : Replace existing reports rather than skipping them.
200+
--delete : Delete processed message files (the XML is stored in the
201+
database for later reference).
202+
--info : Print out number of XML files or emails processed.
203+
--tls : Force TLS-Only Mode.
188204
```
189205

206+
Currently, processing of both DMARC and TLS reports during the same run is only supported from IMAP. All other sources will *always* default to DMARC reports unless the `--tls` flag is provided at runtime.
207+
190208
# Latest Changes
191209

192-
## 0-α1
193-
- Fork renamed
194-
- Incorporate changes made to original repository after fork ([commit 51ba1de](https://github.com/userjack6880/Open-Report-Parser/commit/51ba1de8521559647ebe4b8a1db291c26b572de4))
210+
## 0-α2
211+
- Fixed errors in previous release incorporating postgres support related to table creation.
212+
- Added MTA-TLS report support.
213+
- More useful debug output.
214+
- Code consolodation (eg, subroutine repetative code).
195215

196216
# Tested System Configurations
197217
| OS | Perl | SQL |
@@ -208,7 +228,8 @@ Support will be provided as outlined in the following schedule. For more details
208228

209229
| Version | Support Level | Released | End of Support | End of Life |
210230
| ----------------------------------- | ---------------- | ---------------- | ---------------- | ---------------- |
211-
| Version 1 Alpha 1 | Full Support | TBD | TBD | TBD |
231+
| Version 1 Alpha 2 | Full Support | 26 April 2023 | TBD | TBD |
232+
| Version 1 Alpha 1 | Critical Support | 19 April 2023 | 26 April 2023 | TBD |
212233

213234
# Contributing
214235

‎dbx_mysql.pl‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,40 @@
110110
table_options => "",
111111
indexes => [],
112112
},
113+
"tls" => {
114+
column_definitions => [
115+
"serial" , "int" , "unsigned NOT NULL AUTO_INCREMENT",
116+
"mindate" , "timestamp" , "NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP",
117+
"maxdate" , "timestamp" , "NULL",
118+
"domain" , "varchar(255)" , "NOT NULL",
119+
"org" , "varchar(255)" , "NOT NULL",
120+
"reportid" , "varchar(255)" , "NOT NULL",
121+
"email" , "varchar(255)" , "NULL",
122+
"policy_mode" , "varchar(20)" , "NULL",
123+
"summary_success" , "int" , "NULL",
124+
"summary_failure" , "int" , "NULL",
125+
"raw_json" , "mediumtext" , "",
126+
],
127+
additional_definitions => "PRIMARY KEY (serial), UNIQUE KEY domain(domain, reportid)",
128+
table_options => "ROW_FORMAT=COMPRESSED",
129+
indexes => [],
130+
},
131+
"tlsrecord" => {
132+
column_definitions => [
133+
"id" , "int" , "unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY",
134+
"serial" , "int" , "unsigned NOT NULL",
135+
"send_ip" , "int" , "unsigned",
136+
"send_ip6" , "binary(16)" , "",
137+
"recv_ip" , "int" , "unsigned",
138+
"recv_ip6" , "binary(16)" , "",
139+
"recv_mx" , "varchar(255)" , "",
140+
"type" , "varchar(255)" , "",
141+
"count" , "int" , "unsigned NOT NULL",
142+
],
143+
additional_definitions => "KEY serial (serial, send_ip), KEY serial6 (serial, send_ip6)",
144+
table_options => "",
145+
indexes => [],
146+
}
113147
},
114148

115149
add_column => sub {

‎dbx_postgres.pl‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,45 @@
115115
"CREATE INDEX rptrecord_idx_serial6 ON rptrecord (serial, ip6);",
116116
],
117117
},
118+
"tls" => {
119+
column_definitions => [
120+
"serial" , "bigint" , "GENERATED ALWAYS AS IDENTITY",
121+
"mindate" , "timestamp without time zone" , "NOT NULL",
122+
"maxdate" , "timestamp without time zone" , "NULL",
123+
"domain" , "character varying(255)" , "NOT NULL",
124+
"org" , "character varying(255)" , "NOT NULL",
125+
"reportid" , "character varying(255)" , "NOT NULL",
126+
"email" , "character varying(255)" , "NULL",
127+
"policy_mode" , "character varying(20)" , "NULL",
128+
"summary_success" , "integer" , "NULL",
129+
"summary_failure" , "integer" , "NULL",
130+
"raw_json" , "text" , "",
131+
],
132+
additional_definitions => "PRIMARY KEY (serial)",
133+
table_options => "",
134+
indexes => [
135+
"CREATE UNIQUE INDEX tls_uidx_domain ON tls (domain, reportid);"
136+
],
137+
},
138+
"tlsrecord" => {
139+
column_definitions => [
140+
"id" , "bigint" , "GENERATED ALWAYS AS IDENTITY",
141+
"serial" , "bigint" , "NOT NULL",
142+
"send_ip" , "bigint" , "",
143+
"send_ip6" , "bytea" , "",
144+
"recv_ip" , "bigint" , "",
145+
"recv_ip6" , "bytea" , "",
146+
"recv_mx" , "character varying(255)" , "",
147+
"type" , "character varying(255)" , "",
148+
"count" , "integer" , "NOT NULL",
149+
],
150+
additional_definitions => "PRIMARY KEY (id)",
151+
table_options => "",
152+
indexes => [
153+
"CREATE INDEX tlsrecord_idx_serial ON tlsrecord (serial, ip);",
154+
"CREATE INDEX tlsrecord_idx_serial6 ON tlsrecord (serial, ip6);",
155+
],
156+
}
118157
},
119158

120159
add_column => sub {

‎docs/CHANGELOG.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,10 @@
22

33
## 0-α1
44
- Fork renamed
5-
- Incorporate changes made to original repository after fork ([commit 51ba1de](https://github.com/userjack6880/Open-Report-Parser/commit/51ba1de8521559647ebe4b8a1db291c26b572de4))
5+
- Incorporate changes made to original repository after fork ([commit 51ba1de](https://github.com/userjack6880/Open-Report-Parser/commit/51ba1de8521559647ebe4b8a1db291c26b572de4))
6+
7+
## 0-α2
8+
- Fixed errors in previous release incorporating postgres support related to table creation.
9+
- Added MTA-TLS report support.
10+
- More useful debug output.
11+
- Code consolodation (eg, subroutine repetative code).

‎docs/SECURITY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,4 +20,5 @@ Open DMARC Analyzer Version Security Support
2020

2121
| Version | Supported | End of Life |
2222
| -------------------------- | --------- | ---------------- |
23+
| Version 0 Alpha 2 | YES | TBD |
2324
| Version 0 Alpha 1 | YES | TBD |

‎docs/SUPPORT.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,8 @@ Due to the limited time available to developrs to maintain support, dates must b
44

55
| Version | Support Level | Released | End of Support | End of Life |
66
| ----------------------------------- | ---------------- | ---------------- | ---------------- | ---------------- |
7-
| Version 1 Alpha 1 | Full Support | TBD | TBD | TBD |
7+
| Version 1 Alpha 2 | Full Support | 26 April 2023 | TBD | TBD |
8+
| Version 1 Alpha 1 | Critical Support | 19 April 2023 | 26 April 2023 | TBD |
89

910
# Support Cycle
1011

‎report-parser.conf.pub‎

Lines changed: 25 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -62,40 +62,48 @@
6262

6363
$debug = 0;
6464
$delete_reports = 0;
65+
#$dmarc_only = 0;
66+
# if set to 1, do not process tls reports, if set to -1 do not process
67+
# dmarc reports - defaults to 1 for legacy support
68+
# this is ignored for all methods except IMAP, use --tls to process
69+
# TLS reports for other methods
6570

66-
#$dbtype = 'mysql'; # Supported types - mysql, postgres - defaults to mysql if unset
71+
#$dbtype = 'mysql'; # Supported types - mysql, postgres - defaults to mysql if unset
6772
$dbname = 'dmarc';
6873
$dbuser = 'dmarc';
6974
$dbpass = 'password';
70-
$dbhost = 'dbhost'; # Set the hostname if we can't connect to the local socket.
75+
#$dbhost = 'dbhost'; # Set the hostname if we can't connect to the local socket.
7176
$dbport = '3306';
7277

7378
$imapserver = 'imap.server';
7479
$imapuser = 'username';
7580
$imappass = 'password';
7681
$imapport = '143';
77-
$imapssl = '1'; # If set to 1, remember to change server port to 993 and disable imaptls.
78-
$imaptls = '0'; # Enabled as the default and best-practice.
79-
$tlsverify = '1'; # Enable verify server cert as the default and best-practice.
80-
$imapignoreerror = '0'; # set it to 1 if you see an "ERROR: message_string()
81-
# expected 119613 bytes but received 81873 you may
82-
# need the IgnoreSizeErrors option" because of malfunction
83-
# imap server as MS Exchange 2007, ...
84-
$imapreadfolder = 'dmarc';
82+
$imapssl = '0'; # If set to 1, remember to change server port to 993 and disable imaptls.
83+
$imaptls = '0';
84+
$tlsverify = '0';
85+
$imapignoreerror = '0'; # recommended if you use MS Exchange 2007, ...
8586

86-
# If $imapmovefolder is set, processed IMAP messages will be moved (overruled by
87+
$imapdmarcfolder = 'dmarc';
88+
$imaptlsfolder = 'tls';
89+
90+
# If $imapxxxproc is set, processed IMAP messages will be moved (overruled by
8791
# the --delete option!)
88-
# $imapmovefolder = 'dmarc/processed';
92+
# $imapdmarcproc = 'dmarc.Processed';
93+
# $imaptlsproc = 'tls.Processed';
8994

90-
# If $imapmovefoldererr is set, IMAP messages that fail will be moved. If unset, failed messages
91-
# will move to $imapmovefolder (if it is set). Overruled by the --delete option!
92-
# $imapmovefoldererr = 'Inbox.notProcessed';
95+
# If $imapxxxerr is set, IMAP messages that fail will be moved. If unset, failed messages
96+
# will move to $imapdmarcproc (if it is set). Overruled by the --delete option!
97+
# $imapdmarcerr = 'dmarc.notProcessed';
98+
# $imaptlserr = 'tls.notProcessed';
9399

94-
# maximum size of XML files to store in database, long files can cause transaction aborts
100+
# maximum size of XML/JSON files to store in database, long files can cause transaction aborts
95101
$maxsize_xml = 50000;
102+
$maxsize_json = 50000;
96103

97-
# store XML as base64 encopded gzip in database (save space, harder usable)
104+
# store XML/JSON as base64 encopded gzip in database (save space, harder usable)
98105
$compress_xml = 0;
106+
$compress_json = 0;
99107

100108
# if there was an error during file processing (message does not contain XML or ZIP parts,
101109
# or a database error) the parser reports an error and does not delete the file, even if

0 commit comments

Comments
 (0)