Improve Humio imports.

Author: keithjjones

CHANGES

@@ -1,3 +1,4 @@
+v0.3.15         Improved Humio import.
 v0.3.14         Removed a print statement.
 v0.3.13         Fixed some errors on Humio import.
 v0.3.12         Will continue to populate data after a Humio error.

Readme.md

@@ -17,6 +17,7 @@ logs into [ElasticSearch's bulk load JSON format](https://www.elastic.co/guide/e
 - [Command Line Options](#commandlineoptions)
 - [Requirements](#requirements)
 - [Notes](#notes)
+  - [Humio](#humio)
   - [JSON Log Input](#jsonloginput)
   - [Data Streams](#datastreams)
   - [Helper Scripts](#helperscripts)
@@ -318,6 +319,18 @@ You will need to add -k -u elastic_user:password if you are using Elastic v8+.
 
 ## Notes <a name="notes" />
 
+### Humio <a name="humio" />
+
+To import your data into Humio you will need to set up a repository with the `corelight-json` parser.  Obtain
+the ingest token for the repository and you can import your data with a command such as:
+
+```
+python3 zeek2es.py -s -b --humio http://localhost:8080 b005bf74-1ed3-4871-904f-9460a4687202 http.log 
+```
+
+The URL should be in the format of: `http://yourserver:8080`, as the rest of the path is added by the
+`zeek2es.py` script automatically for you.
+
 ### JSON Log Input <a name="jsonloginput" />
 
 Since Zeek JSON logs do not have type information like the ASCII TSV versions, only limited type information 

zeek2es.py

@@ -445,7 +445,14 @@ def main(**args):
                         # Prepare the output and increment counters
                         if args['humio']:
                             d['ts'] = d['ts'] + "Z"
-                            d["_write_ts"] = d["ts"]
+                            if "_write_ts" in d:
+                                d['_write_ts'] = d['_write_ts'] + "Z"
+                            else:
+                                d["_write_ts"] = d["ts"]
+                            if "_path" not in d:
+                                d["_path"] = zeek_log_path
+                            if (len(args['name'].strip()) > 0):
+                                d["_system_name"] = args['name'].strip()
                         d["@timestamp"] = d["ts"]
                         outstring += json.dumps(d)+"\n"
                         n += 1