Skip to content

Commit fb55109

Browse files
committed
NRL-1320 add read me and log schema generation code for visibility
1 parent a83e6b6 commit fb55109

File tree

2 files changed

+362
-0
lines changed

2 files changed

+362
-0
lines changed

terraform/account-wide-infrastructure/modules/glue/LogSchemaGeneration/LogSchemaGeneration.ipynb

Lines changed: 357 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Log Schema Generation
2+
3+
The Glue script uses pyspark to process log data. Due to the structure of each json document inside of a log group differing, we need to account for this variance.
4+
5+
The notebook provides a way to automatically generate a pyspark schema for a log group without manual intervention. Point it at the desired group, and hit run all, then copy and paste the output into either producer_schema.py or consumer_schema.py.

0 commit comments

Comments
 (0)