Oracle

This repo automates the creation of D3js visualizations based on Dungeon Church lore wiki data.

The result is deployed on GH pages.

Repository Structure

• data/ - Contains the graph data JSON file
  • graph_data.json - Main graph data with connected nodes
    • Contains nodes (articles) and links (relationships between articles)
    • Each link includes metadata about the relationship:
      • creation_time - When the relationship was created
      • direction - Whether the relationship is unidirectional or bidirectional
  • orphan_data.json - Orphaned nodes (nodes with 0 connections)
• scripts/ - Python scripts for data processing and visualization generation
• static/ - Static templates for the visualization (HTML, CSS, JS)
• docs/ - Generated files for GitHub Pages deployment (created by scripts, not stored in the repository)

Setup

Outline database nightly backups are stored in a private S3 bucket:

DUNGEONCHURCH_S3_URL=<pre-authenticated request URL for OCI Object Storage>
DUNGEONCHURCH_S3_NAMESPACE=<OCI namespace>
DUNGEONCHURCH_S3_BUCKET=<OCI bucket name>

The 5E rules collection in the wiki can be included or excluded in the visualization with:

EXCLUDE_5E=true

For future Outline integration:

OUTLINE_API_TOKEN=xxx
OUTLINE_URL=https://lore.dungeon.church/api

Process

Download the Latest Database Dump

python scripts/download_latest_dump.py

This will download the latest PostgreSQL dump file from OCI S3 compatible storage and save it to the data directory.

Process Relationship Data

python scripts/process_relationships.py <path_to_dump_file>

1. Restore the PostgreSQL dump file to a temporary database
2. Extract document relationship data
3. Identify and remove orphaned nodes (nodes with 0 connections)
4. Save the main graph data as graph_data.json in the data directory
5. Save the orphaned nodes as orphan_data.json in the data directory
6. Drop the temporary database

Create Visualization

python scripts/create_viz.py

1. Generate the visualization files in the docs folder using templates from the static folder
2. The docs folder is not stored in the repository but is generated on-demand

Run the Complete Pipeline

python scripts/run_pipeline.py

1. Download the latest database dump
2. Process the dump to extract relationship data
3. Clean up the dump file (unless --keep-dump is specified)

Options:

• --output or -o: Specify the output JSON file path
• --keep-dump or -k: Keep the dump file after processing

Customizing the Visualization

The visualization can be customized by editing the files in the static folder:

• static/index.html - The main HTML template
• static/css/styles.css - The CSS styles for the visualization
• static/js/visualization.js - The JavaScript code for the D3.js visualization

After making changes to these files, you can:

1. Run python scripts/create_viz.py to generate the updated visualization in the docs folder
2. Open docs/index.html in your browser to preview the changes
3. Commit and push your changes to deploy them via GitHub Actions

Automated Updates

This repository includes GitHub Actions workflows that:

1. Run the data pipeline daily at midnight UTC (and can be triggered manually)
2. Rebuild and deploy the visualization when changes are made to:
   • The graph data (data/graph_data.json)
   • The static templates (static/*)
   • The visualization script (scripts/create_viz.py)