This app is like Mint, but better actually exists. It privately collects your personal financial data: no one but you can access it. The data is collected from alert emails sent from your financial institutions. The bank alert emails can be directed to your personal email account, such as a Gmail account. Then setup your account to forward the alert emails to your own self-hosted mailserver (MX). The financial data in the emails is extracted, stored, and made viewable to you and you only.
Thank you @nhopkinson and @whosgonna for their ongoing feedback on this project.
- MSG = message
- TXN = transaction
- UID = unique identifier
- STMT = SQL statement
- MX = mailserver
- md = Maildir
- mb = mbox
- DB = database
This app is mostly production ready! Enjoy!
- Globally install the following software:
Python 3.7 or higher
bash - Run the first setup script. This will create a .env file using the file .env-example as a template.
bash setup/1_setup_sage_directory.sh - Define the following environment variables in the .env file:
DOMAIN: Buy a domain name.
FORWARDING_EMAIL: Set up the email account that receives the transaction alert emails. This account needs to forward all emails to the receiving email address on the MX. The default receiving email address is incoming@DOMAIN. So if you purchased the domain localhost, the receiving email address would me incoming@localhost
RECEIVING_EMAIL_PASSWORD: The password for the MX account that receives transaction emails. The password is used to access the emails programatically (imap).
DO_API_TOKEN: Create a Digital Ocean API Key. It's located in the "API" portion of their menu.
PROD_SSH_PUBLIC_KEY: Create SSH keys for you to SSH to the production server. Ensure the private key permissions are restricted.For help see the section "Production Setup Troubleshooting." Copy/paste the public key here.
PROD_SSH_PRIVATE_KEY_FILE_PATH: Copy/paste the path to the private key file here.
SERVER_USER: Your user the production server.
SERVER_USER_PASSWORD: Your user's password on the production server.
SSH_ALLOWED_PUBLIC_IPS: List the public IPs that can access to the production server.
POSTGRES_PASSWORD: admin role's password.
POSTGRES_ETL_PASSWORD: etl role's password.
POSTGRES_GRAFANA_PASSWORD: grafanareader role's password
POSTGRES_GRAFANA_SSL_MODE: Change this to "enable".
GRAFANA_PASSWORD: Grafana admin user's password. - Define the banks you will be parsing data from in the database
bankstable. - WARNING: RUNNING THIS SCRIPT CAUSES DIGITAL OCEAN TO START CHARGING YOU MONEY ON A MONTHLY BASIS FOR YOUR PRODUCTION SERVER.
Run the script to create a production Digital Ocean Droplet server that runs the application.
bash setup/2_create_prod_server.sh
If an error occurs, go to Digital Ocean and delete the Droplet and firewall before running the script again.
It will prompt you forBECOME password:; enter your sudo password. - Run the script to configure the production Digital Ocean Droplet server.
bash setup/3_configure_prod_server.sh
This script is idempotent: no matter how many times you run it, the result will be the same. So if an error occurs, simply troubleshoot and run the script again until the error is resolved. - WARNING: CREATING DIGITAL OCEAN VOLUMES ALSO COST MONEY, BUT THEY'RE HELPFUL FOR DISASTER RECOVERY. AT THIS TIME, THE VOLUMES ARE NOT OPTIONAL FOR THE DEPLOYMENT. In the Digital Ocean console, go to the "Volumes Block Storage" tab and create two volumes:
sage-dbandsage-mx. - In the Digital Ocean console, attach each volume to the Droplet. SSH to the Droplet and run the commands Digital Ocean provides to mount each volume.
- In the Digital Ocean console, create the NS records for the domain name you purchased, corresponding to
$DOMAINin the.env. - In the Digital Ocean console, create an A record for the Droplet using "prod.< $DOMAIN >" as the hostname. For example, if
$DOMAINis example.com, then the hostname is prod.example.com. Use the floating IP as the value. - In the Digital Ocean console, create a MX record for the Droplet using $DOMAIN as the hostname. For example, if
$DOMAINis example.com, then the hostname is prod.example.com. - Re-run
bash setup/3_configure_prod_server.shto deploy Sage using the mounted volumes.
TODO: Create and mount Digital Ocean volumes during automated production deployment #145 - Go to the Grafana login URL https://prod.< $DOMAIN >. For example, my $DOMAIN is example.com, so the URL is https://prod.example.com. At this URL, you should see "Welcome to Grafana" with a login prompt.
All additional documentation can be found in the docs/ directory.
- Local Development: Provides local development setup instructions, how to create an ephemeral version of Sage for deployment testing (this part costs money), and using Pytest.
- Local Architecture .drawio Diagram
- Prod Architecture .drawio Diagram
- Disaster Recovery: Provides disaster recovery instructions. This is helpful for when something is deeply wrong with your current Sage deployment, but you have data in the MX and/or DB that you don't want to lose. The guide describes how to create a brand new Sage deployment and attach your existing data to it.
- Troubleshooting: Provides troubleshooting advice applicable in both local and prod. If you don't see what you're looking for, it may be in the env specific local troubleshooting or prod troubleshooting files.
- Lists locations of important log files
- Common Help! scenarios
- How to manually test MX operations
- Retrieving emails
- Useful commands
- Digital Ocean Droplet VM
- Ansible
- Docker
- Python
- MX Container (running Postfix and Dovecot)
- Postgres Container
- Local Troubleshooting:
- Environment Variables
- How to manually test MX operations
- Sending emails
- Getting an mbox file from your Gmail account
- Prod Troubleshooting:
- Lists locations of important log files
- Common Help! scenarios
- How to manually test MX operations
- Sending emails
- Getting your Maildir directory from the MX
- Converting Maildir/ to .mbox
- Converting .mbox to Maildir/
- Useful commands
- Server