Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Audit Star [![Build Status](https://travis-ci.org/enova/audit_star.svg?branch=master)](https://travis-ci.org/enova/audit_star)
# Audit Star [![Build Status](https://travis-ci.org/enova/audit_star.svg?branch=master)](https://travis-ci.org/enova/audit_star) [![Documentation](https://img.shields.io/badge/godoc-reference-5272B4.svg?style=flat-square)](https://godoc.org/github.com/enova/audit_star)

The purpose of audit_star is to maintain a history of all of the changes that happen to a database, and to store the history of those changes within the database itself. Additional documentation is provided by clicking the docs badge above.

Expand Down
130 changes: 130 additions & 0 deletions doc.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
/*

Purpose

Audit_Star is a database building block that is intended to provide a standard, easy to use mechanism for capturing row-level data changes in your Postgres databases. It is to provide support for those scenarios where we do not already have another auditing mechanism in place (such as historian functionality or special application level events).

Benefits

The general benefits of auditing are well known. Aside from capturing a historical perspective of the data, it also provides a mechanism to assist in troubleshooting/debugging certain types of problems/issues (particularly those that can be tied to recent data changes). A secondary benefit includes providing a mechanism to communicate to ETL routines or downstream systems that a change (specifically a "delete") has taken place.

Some of the benefits that are specific to using this particular approach include the following:

- There is no maintenance required
- There is no custom code required
- The auditing mechanism never breaks, even when columns are added or removed from a table
- It is easy to apply/deploy
- It can be configured/customized (if needed) via a simple config file

How Does it Work?

This auditing mechanism involves (automatically) generating a number of database components for each table that is to be audited:

- Audit Table - Each "source" table will have a corresponding audit table (to hold the RECENT history of changes)
- Audit Trigger - Each source table will have a DML Trigger that propagates the data changes to its audit table
- Audit View - Each table will have a View that aids in querying/presenting the (JSON) data contained in the audit table

The schema of the audit tables do NOT match/mirror the source table they are auditing. Rather, the audit tables have a single column that holds a JSON representation of the entire "before" row, and a second column that holds a JSON representation of the diff between the "before" and "after" rows. The audit trigger handles creating the JSON representation of the changes, and inserting it into the audit table. The trigger also handle creating new audit table partitions as needed. The audit view converts the JSON back to a row-based representation of the data changes, so that the changes are easier to review. The view also stitches together certain data from the source table (as in the case of initial inserts).

Deployment


HStore Extension

The auditing solution is built on top of the HStore data type in Postgres. Hence, the HStore Extension must be installed in the database.


CREATE EXTENSION hstore with schema public;


Be sure that the extension is bubbled down to staging and development environments. Otherwise the migrations builds/tests will fail.

Run time parameters

The auditing solution uses PostgreSQL runtime parameters to pass metadata such as who made the change from the app to the audit system. The parameters currently used now are `audit_star.changed_by` and `audit_star.changed_reason`. These are defaults that must be set at the database level, usually to an empty string.


ALTER DATABASE <your-db-name> SET audit_star.changed_by TO '';
ALTER DATABASE <your-db-name> SET audit_star.change_reason TO '';


Be sure that the setting is bubbled down to staging and development environments. Otherwise the migrations builds/tests will fail.

Before using audit_star, database-specific configuration must be made to the ```audit.yml```
file. By default, audit_star will look in the current directory from which it is
being executed, but the optional parameter ```-cfg``` allows the user to provide
an alternative path to the ```audit.yml``` file.

The database-specific configuration along with accepted values are detailed in the
example file provided in `audit.yml` (copied below).


database config information
host: database host name
post: database port number; postgres default is 5432
db_name: database_name (name of the db to audit)
username: database username used to connect
password: databsase password used to connect

audit star config information
excluded_tables:
- exclude this_schema.this_table
excluded_schemas:
- exclude_this_schema
owner: app__owner (only audit tables owned by this user, if not specified will audit *every* table it can)
log_client_query: false (toggle logging of query that caused the change)
security: definer/invoker (security level of audit function - usually definer on release to avoid race conditions with defining permissions)


Installation

The preferred way to deploy Go binaries is using `go get` and `go install`.
Optionally, you can clone the repo manually into your $GOPATH and run `go build`.

Since audit_star is written in [Go](https://golang.org/), in order to build audit_star you must have Go
installed on your machine. Alternatively, you can build the binary on one machine
and then copy the binary to a machine, as needed, before invoking it, assuming the
machines share the same architecture. Go also supports [cross-compilation](https://dave.cheney.net/2012/09/08/an-introduction-to-cross-compilation-with-go)
which allows you to build the binary on a machine with one architecture and run
it on a machine with a completely different architecture.

Using `go get` (preferred)
You can download audit_star by running `go get github.com/enova/audit_star`.
This will download audit_star into your $GOPATH and install the binary into your $GOPATH/bin folder.
If your $GOPATH/bin folder is in your $PATH then you can run audit_star from anywhere by executing `audit_star`.

Building with ```go build```
After cloning the git repo, ```cd``` into the directory and run ```go build```. Assuming your Go is set up properly, this will create an ```audit_star``` binary in the current directly. You can then execute this binary by executing ```./audit_star```.

Testing

Audit Star tests use seprate configuration and db migration

Usage

Install pgmgr

go get github.com/rnubel/pgmgr

Resetting DB and Run migration

pgmgr db drop || true
pgmgr db create
pgmgr db migrate


Run tests

go test ./...

Note

Since the purpose of audit_star is to audit a database, in order to test this
functionality, the provided tests create a test database by running the migration
contained within the ```db/migrate``` folder. The migration was created and is
run using the [pgmgr Go migration tool](https://github.com/rnubel/pgmgr). So while
pgmgr is a dependency for the tests, it is not of dependency of audit_star itself.
*/
package main

/**/