Feature: custom outputs#315

Open

doubleswirve wants to merge 82 commits intoopencivicdata:masterfrom

GovHawkDC:feature/custom-outputs

Commits on Dec 4, 2017

Commits on Dec 5, 2017

add google-cloud-pubsub install requirement
Saxon Stiller
committed
try setting up conditional import of module based on env var
Saxon Stiller
committed
tiny var name change since not really dealing w/ class
Saxon Stiller
committed
try to set up pub/sub for env var auth (since it will be difficult to pass json key file into build)
Saxon Stiller
committed
small style changes to abide by flake8
Saxon Stiller
committed
tiny styling change
Saxon Stiller
committed
add in obj pre_save method to match file writing save
Saxon Stiller
committed
change env var for output switch and remove from settings
Saxon Stiller
committed
bunch o name changes
Saxon Stiller
committed
remove unused import_lib
Saxon Stiller
committed
remove extra whitespace (causing diff change)
Saxon Stiller
committed
ditto
Saxon Stiller
committed
remove all just because hehe
Saxon Stiller
committed
single quote consistency
Saxon Stiller
committed

Commits on Dec 6, 2017

add debuggering line
Saxon Stiller
committed
more debug printing
Saxon Stiller
committed
remove debugging
Saxon Stiller
committed
(try) add scopes and tiny flake8 patch
Saxon Stiller
committed
try another way for scopes
Saxon Stiller
committed
oops, forgot to specify keyword for arg
Saxon Stiller
committed
try a tuple instead?
Saxon Stiller
committed
meh, trying this, maybe i missed something
Saxon Stiller
committed
try another way
Saxon Stiller
committed
patch validation to reference caller
Saxon Stiller
committed
one last shot, for this nite my friend
Saxon Stiller
committed
try again
Saxon Stiller
committed
trying json file loaded into single env var
Saxon Stiller
committed
debug step
Saxon Stiller
committed
undo debug step
Saxon Stiller
committed
add reference to issue on github that helped me so dearly
Saxon Stiller
committed
remove kafka for now to make the diff/pr focused on googel cloude
Saxon Stiller
committed
small version change
Saxon Stiller
committed

Commits on Dec 7, 2017

match env var used in google cloud documentation for the project id
Saxon Stiller
committed
remove project_id var and just use env var during topic setup
Saxon Stiller
committed
ditto, remove topic var and just use env var during topic setup
Saxon Stiller
committed
refactor google auth to allow for automatic detection of credentials, in case user is hosting on gcp
Saxon Stiller
committed

Commits on Dec 13, 2017

add bill_identifier prop to vote_event for a more reliable way to reference bill relationship
Saxon Stiller
committed
nevermind re: new field; fallback to env var switch option
Saxon Stiller
committed

Commits on Dec 14, 2017

remove some whitespace from stringified json sent to pub/sub and explicitly encode to utf-8
Saxon Stiller
committed
use pub/sub topic_path publisher helper method to obtain topic path without hardcoding the topic string format
Saxon Stiller
committed
remove automatic topic creation; assume user will create the topic prior to using to avoid the extra request
Saxon Stiller
committed
do not require ordering before publishing to pubsub
Saxon Stiller
committed

Commits on Dec 23, 2017

update google pubsub dependency version
doubleswirve
committed

Commits on Dec 27, 2017

tiny update to service account env var name
doubleswirve
committed

Commits on Jan 17, 2018

merge upstream and settle conflicts
doubleswirve
committed

Commits on Jan 18, 2018

Commits on Jan 19, 2018

remove local file output init method
doubleswirve
committed
Merge pull request #1 from doubleswirve/custom-export--dry-it-up
Saxon Stiller
authored
patch scraper reference prop in google cloud output
doubleswirve
committed

Commits on Jan 21, 2018

add amazon sqs output
doubleswirve
committed
add amazon sqs output target to condition
doubleswirve
committed
add boto3 dep
doubleswirve
committed
do not encode for aws sqs
doubleswirve
committed
update pubsub filename
doubleswirve
committed
update initialization of publisher with new env var
doubleswirve
committed

Commits on Jan 22, 2018

Commits on Jan 23, 2018

try upgrading the grpc lib version
doubleswirve
committed
remove grpc dep, no dice
doubleswirve
committed
Merge pull request #2 from doubleswirve/custom-export--google-pubsub-env
Saxon Stiller
authored
Merge branch 'master' into custom-export
doubleswirve
committed
Merge branch 'custom-export' into custom-export--amazon-sqs
doubleswirve
committed
update info logging
doubleswirve
committed
port over some more helper to sqs
doubleswirve
committed
add s3 handling for messages that are too large
doubleswirve
committed
update s3 key format
doubleswirve
committed

Commits on Jan 24, 2018

update number formatting for py version
doubleswirve
committed

Commits on Mar 27, 2018

Merge pull request #3 from doubleswirve/custom-export--amazon-sqs
Saxon Stiller
authored

Commits on Aug 15, 2018

resolve conflicts
doubleswirve
committed

Commits on Dec 11, 2018

Update opencivicdata min version
showerst
committed