ea_airflow_util v0.4.1

Fixes

• Explicitly pass bucket_name into all method calls to S3Hook.

Full Changelog: v0.4.0...v0.4.1

ea_airflow_util v0.4.0

New features

• Add snowflake_keypair callable for generating Snowflake RSA keypairs and rotating Snowflake user public keys.
• Add SnowflakeKeypairRotationDag for automating Snowflake keypair rotation.

Under the hood

• Add support for Airflow 2.11.
• Implement OpenSSL based key generation for Snowflake keypair authentication.
• Rotate Snowflake user public keys using the slots RSA_PUBLIC_KEY and RSA_PUBLIC_KEY_2 .

Full Changelog: v0.3.8...v0.4.0

ea_airflow_util v0.3.8

Fix release tag

What's Changed

• Feature/sharefile move by @jayckaiser in #74
• Hotfix/formatted paramstore dag mapping by @jayckaiser in #72
• add optional run vars for dbt seed,run,test by @rlittle08 in #73

Full Changelog: v0.3.7...v0.3.8

ea_airflow_util v0.3.7

Fixes

• Fix version update in interface in SlackWebhookHook instantiation in Slack callables (by @tomreitz in PR70).

Full Changelog: v0.3.6...v0.3.7

ea_airflow_util v0.3.6

New Features

• Add recursive flag to sharefile_to_disk() callable (default True). When set to False, only top-level files are copied using an alternative API method (by @jayckaiser in #64).

Under the hood

• Add pagination to SharefileHook._find_items() and downstream-dependent methods.
• Use SharefileHook helper method in sharefile_to_disk() callable to reduce API calls to map filepaths to internal API IDs.

Fixes

• Fix bug where full-refresh DAG config is always set to true in RunDbtDag (by @rlittle08 in #63).
• Fix bug in S3ToSnowflakeOperator where optional arguments being undefined resulted in malformed SQL statements (by @mberrien-fitzsimons in #66).

Full Changelog: v0.3.5...v0.3.6

v0.3.5

New features

• Add optional most_recent_file flag to SharefileToDiskOperator to extract the most recent version of a singleton file from a path.

Under the hood

• Log a warning message when SharefileHook.folder_id_from_path() finds no files for a given path.
• Change logic in callable sql.s3_dir_to_postgres() to raise an AirflowException if any copy of S3 key fails, instead of only when all fail.

Full Changelog: v0.3.4...v0.3.5

v0.3.4

What's Changed

• Feature/update sharefile by @ejoranlienea in #58

Full Changelog: v0.3.3...v0.3.4

ea_airflow_util v0.3.3

Under the hood

• Add dest_filename argument to s3_to_sharefile callable to optionally override filename
• Add LoadSharefileCustomUsersDag to top-level package import path

Fixes

• Run DAG setup method during LoadSharefileCustomUsersDag initialization

Full Changelog: v0.3.2...v0.3.3

ea_airflow_util v0.3.2

New features

• AWSParamStoreToAirflowDAG allows more flexibility when passing Parameter Store paths. Use {tenant_code} when the tenant is in the middle of the path, instead of the end.
• Add s3_to_sharefile and disk_to_sharefile callables
• Add methods to the SharefileHook
• Add ShareFile callable check_for_new_files() to assert expectations in ShareFile directory
• Add S3ToSnowflakeOperator to S3 operators
• Add LoadSharefileCustomUsersDag to automate Heimdall user creation from uploaded authenticated users files in ShareFile

Under the hood

• Code and error-handling improved in callable sharefile_to_disk. Arguments ds_nodash and ts_nodash are deprecated.
• Refactor SFTPToSnowflakeDAG and S3ToSnowflakeDAG to use new S3ToSnowflakeOperator
• Update callable ftp.download_all to accept either a remote directory or file.

Full Changelog: v0.3.1...v0.3.2

ea_airflow_util 0.3.1

New features

• Boolean argument is_manual_upload in S3ToSnowflakeDag rearranges S3 source pathing to easier structure for partners

Under the hood

• Copy statement in S3ToSnowflakeDag uses regex instead of string-splitting to infer pull-date and pull-timestamp

Fixes

• Fix bug in EACustomDAG where default_args were not passed to DAG super init.

Full Changelog: v0.3.0...v0.3.1