Merge pull request #122 from fivetran/release/v0.14.0

Release/v0.14.0

Author: fivetran-catfritz

CHANGELOG.md

@@ -1,3 +1,26 @@
+# dbt_hubspot_source v0.14.0
+[PR #122](https://github.com/fivetran/dbt_hubspot_source/pull/122) includes the following updates:
+
+## Features
+- Added the following staging models, along with documentation and tests:
+  - `stg_hubspot__property`
+  - `stg_hubspot__property_option`
+  - These tables can be disabled by setting `hubspot_property_enabled: False` in your dbt_project.yml vars. See [Step 4 of the README](https://github.com/fivetran/dbt_hubspot_source#step-4-disable-models-for-non-existent-sources) for more details. 
+
+- When including a passthrough `property_hs_*` column, you now have the option to include the corresponding, human-readable label in the staging models. 
+  - The above-mentioned `property` tables are required for this feature. If you do not have them and have to disable them, unfortunately you will not be able to use this feature.
+  - See the [Adding property label section](https://github.com/fivetran/dbt_hubspot_source#adding-property-label) of the README for instructions on how to enable this feature! 
+  - We recommend being selective with the label columns you add. As you add more label columns, your run time will increase due to the underlying logic requirements.
+  - This update applies to models:
+    - `stg_hubspot__company`
+    - `stg_hubspot__contact`
+    - `stg_hubspot__deal`
+    - `stg_hubspot__ticket`
+
+## Bug fixes
+- Updated macro `remove_duplicate_and_prefix_from_columns` to accommodate incoming custom column names containing characters such as `-` or `$` that are not permitted. The resulting column name will have these characters removed or replaced in its `stg_*` model. 
+- Removed extra comma from `stg_hubspot__ticket`, which was causing compilation issues when passing through all columns. 
+
 # dbt_hubspot_source v0.13.0
 
 ## 🚨 Breaking Changes 🚨

README.md

@@ -44,7 +44,7 @@ Include the following hubspot_source package version in your `packages.yml` file
 ```yaml
 packages:
   - package: fivetran/hubspot_source
-    version: [">=0.13.0", "<0.14.0"]
+    version: [">=0.14.0", "<0.15.0"]
 ```
 ## Step 3: Define database and schema variables
 By default, this package runs using your destination and the `hubspot` schema. If this is not where your HubSpot data is (for example, if your HubSpot schema is named `hubspot_fivetran`), add the following configuration to your root `dbt_project.yml` file:
@@ -104,6 +104,7 @@ vars:
   hubspot_engagement_note_enabled: false
   hubspot_engagement_task_enabled: false
   hubspot_owner_enabled: false
+  hubspot_property_enabled: false                         # Disables property and property_option tables
 
   # Service
   hubspot_service_enabled: true                           # Enables all service models
@@ -135,7 +136,7 @@ vars:
   hubspot__ticket_pass_through_columns:
     - name:           "property_mmm"
       alias:          "mmm"
-    - name:           "property_bop"
+    - name:           "property_hs_bop"
       alias:          "bop"
 ```
 **Alternatively**, if you would like to simply pass through **all columns** in the above four tables, add the following configuration to your dbt_project.yml. Note that this will override any `hubspot__[table_name]_pass_through_columns` variables.
@@ -144,6 +145,34 @@ vars:
 vars:
   hubspot__pass_through_all_columns: true # default is false
 ```
+
+### Adding property label
+For `property_hs_*` columns, you can enable the corresponding, human-readable `property_option`.`label` to be included in the staging models. 
+
+#### Important! 
+- You must have sources `property` and `property_option` enabled to enable labels. By default, these sources are enabled.  
+- You CANNOT enable labels if using `hubspot__pass_through_all_columns: true`.` 
+- We recommend being selective with the label columns you add. As you add more label columns, your run time will increase due to the underlying logic requirements.
+
+To enable labels for a given property, set the property attribute `add_property_label: true`, using the below format.
+
+```yml
+vars:
+  hubspot__ticket_pass_through_columns:
+    - name: "property_hs_fieldname"
+      alias: "fieldname"
+      add_property_label: true
+```
+
+Alternatively, you can enable labels for all passthrough properties by using variable `hubspot__enable_all_property_labels: true`, formatted like the below example. 
+
+```yml
+vars:
+  hubspot__enable_all_property_labels: true # cannot use in conjunction with `hubspot__pass_through_all_columns: true`
+  hubspot__ticket_pass_through_columns:
+    - name: "property_hs_fieldname1"
+    - name: "property_hs_fieldname2"
+```
 ### Including calculated fields
 This package also provides the ability to pass calculated fields through to the `company`, `contact`, `deal`, and `ticket` staging models. If you would like to add a calculated field to any of the mentioned staging models, you may configure the respective `hubspot__[table_name]_calculated_fields` variables with the `name` of the field you would like to create, and the `transform_sql` which will be the actual calculation that will make up the calculated field.
 ```yml

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'hubspot_source'
-version: '0.13.0'
+version: '0.14.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:
@@ -47,6 +47,8 @@ vars:
     engagement_note: "{{ source('hubspot','engagement_note') }}"
     engagement_task: "{{ source('hubspot','engagement_task') }}"
     owner: "{{ source('hubspot','owner') }}"
+    property: "{{ source('hubspot','property') }}"
+    property_option: "{{ source('hubspot','property_option') }}"
     ticket_company: "{{ source('hubspot','ticket_company') }}"
     ticket_contact: "{{ source('hubspot','ticket_contact') }}"
     ticket_deal: "{{ source('hubspot','ticket_deal') }}"

docs/catalog.json

@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: hubspot_source_integration_tests_7
+      schema: hubspot_source_integration_tests_999
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: hubspot_source_integration_tests_7
+      schema: hubspot_source_integration_tests_999
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: hubspot_source_integration_tests_7
+      schema: hubspot_source_integration_tests_999
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: hubspot_source_integration_tests_7
+      schema: hubspot_source_integration_tests_999
       threads: 8
     databricks:
       catalog: null
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: hubspot_source_integration_tests_7
+      schema: hubspot_source_integration_tests_999
       threads: 8
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

docs/manifest.json

@@ -1,14 +1,18 @@
 name: 'hubspot_source_integration_tests'
-version: '0.13.0'
+version: '0.14.0'
 profile: 'integration_tests'
 config-version: 2
 models:
   hubspot_source:
     +schema:
 vars:
-  hubspot_schema: hubspot_source_integration_tests_7
+  hubspot_schema: hubspot_source_integration_tests_999
   hubspot_source:
     hubspot_service_enabled: true
+    # hubspot_sales_enabled: true # enable when generating docs
+    # hubspot_marketing_enabled: true # enable when generating docs
+    # hubspot_contact_merge_audit_enabled: true # enable when generating docs
+    # hubspot_using_all_email_events: true # enable when generating docs
     hubspot_company_property_history_identifier: "company_property_history_data"
     hubspot_company_identifier: "company_data"
     hubspot_contact_identifier: "contact_data"
@@ -43,6 +47,8 @@ vars:
     hubspot_engagement_note_identifier: "engagement_note_data"
     hubspot_engagement_task_identifier: "engagement_task_data"
     hubspot_owner_identifier: "owner_data"
+    hubspot_property_identifier: "property_data"
+    hubspot_property_option_identifier: "property_option_data"
     hubspot_ticket_company_identifier: "ticket_company_data"
     hubspot_ticket_contact_identifier: "ticket_contact_data"
     hubspot_ticket_deal_identifier: "ticket_deal_data"

docs/run_results.json

@@ -4,5 +4,4 @@ dbt-redshift>=1.3.0,<2.0.0
 dbt-postgres>=1.3.0,<2.0.0
 dbt-spark>=1.3.0,<2.0.0
 dbt-spark[PyHive]>=1.3.0,<2.0.0
-dbt-databricks>=1.3.0,<2.0.0
-oscrypto @ git+https://github.com/wbond/oscrypto.git@d5f3437
+dbt-databricks>=1.3.0,<2.0.0

integration_tests/ci/sample.profiles.yml

@@ -0,0 +1,11 @@
+_fivetran_id,_fivetran_synced,calculated,created_at,description,field_type,group_name,hubspot_defined,hubspot_object,label,name,show_currency_symbol,type,updated_at
++mlGrLvmij8zluyeRByh7DPbXpM=,2023-11-11 08:18:16.519000 UTC,false,2023-05-03 08:51:08.199000 UTC,M0xKTEL9t51+vD5ztRfm+A==,date,fVQYOcf70/KsHU/KFoXQfQ==,,contact,test_pro,test_pro,false,date,2023-05-03 08:51:08.199000 UTC
+6fnvV+pkZR3di3USRclW2NZMNTA=,2023-11-11 08:18:15.151000 UTC,false,2019-08-06 02:41:52.364000 UTC,k+VMRhjE/W4QB470Jor1yA==,date,j8mgV9/MYzXvxQKD3c0AHw==,true,deal,Close Date,closedate,,datetime,2023-06-20 22:32:28.368000 UTC
+5yTEFwFlLPTKlzXc3PviX5DMdJ4=,2023-11-11 08:18:16.569000 UTC,false,2019-08-06 02:41:09.071000 UTC,nRGOJjIZrs4d0WJLkC15fg==,date,fVQYOcf70/KsHU/KFoXQfQ==,true,contact,Close Date,closedate,,datetime,2022-05-27 20:35:26.640000 UTC
+u0bk1tHdJJir2SmRkwo1r22t2PM=,2023-11-11 08:18:15.837000 UTC,false,2019-08-06 02:41:44.665000 UTC,HRIf2XFKVmDQxMzI2w3mcA==,date,XArNQdwSO+DY7119vj+JEw==,true,company,Close Date,closedate,,datetime,2022-05-27 22:49:16.893000 UTC
+RTJ5KTwQe/zMYl8uKwNFlOeNWUY=,2023-11-11 08:18:15.792000 UTC,false,2022-11-30 10:52:48.277000 UTC,1B2M2Y8AsgTpgAmY7PhCfg==,date,XArNQdwSO+DY7119vj+JEw==,,company,test_date,test_date,false,date,2022-11-30 10:52:48.277000 UTC
+sv5Aa2A1+wJwoKRX4Zfe3CTEHEE=,2023-11-11 08:18:16.575000 UTC,false,2019-08-06 02:41:09.148000 UTC,jjSyQlM9DMhpVol12tp4fg==,date,fVQYOcf70/KsHU/KFoXQfQ==,true,contact,Create Date,createdate,,datetime,2023-05-01 20:02:33.973000 UTC
+GjTSotYsbt+HoRY5e6bHRgNJYFI=,2023-11-11 08:18:36.368000 UTC,false,2019-08-06 02:42:48.261000 UTC,nBljQWslMNrEDwXeYmRYyg==,date,0NoL94xOqMRlkFF+GAX6QQ==,true,product,Create Date,createdate,,datetime,2022-05-28 01:26:21.260000 UTC
+6iFNLSeY0Fz5DgwskGjrMoW3JIg=,2023-11-11 08:18:13.861000 UTC,false,2019-08-06 02:42:38.781000 UTC,lMRanmvFHjNC0pQdw79H5Q==,date,kdJdJeqzIqBPlKTSRZoyGQ==,true,line_item,Create Date,createdate,,datetime,2022-05-28 01:31:47.067000 UTC
+onbRYvHwxXS3NwSP2zaoD+Mv/us=,2023-11-11 08:18:36.567000 UTC,false,2019-08-06 02:43:00.503000 UTC,MD3INB3ab4k2LqaTom0eEA==,date,CfZkcOOMcc/dg5a9ofwkUg==,true,ticket,Create date,createdate,,datetime,2022-08-18 12:55:35.731000 UTC
+XhKQ/eq5pqJEB6smEKCPSSyfUH4=,2023-11-11 08:18:15.836000 UTC,false,2019-08-06 02:41:44.631000 UTC,wyvwVnWdqsdNRapzG8Y/RA==,date,XArNQdwSO+DY7119vj+JEw==,true,company,Create Date,createdate,,datetime,2023-05-01 19:33:57.873000 UTC