Merge pull request #60 from fivetran/feature/add-tracking-category

Add Tracking Categories

Author: fivetran-avinash

.buildkite/scripts/run_models.sh

@@ -19,7 +19,7 @@ dbt deps
 dbt seed --target "$db" --full-refresh
 dbt run --target "$db" --full-refresh
 dbt test --target "$db"
-dbt run --vars '{xero__using_credit_note: false, xero__using_bank_transaction: false}' --target "$db" --full-refresh
+dbt run --vars '{xero__using_credit_note: false, xero__using_bank_transaction: false, xero__using_invoice_line_item_tracking_category: false, xero__using_journal_line_tracking_category: false, xero__using_tracking_categories: false}' --target "$db" --full-refresh
 dbt test --target "$db"
 dbt run --target "$db" --full-refresh
 dbt test --target "$db"

.quickstart/quickstart.yml

@@ -8,6 +8,14 @@ table_variables:
     - _credit_note
   xero__using_bank_transaction:
     - _bank_transaction
+  xero__using_invoice_line_item_tracking_category:
+    - _invoice_line_item_has_tracking_category
+  xero__using_journal_line_tracking_category:
+    - _journal_line_has_tracking_category
+  xero__using_tracking_categories:
+    - _tracking_category
+    - _tracking_category_option
+    - _tracking_category_has_option
 
 destination_configurations:
   databricks:

CHANGELOG.md

@@ -1,8 +1,35 @@
-# dbt_xero v0.version.version
-This release includes the following updates.
+# dbt_xero v0.9.0 
+This release includes the following updates:
+
+## Breaking Changes
+- Added tracking categories by pivoting out fields and grabbing the latest tracking category records from invoice line items and journal line entries. This update will provide more granular reporting capabilities and will result in the following model changes: ([PR #60](https://github.com/fivetran/dbt_xero/pull/60))
+  - `xero__general_ledger` dynamically pivots out journal line tracking category names and populates each line with the active options for the categories for these journals and accounts. 
+  - `xero__profit_and_loss_report` **will see a change in the grain** with these new dynamic pivoted tracking categories added to each journal line. 
+    - Added these dynamic tracking categories to the `profit_and_loss_id` as they are now unique lines within the profit and loss. **IMPORTANT**: This will change the existing values of the `profit_and_loss_id`, so this is a **breaking change**. ([PR #60](https://github.com/fivetran/dbt_xero/pull/60))
+  - `xero__invoice_line_items` dynamically pivots out invoice line item tracking category and populates each line with the active options for those line items. 
+- Created intermediate models `int_xero__invoice_line_item_tracking_categories` and `int_xero__journal_line_tracking_categories` to perform the `dbt_utils` pivot operations. ([PR #60](https://github.com/fivetran/dbt_xero/pull/60))
+- Added the following variables to allow the ability to disable your models if you're not planning to utilize the tracking category configuration. [See the README](https://github.com/fivetran/dbt_xero/blob/main/README.md#disabling-and-enabling-models) for detailed instructions. ([PR #60](https://github.com/fivetran/dbt_xero/pull/60))
+  - `xero__using_invoice_line_item_tracking_category`
+  - `xero__using_journal_line_tracking_category`
+  - `xero__using_tracking_categories`
+- These variables are utilized in the following end models:
+
+| End Models Impacted      |     Variables Implemented  |
+|-------------------------------------------------------| ---------------------------------------------------|
+| `xero__general_ledger`, `xero__profit_and_loss_report`   | `xero__using_journal_line_tracking_category`, `xero__using_tracking_categories` |
+| `xero__invoice_line_items`     | `xero__using_invoice_line_item_tracking_category`, `xero__using_tracking_categories`      |
+
+## Under the Hood
+- Added new table variables in `quickstart.yml` to ensure their respective models are enabled and disabled appropriately. 
+([PR #60](https://github.com/fivetran/dbt_xero/pull/60))
+- Created `get_prefixed_tracking_category_columns` macro to properly bring in and check for pivoted columns within the above `dbt_xero` end models.
+- Created integrity tests to ensure tracking category values for journal lines and invoice items match between the above end models and their source tables. 
+([PR #60](https://github.com/fivetran/dbt_xero/pull/60))
+- Added and updated seed files to properly test out advanced cases for the new tracking categories. ([PR #60](https://github.com/fivetran/dbt_xero/pull/60))
 
 ## Documentation
-- Updated README to better elaborate on why multi-currency can't be fully supported by our dbt package. 
+- Added instructions in the README for how to disable tracking category functionality utilizing the new variables. ([PR #60](https://github.com/fivetran/dbt_xero/pull/60))
+- Updated README to better elaborate on why multi-currency can't be fully supported by our dbt package. ([PR #61](https://github.com/fivetran/dbt_xero/pull/61))
 
 # dbt_xero v0.8.0
 [PR #54](https://github.com/fivetran/dbt_xero/pull/54) includes the following updates:

README.md

@@ -33,7 +33,7 @@ The following table provides a detailed list of all tables materialized within t
 | [xero__invoice_line_items](https://github.com/fivetran/dbt_xero/blob/main/models/xero__invoice_line_items.sql)      | Each record represents an invoice line item enriched with the account, contact, and invoice information.                   |
 
 ### Materialized Models
-Each Quickstart transformation job run materializes 22 models if all components of this data model are enabled. This count includes all staging, intermediate, and final models materialized as `view`, `table`, or `incremental`.
+Each Quickstart transformation job run materializes 35 models if all components of this data model are enabled. This count includes all staging, intermediate, and final models materialized as `view`, `table`, or `incremental`.
 <!--section-end-->
 
 ## How do I use the dbt package?
@@ -50,7 +50,7 @@ Include the following xero package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/xero
-    version: [">=0.8.0", "<0.9.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.9.0", "<0.10.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 Do NOT include the `xero_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well.
 ### Step 3: Define database and schema variables
@@ -96,6 +96,9 @@ config-version: 2
 vars:
     xero__using_credit_note: false                      # default is true
     xero__using_bank_transaction: false                 # default is true
+    xero__using_invoice_line_item_tracking_category: false  # default is true
+    xero__using_journal_line_tracking_category: false # default is true
+    xero__using_tracking_categories: false                # default is true
 ```
 
 For additional configurations for the source models, visit the [Xero source package](https://github.com/fivetran/dbt_xero_source).
@@ -139,8 +142,8 @@ This dbt package is dependent on the following dbt packages. These dependencies
 
 ```yml
 packages:
-    - package: fivetran/dbt_xero_source
-      version: [">=0.6.0", "<0.7.0"]
+    - package: fivetran/xero_source
+      version: [">=0.7.0", "<0.8.0"]
 
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'xero'
-version: '0.8.0'
+version: '0.9.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 vars:
@@ -13,6 +13,11 @@ vars:
     organization: "{{ ref('stg_xero__organization') }}"
     credit_note: "{{ ref('stg_xero__credit_note') }}"
     bank_transaction: "{{ ref('stg_xero__bank_transaction') }}"
+    invoice_line_item_has_tracking_category: "{{ ref('stg_xero__invoice_line_item_has_tracking_category') }}"
+    journal_line_has_tracking_category: "{{ ref('stg_xero__journal_line_has_tracking_category') }}"
+    tracking_category_has_option: "{{ ref('stg_xero__tracking_category_has_option') }}"
+    tracking_category_option: "{{ ref('stg_xero__tracking_category_option') }}"
+    tracking_category: "{{ ref('stg_xero__tracking_category') }}"
     xero__using_bank_transaction: true
     xero__using_credit_note: true
 models:

docs/catalog.json

@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: xero_integration_tests_2
+      schema: xero_integration_tests_7
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: xero_integration_tests_2
+      schema: xero_integration_tests_7
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: xero_integration_tests_2
+      schema: xero_integration_tests_7
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: xero_integration_tests_2
+      schema: xero_integration_tests_7
       threads: 8
     databricks:
       catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: xero_integration_tests_2
+      schema: xero_integration_tests_7
       threads: 8
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

docs/manifest.json

@@ -1,5 +1,5 @@
 name: 'xero_integration_tests'
-version: '0.8.0'
+version: '0.9.0'
 profile: 'integration_tests'
 config-version: 2
 
@@ -14,7 +14,12 @@ vars:
     xero_organization_identifier: "xero_organization_data"
     xero_bank_transaction_identifier: "xero_bank_transaction_data"
     xero_credit_note_identifier: "xero_credit_note_data"
-    xero_schema: xero_integration_tests_2
+    xero_invoice_line_item_has_tracking_category_identifier: "xero_invoice_line_item_has_tracking_category_data"
+    xero_journal_line_has_tracking_category_identifier: "xero_journal_line_has_tracking_category_data"
+    xero_tracking_category_identifier: "xero_tracking_category_data"
+    xero_tracking_category_has_option_identifier: "xero_tracking_category_has_option_data"
+    xero_tracking_category_option_identifier: "xero_tracking_category_option_data"    
+  xero_schema: xero_integration_tests_7
 
 models:
   +schema: "xero_{{ var('directed_schema','dev') }}"

integration_tests/ci/sample.profiles.yml

@@ -0,0 +1,9 @@
+invoice_id,line_item_id,tracking_category_id,option,_fivetran_synced
+533adbeb-5ebb-43c1-9078-7db8c48a057c,52ac1d34-e292-41e9-8760-518f166c6f2d,ab27f5a3-e4d6-49a2-b7c4-812f7e931a50,Client Project A,2021-02-04 18:00:17.876 UTC
+533adbeb-5ebb-43c1-9078-7db8c48a057c,52ac1d34-e292-41e9-8760-518f166c6f2d,f3129a78-67cf-4d30-a501-20b6d98c837e,Engineering Cost,2021-02-04 18:00:17.876 UTC
+533adbeb-5ebb-43c1-9078-7db8c48a057c,52ac1d34-e292-41e9-8760-518f166c6f2d,20fd4c53-9ae2-4229-8d9e-9d7cb6421f14,North America,2021-02-04 18:00:17.876 UTC
+b2e2a55c-2f15-472a-8dbc-2012ad19a793,81831b9f-daa9-4808-9b66-a41c3d6a2961,8e91ad4b-6d04-4b77-918a-5127b24f35ec,Engineering,2022-02-04 18:00:17.876 UTC
+b2e2a55c-2f15-472a-8dbc-2012ad19a793,81831b9f-daa9-4808-9b66-a41c3d6a2961,5c8e34b2-ff1a-4e7b-9023-654db41ce9d2,New York,2022-02-04 18:00:17.876 UTC
+b2e2a55c-2f15-472a-8dbc-2012ad19a793,81831b9f-daa9-4808-9b66-a41c3d6a2961,20fd4c53-9ae2-4229-8d9e-9d7cb6421f14,Europe,2022-02-04 18:00:17.876 UTC
+231940bd-bcdd-460f-83aa-05da8383e967,eb9d72ea-00e3-4b65-8e30-8e98a7177053,ab27f5a3-e4d6-49a2-b7c4-812f7e931a50,Internal Project B,2023-02-04 18:00:17.876 UTC
+231940bd-bcdd-460f-83aa-05da8383e967,eb9d72ea-00e3-4b65-8e30-8e98a7177053,8e91ad4b-6d04-4b77-918a-5127b24f35ec,Sales,2023-02-04 18:00:17.876 UTC