Request to Add Incremental Functionality to bigquery_analytics Package

[jessicaoblira]

Description:

Currently, the bigquery_analytics package used in the Addiante project is experiencing performance issues due to the growing size of the metadata table, which is information_schema_jobs. Since its addition to the project on 04/16/2024, the table has accumulated a total of 1,286,869 rows, resulting in a table size of 25.87 GB.

To improve efficiency and performance, I would like to request the inclusion of incremental functionality in the bigquery_analytics package model. Implementing an incremental approach will allow only new or changed data since the last run to be processed, reducing execution time and system load.

Justification:

Performance: The information_schema table is becoming very large, making full processing increasingly slow.
Efficiency: An incremental approach will enable the package to process only recent changes, optimizing resource use and improving response time.
Scalability: Incremental tables will help ensure that the package can handle the continuous data growth without compromising performance.

Additional Information:

Package: bigquery_analytics
Metadata Table: information_schema_jobs
Min data for the Addiante project: 04/16/2024
Number of Rows Accumulated: 1,286,869
Table Size: 25.87 GB

Please let me know if you need more information or if there are any additional details to discuss.

Thank you!