Adding a partitioned reporting_patient_states table (#5614)

**Story card:**
[sc-15583](https://app.shortcut.com/simpledotorg/story/15583/new-reporting-pipeline-for-reporting-patient-states)

## Because

Materialized view refresh is very slow.
We want to introduce partitioned table approach, where with each month
it will drop that partition, regenerate data for that partition and
attach it to the main table.
With this new approach we will not be doing a full refresh of the table
every day.

## This addresses

This is the first step towards redesigning our reporting pipeline.
We are adding a partitioned reporting_patient_states table, under a
different schema. The Mat view version of same would continue to behave
as it is, until we decide to finally make the partitioned table live.
Dashboards are still going to use the mat view for getting the data. 

When the daily rake task for runs, then for reporting_patient_states
will refresh only the current month and previous month.
We still want to refresh last 15 months data to handle delayed sync
scenario. So last 15 months data would be refreshed once at some point
during the current month.

There is a rake task for getting this table fully refreshed. This would
delete everything from the table and regenerate all the data from the
beginning. Ideally, this task would run only once, in the start when the
feature is deployed.

## Test instructions

Run the migrations.
rake "db:refresh_reporting_views" -  for monthly/daily refresh
rake "reporting:full_partitioned_refresh" - for full refresh of
reporting_patient_states.

---------

Co-authored-by: Ayushi Agrawal <[email protected]>

Author: aagrawalrtsl

app/jobs/refresh_reporting_partitioned_table_job.rb

@@ -0,0 +1,12 @@
+class RefreshReportingPartitionedTableJob
+  include Sidekiq::Worker
+
+  sidekiq_options queue: :default
+
+  def perform(reporting_month, table_name)
+    Rails.logger.info "Starting refresh for '#{table_name}' for month '#{reporting_month}' at #{Time.now.utc}"
+    ActiveRecord::Base.connection.exec_query(
+      "CALL simple_reporting.add_shard_to_table('#{reporting_month}', '#{table_name}')"
+    )
+  end
+end

app/models/reports/patient_state.rb

@@ -13,6 +13,16 @@ def self.materialized?
       true
     end
 
+    def self.partitioned?
+      true
+    end
+
+    def self.partitioned_refresh(refresh_month)
+      ActiveRecord::Base.connection.exec_query(
+        "CALL simple_reporting.add_shard_to_table('#{refresh_month}', 'reporting_patient_states')"
+      )
+    end
+
     def self.by_assigned_region(region_or_source)
       region = region_or_source.region
       where("assigned_#{region.region_type}_region_id" => region.id)

app/models/reports/refreshable.rb

@@ -10,6 +10,10 @@ def refresh(transaction: true)
       end
     end
 
+    def partitioned?
+      false
+    end
+
     private
 
     def refresh_view

app/models/reports/view.rb

@@ -33,5 +33,13 @@ def self.add_column_description(column_description, column_name)
     def self.materialized?
       raise NotImplementedError
     end
+
+    def self.get_refresh_months
+      current_date = Date.today
+      current_day = current_date.day
+      current_month = current_date.beginning_of_month
+      month_offset = (current_day / 2) + 1
+      current_day.odd? ? [current_month, current_month.prev_month] : [current_month, current_month - month_offset.month]
+    end
   end
 end

app/services/refresh_reporting_views.rb

@@ -105,18 +105,28 @@ def all_views_refreshed?
 
   def refresh
     views.each do |name|
+      klass = name.constantize
       benchmark_and_statsd(name) do
-        klass = name.constantize
         klass.refresh
       end
+
+      if klass.partitioned?
+        benchmark_and_statsd(name, true) do
+          klass.get_refresh_months.each do |refresh_month|
+            klass.partitioned_refresh(refresh_month)
+          end
+        end
+      end
     end
   end
 
-  def benchmark_and_statsd(operation)
+  def benchmark_and_statsd(operation, partitioned_refresh = false)
     view = operation == "all" ? "all" : operation.constantize.table_name
     name = "reporting_views_refresh_duration_seconds"
     result = nil
-    Metrics.benchmark_and_gauge(name, {view: view}) do
+    options_hash = {view: view}
+    options_hash[:partitioned_refresh] = true if partitioned_refresh
+    Metrics.benchmark_and_gauge(name, options_hash) do
       result = yield
     end
     result