Feature/metrics/rich text links (#787)

This pull request introduces a comprehensive set of features for link
checking and reporting within the BetterTogether metrics system. It adds
new models, background jobs, controllers, and mailers to support
automated link validation, reporting, and visualization. The changes are
grouped into three main themes: link checking jobs, reporting and
controller logic, and metrics visualization.

**Link Checking Infrastructure:**

* Added background jobs for checking external and internal links
(`ExternalLinkCheckerJob`, `InternalLinkCheckerJob`), including queueing
logic for distributing link checks across hosts and time windows to
avoid overload (`RichTextLinkCheckerQueueJob` and its subclasses).
[[1]](diffhunk://#diff-892afcf9e3cfd685faa3728292aa5ed13c702044e463e59ecd17d80f96e2294cR1-R36)
[[2]](diffhunk://#diff-3730260488b5742d6a5347f090b2fe1b59912fdeee917619322f35b89141e641R1-R36)
[[3]](diffhunk://#diff-1d331c511cdc570fc77a99bb25535df3a08978c61df959203e2851f4193e6e4cR1-R72)
[[4]](diffhunk://#diff-5a0fdabb875d8c63dd7b51d2f6df28fd2997f78490bde8957e0282d2fc0e6355R1-R20)
[[5]](diffhunk://#diff-ac53c5dee8b264f713f404351d664e61dbe89219a8ee72875d0501795fab0decR1-R24)
* Introduced the `BetterTogether::Content::Link` model to persist
discovered links and their metadata, and a namespace helper for
links-related tables.
[[1]](diffhunk://#diff-3fc39697bcc3a4d31a333d88732f189e98265dc7ffc2fa29ee337dff304e17d6R1-R22)
[[2]](diffhunk://#diff-fb999437387b6d81186947f52d8c17a9399c270af2f106e469615284be454f06R1-R11)

**Reporting and Controller Logic:**

* Implemented `LinkCheckerReportsController` to support creating,
listing, and downloading link checker reports, including Turbo Stream
support for dynamic updates.
* Added a scheduler job for generating and emailing link checker reports
(`LinkCheckerReportSchedulerJob`) and a mailer for delivering these
reports.
[[1]](diffhunk://#diff-64c7d345d921d3b7820f2dd9e270c04cdf3c4bace7e649e4c9de41c4dc109a89R1-R20)
[[2]](diffhunk://#diff-116ca85b0e59523938b0694f2fd80d23c50bfa970460b583222f3ca0e5519729R1-R25)
* Added an event reminder scan job to periodically enqueue reminders for
upcoming events.

**Metrics Visualization and Testing:**

* Enhanced metrics reporting by aggregating link data for charts (links
by host, invalid links by host, failures over time) and exposed these in
the metrics reports controller.
* Updated the metrics charts controller and JavaScript to render new
charts for link-related metrics.
[[1]](diffhunk://#diff-92ebbff7abacb731625c22d4c9c4a62d0de515eab1fb84c6bb6f8cc6160dbae3L61-R61)
[[2]](diffhunk://#diff-92ebbff7abacb731625c22d4c9c4a62d0de515eab1fb84c6bb6f8cc6160dbae3R71-R73)
[[3]](diffhunk://#diff-92ebbff7abacb731625c22d4c9c4a62d0de515eab1fb84c6bb6f8cc6160dbae3R209-R262)
* Added the `webmock` gem for stubbing external HTTP requests in specs,
improving test reliability.

Author: rsmithlal

Gemfile

@@ -98,6 +98,8 @@ group :test do
   # Capybara for integration testing
   gem 'capybara', '>= 2.15'
   gem 'capybara-screenshot'
+  # WebMock for stubbing external HTTP requests in specs
+  gem 'webmock'
   # Coveralls for test coverage reporting
   gem 'coveralls_reborn', require: false
   # Database cleaner for test database cleaning

Gemfile.lock

@@ -57,6 +57,7 @@ PATH
       reform-rails (>= 0.2, < 0.4)
       rswag (>= 2.3.1, < 2.17.0)
       ruby-openai
+      sidekiq-scheduler
       simple_calendar
       sprockets-rails
       stackprof
@@ -229,6 +230,9 @@ GEM
       term-ansicolor (~> 1.7)
       thor (~> 1.2)
       tins (~> 1.32)
+    crack (1.0.0)
+      bigdecimal
+      rexml
     crass (1.0.6)
     css_parser (1.21.1)
       addressable
@@ -297,6 +301,8 @@ GEM
       multi_json
     erb (5.0.2)
     erubi (1.13.1)
+    et-orbi (1.3.0)
+      tzinfo
     event_stream_parser (1.0.0)
     excon (1.2.8)
       logger
@@ -343,6 +349,9 @@ GEM
     friendly_id-mobility (1.0.4)
       friendly_id (>= 5.0.0, < 5.5)
       mobility (>= 1.0.1, < 2.0)
+    fugit (1.11.2)
+      et-orbi (~> 1, >= 1.2.11)
+      raabro (~> 1.4)
     fuubar (2.5.1)
       rspec-core (~> 3.0)
       ruby-progressbar (~> 1.4)
@@ -362,6 +371,7 @@ GEM
       rake (>= 13)
     groupdate (6.7.0)
       activesupport (>= 7.1)
+    hashdiff (1.2.0)
     hashie (5.0.0)
     highline (3.1.2)
       reline
@@ -521,6 +531,7 @@ GEM
       nio4r (~> 2.0)
     pundit (2.5.0)
       activesupport (>= 3.0.0)
+    raabro (1.4.0)
     racc (1.8.1)
     rack (3.1.16)
     rack-attack (6.7.0)
@@ -699,6 +710,8 @@ GEM
       ffi (~> 1.12)
       logger
     rubyzip (3.0.1)
+    rufus-scheduler (3.9.2)
+      fugit (~> 1.1, >= 1.11.1)
     sass-embedded (1.86.3-aarch64-linux-gnu)
       google-protobuf (~> 4.30)
     sass-embedded (1.86.3-arm64-darwin)
@@ -732,6 +745,9 @@ GEM
       logger (>= 1.6.2)
       rack (>= 3.1.0)
       redis-client (>= 0.23.2)
+    sidekiq-scheduler (6.0.1)
+      rufus-scheduler (~> 3.2)
+      sidekiq (>= 7.3, < 9)
     simple_calendar (3.1.0)
       rails (>= 6.1)
     simplecov (0.22.0)
@@ -801,6 +817,10 @@ GEM
       activemodel (>= 6.0.0)
       bindex (>= 0.4.0)
       railties (>= 6.0.0)
+    webmock (3.25.1)
+      addressable (>= 2.8.0)
+      crack (>= 0.3.2)
+      hashdiff (>= 0.4.0, < 2.0.0)
     websocket (1.2.11)
     websocket-driver (0.8.0)
       base64
@@ -873,6 +893,7 @@ DEPENDENCIES
   storext!
   uglifier (>= 1.3.0)
   web-console (>= 3.3.0)
+  webmock
 
 RUBY VERSION
    ruby 3.4.4p34

app/controllers/better_together/metrics/link_checker_reports_controller.rb

@@ -0,0 +1,109 @@
+# frozen_string_literal: true
+
+module BetterTogether
+  module Metrics
+    # Controller for creating and downloading Link Checker reports
+    class LinkCheckerReportsController < ApplicationController
+      before_action :set_report, only: %i[download]
+
+      def index
+        @link_checker_reports = BetterTogether::Metrics::LinkCheckerReport.order(created_at: :desc)
+
+        if request.headers['Turbo-Frame'].present?
+          render partial: 'better_together/metrics/link_checker_reports/index',
+                 locals: { reports: @link_checker_reports },
+                 layout: false
+        else
+          render :index
+        end
+      end
+
+      def new
+        @link_checker_report = BetterTogether::Metrics::LinkCheckerReport.new
+      end
+
+      # rubocop:todo Metrics/AbcSize
+      # rubocop:todo Metrics/MethodLength
+      # rubocop:todo Metrics/BlockLength
+      def create
+        opts = {
+          from_date: params.dig(:metrics_link_checker_report, :filters, :from_date),
+          to_date: params.dig(:metrics_link_checker_report, :filters, :to_date),
+          file_format: params.dig(:metrics_link_checker_report, :file_format) || 'csv'
+        }
+
+        @link_checker_report = BetterTogether::Metrics::LinkCheckerReport.create_and_generate!(**opts)
+
+        respond_to do |format| # rubocop:todo Metrics/BlockLength
+          if @link_checker_report.persisted?
+            flash[:notice] = t('flash.generic.created', resource: t('resources.report'))
+            format.html { redirect_to metrics_link_checker_reports_path, notice: flash[:notice] }
+            format.turbo_stream do
+              render turbo_stream: [
+                turbo_stream.prepend(
+                  'link_checker_reports_table_body',
+                  partial: 'better_together/metrics/link_checker_reports/link_checker_report',
+                  locals: { link_checker_report: @link_checker_report }
+                ),
+                turbo_stream.replace(
+                  'flash_messages',
+                  partial: 'layouts/better_together/flash_messages',
+                  locals: { flash: flash }
+                ),
+                turbo_stream.replace('new_link_checker_report',
+                                     '<turbo-frame id="new_link_checker_report"></turbo-frame>')
+              ]
+            end
+          else
+            flash.now[:alert] = t('flash.generic.error_create', resource: t('resources.report'))
+            format.html { render :new, status: :unprocessable_content }
+            format.turbo_stream do
+              render turbo_stream: [
+                turbo_stream.update('form_errors', partial: 'layouts/errors', locals: { object: @link_checker_report }),
+                turbo_stream.replace(
+                  'flash_messages',
+                  partial: 'layouts/better_together/flash_messages',
+                  locals: { flash: flash }
+                )
+              ]
+            end
+          end
+        end
+      end
+      # rubocop:enable Metrics/BlockLength
+      # rubocop:enable Metrics/MethodLength
+      # rubocop:enable Metrics/AbcSize
+
+      # rubocop:todo Metrics/AbcSize
+      # rubocop:todo Metrics/MethodLength
+      def download
+        if @link_checker_report.report_file.attached?
+          BetterTogether::Metrics::TrackDownloadJob.perform_later(
+            @link_checker_report,
+            @link_checker_report.report_file.filename.to_s,
+            @link_checker_report.report_file.content_type,
+            @link_checker_report.report_file.byte_size,
+            I18n.locale.to_s
+          )
+
+          send_data @link_checker_report.report_file.download,
+                    filename: @link_checker_report.report_file.filename.to_s,
+                    type: @link_checker_report.report_file.content_type,
+                    disposition: 'attachment'
+
+          return
+        end
+
+        redirect_to metrics_link_checker_reports_path, alert: t('resources.download_failed')
+      end
+      # rubocop:enable Metrics/MethodLength
+      # rubocop:enable Metrics/AbcSize
+
+      private
+
+      def set_report
+        @link_checker_report = BetterTogether::Metrics::LinkCheckerReport.find(params[:id])
+      end
+    end
+  end
+end

app/controllers/better_together/metrics/reports_controller.rb

@@ -67,6 +67,12 @@ def index # rubocop:todo Metrics/AbcSize, Metrics/MethodLength
             }
           end
         }
+
+        # Link Checker charts: aggregate data from stored links
+        links_scope = BetterTogether::Content::Link.all
+        @links_by_host = links_scope.group(:host).count
+        @invalid_by_host = links_scope.where(valid_link: false).group(:host).count
+        @failures_daily = links_scope.where(valid_link: false).group_by_day(:last_checked_at).count
       end
 
       # A helper method to generate a random color for each platform (this can be customized).

app/javascript/controllers/better_together/metrics_charts_controller.js

@@ -58,7 +58,7 @@ const platformBorderColors = {
 };
 
 export default class extends Controller {
-  static targets = ["pageViewsChart", "dailyPageViewsChart", "linkClicksChart", "dailyLinkClicksChart", "downloadsChart", "sharesChart", "sharesPerUrlPerPlatformChart"]
+  static targets = ["pageViewsChart", "dailyPageViewsChart", "linkClicksChart", "dailyLinkClicksChart", "downloadsChart", "sharesChart", "sharesPerUrlPerPlatformChart", "linksByHostChart", "invalidByHostChart", "failuresDailyChart"]
 
   connect() {
     this.renderPageViewsChart()
@@ -68,6 +68,9 @@ export default class extends Controller {
     this.renderDownloadsChart()
     this.renderSharesChart()
     this.renderSharesPerUrlPerPlatformChart()
+  this.renderLinksByHostChart()
+  this.renderInvalidByHostChart()
+  this.renderFailuresDailyChart()
   }
 
   renderPageViewsChart() {
@@ -203,4 +206,58 @@ export default class extends Controller {
       })
     })
   }
+
+  renderLinksByHostChart() {
+    const data = JSON.parse(this.linksByHostChartTarget.dataset.chartData)
+    new Chart(this.linksByHostChartTarget, {
+      type: 'bar',
+      data: {
+        labels: data.labels,
+        datasets: [{
+          label: 'Links by Host',
+          data: data.values,
+          backgroundColor: 'rgba(99, 132, 255, 0.2)',
+          borderColor: 'rgba(99, 132, 255, 1)',
+          borderWidth: 1
+        }]
+      },
+      options: Object.assign({}, sharedChartOptions)
+    })
+  }
+
+  renderInvalidByHostChart() {
+    const data = JSON.parse(this.invalidByHostChartTarget.dataset.chartData)
+    new Chart(this.invalidByHostChartTarget, {
+      type: 'bar',
+      data: {
+        labels: data.labels,
+        datasets: [{
+          label: 'Invalid Links by Host',
+          data: data.values,
+          backgroundColor: 'rgba(255, 159, 64, 0.2)',
+          borderColor: 'rgba(255, 159, 64, 1)',
+          borderWidth: 1
+        }]
+      },
+      options: Object.assign({}, sharedChartOptions)
+    })
+  }
+
+  renderFailuresDailyChart() {
+    const data = JSON.parse(this.failuresDailyChartTarget.dataset.chartData)
+    new Chart(this.failuresDailyChartTarget, {
+      type: 'line',
+      data: {
+        labels: data.labels,
+        datasets: [{
+          label: 'Invalid Links Over Time',
+          data: data.values,
+          backgroundColor: 'rgba(255, 99, 132, 0.2)',
+          borderColor: 'rgba(255, 99, 132, 1)',
+          borderWidth: 1
+        }]
+      },
+      options: Object.assign({}, sharedChartOptions)
+    })
+  }
 }

app/jobs/better_together/event_reminder_scan_job.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module BetterTogether
+  # Scans upcoming events and schedules per-event reminders via
+  # BetterTogether::EventReminderSchedulerJob. This job is intended to be run
+  # periodically by the scheduler (sidekiq-scheduler / sidekiq-cron).
+  class EventReminderScanJob < ApplicationJob
+    queue_as :notifications
+
+    # Keep lightweight: find events starting in the near future and enqueue the
+    # existing per-event scheduler job which handles cancellation/rescheduling.
+    # default: next 7 days
+    def perform(window_hours: 168)
+      cutoff = Time.current + window_hours.hours
+
+      BetterTogether::Event.where('starts_at <= ? AND starts_at >= ?', cutoff, Time.current).find_each do |event|
+        # Use the id to avoid serializing AR objects into the job payload
+        BetterTogether::EventReminderSchedulerJob.perform_later(event.id)
+      rescue StandardError => e
+        Rails.logger.error "Failed to enqueue reminder scheduler for event #{event&.id}: #{e.message}"
+      end
+    end
+  end
+end

app/jobs/better_together/metrics/external_link_checker_job.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+require 'net/http'
+require 'uri'
+
+module BetterTogether
+  module Metrics
+    # Background job that checks an external link's HTTP status and updates
+    # the corresponding BetterTogether::Content::Link record with the
+    # latest check timestamp, status code and validity flag.
+    class ExternalLinkCheckerJob < ApplicationJob
+      queue_as :default
+
+      def perform(link_id)
+        link = BetterTogether::Content::Link.find(link_id)
+        checker = BetterTogether::Metrics::HttpLinkChecker.new(link.url)
+        result = checker.call
+
+        update_link_from_result(link, result)
+      end
+
+      private
+
+      def update_link_from_result(link, result)
+        attrs = {
+          last_checked_at: Time.current,
+          latest_status_code: result.status_code,
+          valid_link: result.success
+        }
+
+        attrs[:error_message] = result.error&.message unless result.success
+        link.update!(attrs)
+      end
+    end
+  end
+end

app/jobs/better_together/metrics/internal_link_checker_job.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+require 'net/http'
+require 'uri'
+
+module BetterTogether
+  module Metrics
+    # Background job that checks an internal link's HTTP status and updates
+    # the corresponding BetterTogether::Content::Link record with the
+    # latest check timestamp, status code and validity flag.
+    class InternalLinkCheckerJob < ApplicationJob
+      queue_as :default
+
+      def perform(link_id)
+        link = BetterTogether::Content::Link.find(link_id)
+        checker = BetterTogether::Metrics::HttpLinkChecker.new(link.url)
+        result = checker.call
+
+        update_link_from_result(link, result)
+      end
+
+      private
+
+      def update_link_from_result(link, result)
+        attrs = {
+          last_checked_at: Time.current,
+          latest_status_code: result.status_code,
+          valid_link: result.success
+        }
+
+        attrs[:error_message] = result.error&.message unless result.success
+        link.update!(attrs)
+      end
+    end
+  end
+end