Setup Azure storage for ActiveStorage (#8531)

scruti · web-flow · commit 3c32d48cd3b1 · 2026-03-02T10:01:25.000Z
## Trello card URL - https://trello.com/c/UtzBnbCV/2097-move-from-aws-s3-to-azure-storage ## Changes in this PR: This PR sets and backfills the synchronisation between AWS S3 and Azure Storage for all the service attachments. ### Sets up the connection to Azure storage services: - Include the necessary gem (azure-blob) - Set env variables from Terraform module. - Use env variables and configure storage in Rails Active Storage. ### Sets up mirroring between S3 and Azure storage - Define and point to [mirror services](https://guides.rubyonrails.org/active_storage_overview.html#mirror-service) so any new attachments will be pushed to S3 and also to Azure. So all new uploads will be sync. ### Add backfill task for existing attachments - A rake task will re-point all the existing attachments to the mirror services and then trigger a mirror request for all of them. This will do a backfill, syncing all the existing attachments into Azure. ## What will need to come as a follow-up After we are satisfied with the backfill, we will need to push a further PR: - Removing S3 and Mirror services and pointing instead to Azure services. - Adding a rake task to repoint all the existing blobs to the Azure services. ## Screenshots of UI changes: After setting up mirrors and backfilling with the rake task. The blobs in both storages match: ### For documents <img width="1559" height="733" alt="image" src="https://github.com/user-attachments/assets/2556e4ec-3cfd-47ef-9ef7-3581c70edeaa" /> ### For images and logos <img width="1397" height="1093" alt="image" src="https://github.com/user-attachments/assets/04ad0637-91a5-4569-b671-503fe6320fe3" /> ## Checklists: ### Data & Schema Changes If this PR modifies data structures or validations, check the following: - [ ] Adds/removes model validations - [ ] Adds/removes database fields - [ ] Modifies Vacancy enumerables (phases, working patterns, job roles, key stages, etc.) <details> <summary>If any of the above options has changed then the author must check/resolve all of the following...</summary> ### Integration Impact Does this change affect any of these integrations? - [ ] DfE Analytics platform - [ ] Legacy imports mappings - [ ] DWP Find a Job export mappings - [ ] Publisher ATS API (may require mapping updates or API versioning) ### User Experience & Data Integrity Could this change impact: - [ ] Existing subscription alerts (will legacy subscription search filters break?) - [ ] Legacy vacancy copying (will copied vacancies fail new validations?) - [ ] In-progress drafts for Vacancies or Job Applications </details>
diff --git a/Dockerfile b/Dockerfile
@@ -37,6 +37,8 @@ COPY . .
 
 ENV DOCUMENTS_S3_BUCKET=throwaway_value
 ENV SCHOOLS_IMAGES_LOGOS_S3_BUCKET=throwaway_value
+ENV DOCUMENTS_AZURE_STORAGE_ACCESS_KEY=throwaway_value
+ENV IMAGES_LOGOS_AZURE_STORAGE_ACCESS_KEY=throwaway_value
 
 RUN --mount=type=secret,id=master_key,env=RAILS_MASTER_KEY RAILS_ENV=production SECRET_KEY_BASE=required-to-run-but-not-used RAILS_SERVE_STATIC_FILES=1 bundle exec rake assets:precompile
 
diff --git a/Gemfile b/Gemfile
@@ -24,6 +24,7 @@ gem "active_storage_validations"
 gem "addressable"
 gem "array_enum"
 gem "aws-sdk-s3"
+gem "azure-blob", require: false
 gem "business_time"
 gem "chartkick"
 gem "devise"
diff --git a/Gemfile.lock b/Gemfile.lock
@@ -154,6 +154,9 @@ GEM
       descendants_tracker (~> 0.0.4)
       ice_nine (~> 0.11.0)
       thread_safe (~> 0.3, >= 0.3.1)
+    azure-blob (0.8.0)
+      cgi
+      rexml
     backport (1.2.0)
     base64 (0.3.0)
     bcrypt (3.1.21)
@@ -991,6 +994,7 @@ DEPENDENCIES
   aws-sdk-ssm
   axe-core-capybara
   axe-core-rspec
+  azure-blob
   better_errors
   binding_of_caller
   brakeman
@@ -1153,6 +1157,7 @@ CHECKSUMS
   axe-core-capybara (4.11.1) sha256=5d374e6e08b1831520e2a24ca5d2158865b8ab86346060131bf773f7f9fd1449
   axe-core-rspec (4.11.1) sha256=dc6c0e166405cd3a28c4a0937f6521ee5b511c12c0ca1627144a1ee7d5014aec
   axiom-types (0.1.1) sha256=c1ff113f3de516fa195b2db7e0a9a95fd1b08475a502ff660d04507a09980383
+  azure-blob (0.8.0) sha256=5c8d50e5c8b4fa9228f6a8d6bf003aba9f33d03c15f525235adeca8ad7879c10
   backport (1.2.0) sha256=912c7dfdd9ee4625d013ddfccb6205c3f92da69a8990f65c440e40f5b2fc7f75
   base64 (0.3.0) sha256=27337aeabad6ffae05c265c450490628ef3ebd4b67be58257393227588f5a97b
   bcrypt (3.1.21) sha256=5964613d750a42c7ee5dc61f7b9336fb6caca429ba4ac9f2011609946e4a2dcf
diff --git a/app/jobs/active_storage/mirror_job.rb b/app/jobs/active_storage/mirror_job.rb
@@ -0,0 +1,20 @@
+class ActiveStorage::MirrorJob < ActiveStorage::BaseJob
+  queue_as { ActiveStorage.queues[:mirror] }
+
+  discard_on ActiveStorage::FileNotFoundError
+  retry_on ActiveStorage::IntegrityError, attempts: 10, wait: :polynomially_longer
+
+  # Override to fix Rails bug where MirrorJob always uses the default service
+  # instead of the blob's actual service. This causes mirroring to fail for
+  # attachments using non-default ActiveStorage services (e.g., images_and_logos).
+  #
+  # See: https://github.com/rails/rails/issues/46806
+  # Proposed fix: https://github.com/Sandgarden-Demo/rails/pull/31/changes
+  def perform(key, checksum:)
+    if (blob = ActiveStorage::Blob.find_by(key: key))
+      blob.service.try(:mirror, blob.key, checksum: blob.checksum)
+    else
+      ActiveStorage::Blob.service.try(:mirror, key, checksum: checksum)
+    end
+  end
+end
diff --git a/app/models/job_application.rb b/app/models/job_application.rb
@@ -141,7 +141,7 @@ class JobApplication < ApplicationRecord
 
   validates :email_address, email_address: true, if: -> { email_address_changed? } # Allows data created prior to validation to still be valid
 
-  has_one_attached :baptism_certificate, service: :amazon_s3_documents
+  has_one_attached :baptism_certificate, service: :mirror_documents
 
   validate :status_transition, if: -> { status_changed? }
 
diff --git a/app/models/organisation.rb b/app/models/organisation.rb
@@ -12,8 +12,8 @@ class Organisation < ApplicationRecord
 
   friendly_id :slug_candidates, use: %i[slugged history]
 
-  has_one_attached :logo, service: :amazon_s3_images_and_logos
-  has_one_attached :photo, service: :amazon_s3_images_and_logos
+  has_one_attached :logo, service: :mirror_images_and_logos
+  has_one_attached :photo, service: :mirror_images_and_logos
 
   has_many :organisation_vacancies, dependent: :destroy
   has_many :vacancies, through: :organisation_vacancies
diff --git a/app/models/vacancy.rb b/app/models/vacancy.rb
@@ -87,12 +87,12 @@ class Vacancy < ApplicationRecord
     valid_file_types: %i[PDF DOC DOCX],
   }.freeze
 
-  has_many_attached :supporting_documents, service: :amazon_s3_documents
+  has_many_attached :supporting_documents, service: :mirror_documents
 
   validates :supporting_documents, content_type: DOCUMENT_CONTENT_TYPES,
                                    size: { less_than: DOCUMENT_FILE_SIZE_LIMIT }, virus_free: true, if: -> { include_additional_documents }
 
-  has_one_attached :application_form, service: :amazon_s3_documents
+  has_one_attached :application_form, service: :mirror_documents
 
   has_many :saved_jobs, dependent: :destroy
   has_many :saved_by, through: :saved_jobs, source: :jobseeker
diff --git a/config/environments/production.rb b/config/environments/production.rb
@@ -5,7 +5,7 @@
 
   # The application uses multiple services for storing files. This sets up a default value which gets overridden
   # in every specific use case.
-  config.active_storage.service = :amazon_s3_documents
+  config.active_storage.service = :mirror_documents
 
   # Configure the domains permitted to access coordinates API
   config.allowed_cors_origin = proc { "https://#{DOMAIN}" }
diff --git a/config/storage/development.yml b/config/storage/development.yml
@@ -6,6 +6,22 @@ amazon_s3_images_and_logos:
   service: Disk
   root: <%= Rails.root.join("storage") %>
 
+azure_storage_documents:
+  service: Disk
+  root: <%= Rails.root.join("tmp/storage") %>
+
+azure_storage_images_and_logos:
+  service: Disk
+  root: <%= Rails.root.join("tmp/storage") %>
+
+mirror_documents:
+  service: Disk
+  root: <%= Rails.root.join("tmp/storage") %>
+
+mirror_images_and_logos:
+  service: Disk
+  root: <%= Rails.root.join("tmp/storage") %>
+
 local:
   service: Disk
   root: <%= Rails.root.join("storage") %>
diff --git a/config/storage/production.yml b/config/storage/production.yml
@@ -11,3 +11,27 @@ amazon_s3_images_and_logos:
   secret_access_key: <%= ENV["SCHOOLS_IMAGES_LOGOS_ACCESS_KEY_SECRET"] %>
   bucket: <%= ENV["SCHOOLS_IMAGES_LOGOS_S3_BUCKET"] %>
   region: eu-west-2
+
+azure_storage_documents:
+  service: AzureBlob
+  storage_account_name: <%= ENV["DOCUMENTS_AZURE_STORAGE_ACCOUNT_NAME"] %>
+  storage_access_key: <%= ENV["DOCUMENTS_AZURE_STORAGE_ACCESS_KEY"] %>
+  container: documents
+
+azure_storage_images_and_logos:
+  service: AzureBlob
+  storage_account_name: <%= ENV["IMAGES_LOGOS_AZURE_STORAGE_ACCOUNT_NAME"] %>
+  storage_access_key: <%= ENV["IMAGES_LOGOS_AZURE_STORAGE_ACCESS_KEY"] %>
+  container: images-logos
+
+mirror_documents:
+  service: Mirror
+  primary: amazon_s3_documents
+  mirrors:
+    - azure_storage_documents
+
+mirror_images_and_logos:
+  service: Mirror
+  primary: amazon_s3_images_and_logos
+  mirrors:
+    - azure_storage_images_and_logos
diff --git a/config/storage/test.yml b/config/storage/test.yml
@@ -6,6 +6,22 @@ amazon_s3_images_and_logos:
   service: Disk
   root: <%= Rails.root.join("tmp/storage") %>
 
+azure_storage_documents:
+  service: Disk
+  root: <%= Rails.root.join("tmp/storage") %>
+
+azure_storage_images_and_logos:
+  service: Disk
+  root: <%= Rails.root.join("tmp/storage") %>
+
+mirror_documents:
+  service: Disk
+  root: <%= Rails.root.join("tmp/storage") %>
+
+mirror_images_and_logos:
+  service: Disk
+  root: <%= Rails.root.join("tmp/storage") %>
+
 test:
   service: Disk
   root: <%= Rails.root.join("tmp/storage") %>
diff --git a/lib/tasks/backfill_azure_storage.rake b/lib/tasks/backfill_azure_storage.rake
@@ -0,0 +1,61 @@
+desc "Backfill Azure storage by migrating blobs to mirror services and triggering mirroring"
+# rubocop:disable Metrics/BlockLength
+
+task backfill_azure_storage: :environment do
+  puts "Starting Azure storage backfill..."
+  puts "=" * 80
+
+  # We need to update the service_name for blobs using the old services to point to the new mirror services, so that
+  # the mirroring service can pick them up and mirror them to Azure.
+  # This is necessary because the MirrorJob looks for blobs using the mirror service names.
+  migrations = {
+    "amazon_s3_documents" => "mirror_documents",
+    "amazon_s3_images_and_logos" => "mirror_images_and_logos",
+  }
+
+  total_updated = 0
+  total_mirrored = 0
+
+  migrations.each do |old_service, new_service|
+    puts "\nMigrating blobs from '#{old_service}' to '#{new_service}'..."
+
+    blobs = ActiveStorage::Blob.where(service_name: old_service)
+    count = blobs.count
+
+    if count.zero?
+      puts "No blobs found for service '#{old_service}'"
+    else
+      puts "Found #{count} blob(s) to migrate"
+      # Update service names in batches
+      updated_count = blobs.update_all(service_name: new_service)
+      total_updated += updated_count
+      puts "✓ Updated #{updated_count} blob(s) to use '#{new_service}'"
+    end
+  end
+
+  # Now trigger mirroring for all blobs using mirror services
+  puts "\n#{'=' * 80}"
+  puts "Triggering mirroring jobs..."
+
+  mirror_services = migrations.values
+  mirror_blobs = ActiveStorage::Blob.where(service_name: mirror_services)
+  mirror_count = mirror_blobs.count
+
+  if mirror_count.positive?
+    puts "Found #{mirror_count} blob(s) to mirror"
+    mirror_blobs.find_each do |blob|
+      blob.mirror_later
+      total_mirrored += 1
+    end
+    puts "✓ Queued #{total_mirrored} blob(s) for mirroring"
+  else
+    puts "No blobs found for mirroring"
+  end
+
+  puts "\n#{'=' * 80}"
+  puts "Backfill complete!"
+  puts "Service names updated: #{total_updated}"
+  puts "Mirror jobs queued: #{total_mirrored}"
+  puts "=" * 80
+end
+# rubocop:enable Metrics/BlockLength
diff --git a/spec/jobs/active_storage/mirror_job_spec.rb b/spec/jobs/active_storage/mirror_job_spec.rb
@@ -0,0 +1,103 @@
+require "rails_helper"
+
+RSpec.describe ActiveStorage::MirrorJob do
+  let(:key) { "test-key-123" }
+  let(:checksum) { "abc123checksum" }
+
+  describe "#perform" do
+    context "when the blob exists" do
+      let(:service) { instance_double(ActiveStorage::Service, try: true) }
+      let(:blob) do
+        instance_double(ActiveStorage::Blob, key: key, checksum: checksum, service: service)
+      end
+
+      before do
+        allow(ActiveStorage::Blob).to receive(:find_by).with(key: key).and_return(blob)
+      end
+
+      it "uses the blob's service to mirror the file" do
+        expect(service).to receive(:try).with(:mirror, blob.key, checksum: blob.checksum)
+
+        described_class.perform_now(key, checksum: checksum)
+      end
+
+      it "uses the blob's checksum rather than the provided checksum" do
+        different_checksum = "different-checksum"
+        expect(service).to receive(:try).with(:mirror, blob.key, checksum: blob.checksum)
+
+        described_class.perform_now(key, checksum: different_checksum)
+      end
+    end
+
+    context "when the blob does not exist" do
+      let(:default_service) { instance_double(ActiveStorage::Service, try: true) }
+
+      before do
+        allow(ActiveStorage::Blob).to receive(:find_by).with(key: key).and_return(nil)
+        allow(ActiveStorage::Blob).to receive(:service).and_return(default_service)
+      end
+
+      it "uses the default ActiveStorage service with the provided checksum to mirror the file" do
+        expect(default_service).to receive(:try).with(:mirror, key, checksum: checksum)
+
+        described_class.perform_now(key, checksum: checksum)
+      end
+    end
+
+    context "when the service does not respond to mirror" do
+      let(:service) { instance_double(ActiveStorage::Service) }
+      let(:blob) do
+        instance_double(ActiveStorage::Blob, key: key, checksum: checksum, service: service)
+      end
+
+      before do
+        allow(ActiveStorage::Blob).to receive(:find_by).with(key: key).and_return(blob)
+        allow(service).to receive(:try).with(:mirror, blob.key, checksum: blob.checksum).and_return(nil)
+      end
+
+      it "does not raise an error" do
+        expect { described_class.perform_now(key, checksum: checksum) }.not_to raise_error
+      end
+    end
+  end
+
+  describe "error handling" do
+    context "when ActiveStorage::FileNotFoundError is raised" do
+      let(:service) { instance_double(ActiveStorage::Service) }
+      let(:blob) do
+        instance_double(ActiveStorage::Blob, key: key, checksum: checksum, service: service)
+      end
+
+      before do
+        allow(ActiveStorage::Blob).to receive(:find_by).with(key: key).and_return(blob)
+        allow(service).to receive(:try).and_raise(ActiveStorage::FileNotFoundError)
+      end
+
+      it "discards the job without raising an error" do
+        expect { described_class.perform_now(key, checksum: checksum) }.not_to raise_error
+      end
+    end
+
+    context "when ActiveStorage::IntegrityError is raised" do
+      let(:service) { instance_double(ActiveStorage::Service) }
+      let(:blob) do
+        instance_double(ActiveStorage::Blob, key: key, checksum: checksum, service: service)
+      end
+
+      before do
+        allow(ActiveStorage::Blob).to receive(:find_by).with(key: key).and_return(blob)
+        allow(service).to receive(:try).and_raise(ActiveStorage::IntegrityError)
+      end
+
+      it "schedules a retry instead of discarding" do
+        # The retry_on configuration means the job will retry automatically
+        # rather than being discarded like FileNotFoundError
+        expect {
+          Sidekiq::Testing.inline! do
+            described_class.perform_now(key, checksum: checksum)
+          end
+        }.not_to raise_error
+      end
+    end
+  end
+end
diff --git a/spec/lib/tasks/backfill_azure_storage_rake_task_spec.rb b/spec/lib/tasks/backfill_azure_storage_rake_task_spec.rb
diff --git a/terraform/app/modules/paas/aks_application.tf b/terraform/app/modules/paas/aks_application.tf
diff --git a/terraform/app/modules/paas/variables.tf b/terraform/app/modules/paas/variables.tf
