Issue when releasing deploy lock while using destinations

[neiljohari]

I've got a deploy.yml which omits the app section entirely:

service: mock-interviewer
image: mockinterviewer-be-shared-monolith

<%
  terraform_shared_dir = "./terraform/shared"

  repo_uri = %x(terraform -chdir=#{terraform_shared_dir} output -json ecr_repo_url | jq -r '.').strip
  image_registry = repo_uri.split("/").first
%>

proxy:
  ssl: false
  app_port: 80
  healthcheck:
    path: /api/health
    interval: 2
    timeout: 2

registry:
  server: <%= image_registry %>
  username: AWS
  password: <%= %x(aws ecr get-login-password) %>

builder:
  arch: arm64
  cache:
    type: gha

Then I've got a destination file which templates in ips using terraform outputs:

<%
  # Define the Terraform directory for staging
  terraform_env_dir = "./terraform/envs/staging"

  # Fetch the list of appserver IPs from Terraform outputs.
  app_ips_json = %x(terraform -chdir=#{terraform_env_dir} output -json appserver_ips)
  app_ips = JSON.parse(app_ips_json)

  # Fetch the DB host from Terraform outputs.
  db_host = %x(terraform -chdir=#{terraform_env_dir} output -json db_host | jq -r '.').strip
%>

# Deploy to these servers.
servers:
  web:
<% app_ips.each do |ip| %>
    - <%= ip %>
<% end %>

env:
  clear:
    DB_HOST: <%= db_host %>

When I deploy it fails to release the deploy lock:

 ERROR Failed to boot web on <IP OMITTED>
Releasing the deploy lock...
Error releasing the deploy lock: servers: should be an array or a hash
  Finished all in 33.1 seconds
  ERROR (Kamal::ConfigurationError): servers: should be an array or a hash

I can release the lock manually via kamal lock release -d staging (but if I omit the destination flag, we see the issue).

Note that even if it's a successful deployment and if I add an empty array in the base config, I still receive an error:

  INFO [a6040406] Finished in 12.135 seconds with exit status 0 (successful).
Releasing the deploy lock...
Error releasing the deploy lock: No servers specified for the web primary_role
  Finished all in 57.8 seconds
  ERROR (Kamal::ConfigurationError): No servers specified for the web primary_role

If I use the simplified syntax for defining servers (without the web: section since all servers are identical):

  ERROR (Kamal::ConfigurationError): Missing required configuration for servers

This is the stack produced by debug mode:

/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/kamal-2.4.0/lib/kamal/configuration.rb:310:in 'block in Kamal::Configuration#ensure_required_keys_present'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/kamal-2.4.0/lib/kamal/configuration.rb:309:in 'Array#each'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/kamal-2.4.0/lib/kamal/configuration.rb:309:in 'Kamal::Configuration#ensure_required_keys_present'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/kamal-2.4.0/lib/kamal/configuration.rb:76:in 'Kamal::Configuration#initialize'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/kamal-2.4.0/lib/kamal/configuration.rb:28:in 'Class#new'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/kamal-2.4.0/lib/kamal/configuration.rb:28:in 'Kamal::Configuration.create_from'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/kamal-2.4.0/lib/kamal/commander.rb:17:in 'Kamal::Commander#config'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/kamal-2.4.0/lib/kamal/cli/app.rb:316:in 'Kamal::Cli::App#using_version'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/kamal-2.4.0/lib/kamal/cli/app.rb:6:in 'block in Kamal::Cli::App#boot'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/kamal-2.4.0/lib/kamal/cli/base.rb:75:in 'Kamal::Cli::Base#with_lock'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/kamal-2.4.0/lib/kamal/cli/app.rb:4:in 'Kamal::Cli::App#boot'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/thor-1.3.2/lib/thor/command.rb:28:in 'Thor::Command#run'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/thor-1.3.2/lib/thor/invocation.rb:127:in 'Thor::Invocation#invoke_command'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/thor-1.3.2/lib/thor.rb:538:in 'Thor.dispatch'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/thor-1.3.2/lib/thor/invocation.rb:116:in 'Thor::Invocation#invoke'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/kamal-2.4.0/lib/kamal/cli/main.rb:45:in 'block (2 levels) in Kamal::Cli::Main#deploy'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/kamal-2.4.0/lib/kamal/cli/base.rb:80:in 'Kamal::Cli::Base#with_lock'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/kamal-2.4.0/lib/kamal/cli/main.rb:34:in 'block in Kamal::Cli::Main#deploy'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/kamal-2.4.0/lib/kamal/cli/base.rb:66:in 'Kamal::Cli::Base#print_runtime'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/kamal-2.4.0/lib/kamal/cli/main.rb:20:in 'Kamal::Cli::Main#deploy'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/thor-1.3.2/lib/thor/command.rb:28:in 'Thor::Command#run'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/thor-1.3.2/lib/thor/invocation.rb:127:in 'Thor::Invocation#invoke_command'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/thor-1.3.2/lib/thor.rb:538:in 'Thor.dispatch'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/thor-1.3.2/lib/thor/base.rb:584:in 'Thor::Base::ClassMethods#start'
/opt/hostedtoolcache/Ruby/3.4.1/x64/lib/ruby/gems/3.4.0/gems/kamal-2.4.0/bin/kamal:9:in '<top (required)>'
/opt/hostedtoolcache/Ruby/3.4.1/x64/bin/kamal:25:in 'Kernel#load'
/opt/hostedtoolcache/Ruby/3.4.1/x64/bin/kamal:25:in '<main>'

Are we missing forwarding the destination somewhere when releasing locks? It works fine if I hardcode the server into the base config, so it really seems like some merging is being done incorrectly.

Weirdly, this issue only happens on GHA for me but deploys work fine locally. In GHA, I've verified that kamal secrets print -d staging & kamal config -d staging return the same things as they do locally.

I'm not sure what environmental difference is causing the lock failure.

[neiljohari]

@djmb Sorry for the direct ping, any ideas on this one?

Completely stumped 🤔

[djmb]

@neiljohari - it looks like the servers section is maybe empty. Are you sure that the values are being added correctly?

You could add something like raise app_ips.inspect to the config. It would blow up because of the exception, but it would allow you to see what the values are.

[neiljohari]

@djmb Thanks for the response! Is there a way to make it only raise when trying to release the lock?

Since it does work at first (it ssh’s in, deploys the new image and updates the proxy etc). It’s only when it tries to release the lock, it seems like the reloaded config might be missing the ips

[neiljohari]

@djmb Using a counter now and sweeping through threshold values:

  # Path to the file storing the run count.
  counter_file = "./deploy.staging_run_count"

  # Determine the directory for the counter file.
  flag_dir = File.dirname(counter_file)
  Dir.mkdir(flag_dir) unless Dir.exist?(flag_dir)

  # Read the current count, or initialize to 0 if the file doesn't exist.
  run_count = File.exist?(counter_file) ? File.read(counter_file).to_i : 0

  # Increment the run count.
  run_count += 1

  # Write the updated run count back to the file.
  File.write(counter_file, run_count.to_s)

  # Set threshold (N); on the Nth run, raise an error.
  threshold = 3

  if run_count == threshold
    raise app_ips.inspect
  end

Currently everr threshold I set seems to correctly output ERROR (RuntimeError): ["<IP REDACTED>"]

[djmb]

Oh ok, it seems like then that it's incorrectly loading just the base deploy.yml.

Are you running an unreleased version? There was a bug on main that might exhibit this behaviour that I fixed today just before the Kamal 2.5.0 release.

[neiljohari]

Oh ok, it seems like then that it's incorrectly loading just the base deploy.yml.

Are you running an unreleased version? There was a bug on main that might exhibit this behaviour that I fixed today just before the Kamal 2.5.0 release.

Oh, yes I was on my fork to contribute #1392 (my deploy setup required this feature). I guess in unblocking myself on that, I got hit by the bug on main haha.

It works on v2.5.0 which also has the AWS thing I require, thank you so much! This was driving me nuts, but that's what I get for using bleeding edge 😆