Crash on startup with `TypeError: can't quote SolidQueue::Processes::ProcessMissingError`, perhaps wrong column name `exception`

[ttilberg]

I returned to a project I was working on last month and couldn't start bin/jobs

 ▶ bin/jobs
SolidQueue-1.2.4 Register Supervisor (30.3ms)  pid: 46471, hostname: "<HOST>", process_id: 561, name: "supervisor-b60576c40bcf94f63fc1"
SolidQueue-1.2.4 Fail claimed jobs (91.4ms)
SolidQueue-1.2.4 Started Supervisor (154.1ms)  pid: 46471, hostname: "<HOST>", process_id: 561, name: "supervisor-b60576c40bcf94f63fc1"
/Users/ttilberg/.asdf/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/activerecord-8.1.1/lib/active_record/connection_adapters/abstract/quoting.rb:87:in 'ActiveRecord::ConnectionAdapters::Quoting#quote': can't quote SolidQueue::Processes::ProcessMissingError (TypeError)

          raise TypeError, "can't quote #{value.class.name}"
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	from /Users/ttilberg/.asdf/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/activerecord-8.1.1/lib/active_record/connection_adapters/sqlite3/quoting.rb:62:in 'ActiveRecord::ConnectionAdapters::SQLite3::Quoting#quote'
	from /Users/ttilberg/.asdf/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/activerecord-8.1.1/lib/arel/collectors/substitute_binds.rb:20:in 'Arel::Collectors::SubstituteBinds#add_bind'
	from /Users/ttilberg/.asdf/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/activerecord-8.1.1/lib/arel/visitors/to_sql.rb:747:in 'Arel::Visitors::ToSql#visit_ActiveModel_Attribute'
	...
	from /Users/ttilberg/.asdf/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/activerecord-8.1.1/lib/active_record/connection_adapters/abstract/database_statements.rb:43:in 'ActiveRecord::ConnectionAdapters::DatabaseStatements#to_sql_and_binds'
	from /Users/ttilberg/.asdf/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/activerecord-8.1.1/lib/active_record/connection_adapters/abstract/database_statements.rb:74:in 'ActiveRecord::ConnectionAdapters::DatabaseStatements#select_all'
	...
	from /Users/ttilberg/.asdf/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/activerecord-8.1.1/lib/active_record/relation/finder_methods.rb:118:in 'ActiveRecord::FinderMethods#find_by!'
	from /Users/ttilberg/.asdf/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/activerecord-8.1.1/lib/active_record/relation.rb:303:in 'block in ActiveRecord::Relation#create_or_find_by!'
  ...
	from /Users/ttilberg/.asdf/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/activerecord-8.1.1/lib/active_record/relation.rb:294:in 'ActiveRecord::Relation#create_or_find_by!'
	from /Users/ttilberg/.asdf/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/activerecord-8.1.1/lib/active_record/querying.rb:24:in 'ActiveRecord::Querying#create_or_find_by!'
	from /Users/ttilberg/.asdf/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/solid_queue-1.2.4/app/models/solid_queue/job/retryable.rb:19:in 'SolidQueue::Job::Retryable#failed_with'
	from /Users/ttilberg/.asdf/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/solid_queue-1.2.4/app/models/solid_queue/claimed_execution.rb:93:in 'block in SolidQueue::ClaimedExecution#failed_with'

I asked for help from Claude to diagnose, and this seems to make sense:

In

solid_queue/app/models/solid_queue/job/retryable.rb

Line 18 in d330516

def failed_with(exception)

def failed_with(exception)
  FailedExecution.create_or_find_by!(job_id: id, exception: exception)
end

If find_by is triggered, it includes exception, which is not a column in the db. The field in the DB is error:

solid_queue/lib/generators/solid_queue/install/templates/db/queue_schema.rb

Line 24 in d330516

t.text "error"

Claude has helped me describe why this seems rare and no one has reported it:

1. A job fails with an exception, creating a FailedExecution record
2. User retries the failed job (e.g., via Mission Control)
3. The retry removes the FailedExecution and re-queues the job
4. A worker claims the job, creating a ClaimedExecution
5. The job fails again, creating a new FailedExecution record
6. The worker process dies/crashes before cleaning up the ClaimedExecution
7. When SolidQueue starts up, it tries to fail the orphaned ClaimedExecution with ProcessMissingError
8. Since a FailedExecution already exists for this job_id, the error occurs

Without having the overall system in mind, I'm not sure if the correct fix is to move exception into the on create block:

  FailedExecution.create_or_find_by!(job_id: id) {|j| j.exception = exception }  # exception virtual attr

or use the error column instead

 def failed_with(error)
   FailedExecution.create_or_find_by!(job_id: id, error: error)
 end

I think the first one is probably most appropriate, since you'd only want a single record per id.

[rosa]

Huh, this is so strange! I think Claude's diagnosis is not correct 🤔 In particular, this part:

4. A worker claims the job, creating a ClaimedExecution
5. The job fails again, creating a new FailedExecution record
6. The worker process dies/crashes before cleaning up the ClaimedExecution

On 5, the ClaimedExecution is deleted in the same transaction as the FailedExecution record is created. This means that on 6, there's no ClaimedExecution to clean up. It's either one or the other: if there's a ClaimedExecution, there can't be a FailedExecution, and the other way around.

Is it possible you've manually done something that could bring the system into this state, of having both a ClaimedExecution and a FailedExecution for the same job?

[ttilberg]

😅 You wouldn't believe how much time I went back and forth asking to try to help sort this out so as to not embarrass myself bothering you with AI slop haha, particularly since no one else seems to have had this come up. But trying to create a reproduction of what it described was all sorts of crazy because of the unique conditions. I felt it's reasoning seemed plausible :X

Unfortunately I don't know what may have happened. The resolution to get me back up involved manually deleting records from the claimed_executions, and I think the failed_executions table.

It's in a scraping project, so there's lots of failures expected, and it's in my dev env, so thing might have gotten killed in an unclean way. I don't know if that's helpful at all.

Feel free to close if you think that create_or_find_by statement looks fine with the exception attribute!

Thanks Rosa!

[rosa]

Oh, no worries at all! This is a tricky one, and your report did expose that the create_or_find_by statement there is wrong. It should simply be create 🤔. I think I used create_or_find_by because it's what I was using with the other execution types, and I just copied it from there, but it doesn't make sense there. I'll change that!