Job.data_model may reference deleted BaseDataModel on save failure

[gks281263]

Location

File: api_app/engines_manager/models.py
Method: EngineConfig.run()

Description

In EngineConfig.run(), when a job already has a data model, the existing data model is deleted and a new one is assigned. These operations are not atomic. If job.save() fails after the deletion, the job's GenericForeignKey fields (data_model_content_type, data_model_object_id) can end up referencing a deleted BaseDataModel.

This leaves the job in a partially updated and inconsistent state.

Execution Flow

Current order of operations:

1. Existing job.data_model is deleted from the database
2. New BaseDataModel is assigned to job.data_model in memory
3. job.save() is called to persist the change

If step 3 fails, the previous deletion is not rolled back.

Trigger Scenarios

job.save() can fail due to normal conditions such as:

• Validation errors
• Signal exceptions
• Database constraint errors
• Connection or transaction failures

Observed Invalid State

After a failed save:

• The old BaseDataModel is already deleted
• The database still contains FK values pointing to the deleted object
• The new BaseDataModel exists but is not linked to the job
• Accessing job.data_model raises DoesNotExist or returns None

This results in an inconsistent GenericForeignKey relationship.

Impact

If this occurs, it can lead to:

• Runtime errors when accessing job.data_model
• Engines, pivots, or connectors operating on inconsistent job state
• Queries involving job.data_model returning incorrect results
• Orphaned BaseDataModel records accumulating over time
• Difficult debugging due to silent data corruption

Expected Safe Behavior

The update should be atomic, so that either:

• The old data model remains until the new one is safely saved, or
• Partial state is prevented if the operation fails

This ensures job.data_model always references a valid object.

Notes

This issue is about atomicity and data integrity, not about changing intended behavior. The exact solution approach is open for discussion. Scope is limited to EngineConfig.run().

[fgibertoni]

Hey, thanks for raising this issue. Have you been able to replicate the error on your local instance? Can you provide some examples and more information about your environment?

[gks281263]

Thanks for the follow-up.

From my reading of EngineConfig.run(), this looks like a real edge-case consistency issue rather than intended behavior.

When a job already has a data_model, the existing BaseDataModel is deleted first and only afterwards is job.save() called to persist the new GenericForeignKey assignment.
If job.save() raises for any reason, the earlier delete is not rolled back, which can leave the job with GFK fields still pointing to a deleted object.

---

Replication status

I haven’t observed this organically in local runs yet. I’m treating it as a realistic failure mode that can occur on error paths such as:

• validation errors,
• failing pre_save / post_save signals,
• database constraint issues,
• connection problems.

Because this is a GenericForeignKey, there’s no DB-level constraint to catch it, so the inconsistency only becomes visible later when resolving job.data_model.

---

Possible local reproduction approach

1. Force job.save() to fail after the new data model is built
   (for example via a temporary failing signal or an invalid field).
2. Run EngineConfig.run(job).

Resulting state:

• the old data model is already deleted,
• the new one created by merge() exists,
• but the job still points to the deleted object and job.data_model raises DoesNotExist.

---

Impact

If this happens in production, components that assume job.data_model is always valid
(engines, pivots, connectors) can see unexpected errors, and the newly created
BaseDataModel is left unattached.

I’m happy to put together a small repro snippet if that would be helpful.

[AnshSinghal]

@mlodic please assign

[mlodic]

this is still under discussion, please choose a different task

[AnshSinghal]

@mlodic
Hi, thanks for the feedback!

According to me the issue is in EngineConfig.run(), the old data_model is deleted before job.save(). If save fails, the job's GenericForeignKey points to a deleted object.

So I propose taht we wrap the delete + save in transaction.atomic() so failures roll back the deletion.

Should I proceed with a PR for this approach, or would you prefer I look at a different issue? Happy to wait if this needs more discussion first.

[gks281263]

Thanks for the update and for taking a look at the atomicity approach.
I see the PR was closed for now. I’m happy to step back and wait for the preferred direction here.
If it’s useful later on, I’m glad to help with repro cases, tests, or validating an alternative solution.

[AnshSinghal]

Hi I am currently working on this. There was some issue with that PR. Will close this by tomorrow