Skip to content

fix: improve error handling when make_fetch referential integrity fails #1245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 16, 2025
Merged

fix: improve error handling when make_fetch referential integrity fails #1245

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e7c8943
fix: improve error handling when `make_fetch` referential integrity f…
ttngu207 Jun 13, 2025
e55bbcb
style: black format
ttngu207 Jun 13, 2025
964743e
style: format
ttngu207 Jun 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 7 additions & 8 deletions datajoint/autopopulate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,8 +105,8 @@ def make(self, key):
The method can be implemented either as:
(a) Regular method: All three steps are performed in a single database transaction.
The method must return None.
(b) Generator method:
The make method is split into three functions:
(b) Generator method:
The make method is split into three functions:
- `make_fetch`: Fetches data from the parent tables.
- `make_compute`: Computes secondary attributes based on the fetched data.
- `make_insert`: Inserts the computed data into the current table.
Expand All @@ -124,7 +124,7 @@ def make(self, key):
self.make_insert(key, *computed_result)
commit_transaction
<pseudocode>

Importantly, the output of make_fetch is a tuple that serves as the input into `make_compute`.
The output of `make_compute` is a tuple that serves as the input into `make_insert`.

Expand Down Expand Up @@ -412,11 +412,10 @@ def _populate1(
!= deepdiff.DeepHash(fetched_data, ignore_iterable_order=False)[
fetched_data
]
): # rollback due to referential integrity fail
self.connection.cancel_transaction()
logger.warning(
f"Referential integrity failed for {key} -> {self.target.full_table_name}")
return False
): # raise error if fetched data has changed
raise DataJointError(
"Referential integrity failed! The `make_fetch` data has changed"
)
gen.send(computed_result) # insert

except (KeyboardInterrupt, SystemExit, Exception) as error:
Expand Down