Skip to content

Commit 4637708

Browse files
committed
Add table drop/alter behavior and schema.jobs list API
- Jobs table automatically dropped when target table is dropped/altered - schema.jobs returns list of JobsTable objects for all auto-populated tables - Updated dashboard examples to use schema.jobs iteration
1 parent 9110515 commit 4637708

File tree

1 file changed

+53
-12
lines changed

1 file changed

+53
-12
lines changed

docs/src/design/autopopulate-2.0-spec.md

Lines changed: 53 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -389,6 +389,26 @@ result = FilteredImage.jobs.refresh()
389389
- Stale jobs are harmless until refresh—they simply won't match key_source
390390
- The `refresh()` approach is more efficient for batch cleanup
391391

392+
### Table Drop and Alter Behavior
393+
394+
When an auto-populated table is **dropped**, its associated jobs table is automatically dropped:
395+
396+
```python
397+
# Dropping FilteredImage also drops ~filtered_image__jobs
398+
FilteredImage.drop()
399+
```
400+
401+
When an auto-populated table is **altered** (e.g., primary key changes), the jobs table is dropped and can be recreated via `refresh()`:
402+
403+
```python
404+
# Alter that changes primary key structure
405+
# Jobs table is dropped since its structure no longer matches
406+
FilteredImage.alter()
407+
408+
# Recreate jobs table with new structure
409+
FilteredImage.jobs.refresh()
410+
```
411+
392412
### Migration from Current System
393413

394414
The schema-level `~jobs` table will be:
@@ -508,20 +528,30 @@ FilteredImage.jobs.reset(include_errors=True)
508528
### Dashboard Queries
509529

510530
```python
511-
# Get pipeline-wide status
531+
# Get pipeline-wide status using schema.jobs
512532
def pipeline_status(schema):
513-
status = {}
514-
for table in schema.list_tables():
515-
tbl = getattr(schema, table)
516-
if hasattr(tbl, 'jobs'):
517-
status[table] = tbl.jobs.progress()
518-
return status
533+
return {
534+
jt.target.table_name: jt.progress()
535+
for jt in schema.jobs
536+
}
519537

520538
# Example output:
521539
# {
522540
# 'FilteredImage': {'pending': 150, 'reserved': 3, 'success': 847, 'error': 12},
523541
# 'Analysis': {'pending': 500, 'reserved': 0, 'success': 0, 'error': 0},
524542
# }
543+
544+
# Refresh all jobs tables in the schema
545+
for jobs_table in schema.jobs:
546+
jobs_table.refresh()
547+
548+
# Get all errors across the pipeline
549+
all_errors = []
550+
for jt in schema.jobs:
551+
errors = jt.errors.fetch(as_dict=True)
552+
for err in errors:
553+
err['_table'] = jt.target.table_name
554+
all_errors.append(err)
525555
```
526556

527557
## Backward Compatibility
@@ -544,16 +574,27 @@ def pipeline_status(schema):
544574

545575
### API Compatibility
546576

547-
The `schema.jobs` property will continue to work but return a unified view:
577+
The `schema.jobs` property returns a list of all jobs table objects for auto-populated tables in the schema:
548578

549579
```python
550-
# Returns all jobs across all tables in the schema
551-
schema.jobs # Deprecated, shows warning
580+
# Returns list of JobsTable objects
581+
schema.jobs
582+
# [FilteredImage.jobs, Analysis.jobs, ...]
552583

553-
# Equivalent to:
554-
# SELECT * FROM _table1__jobs UNION SELECT * FROM _table2__jobs ...
584+
# Iterate over all jobs tables
585+
for jobs_table in schema.jobs:
586+
print(f"{jobs_table.target.table_name}: {jobs_table.progress()}")
587+
588+
# Query all errors across the schema
589+
all_errors = [job for jt in schema.jobs for job in jt.errors.fetch(as_dict=True)]
590+
591+
# Refresh all jobs tables
592+
for jobs_table in schema.jobs:
593+
jobs_table.refresh()
555594
```
556595

596+
This replaces the legacy single `~jobs` table with direct access to per-table jobs.
597+
557598
## Future Extensions
558599

559600
- [ ] Web-based dashboard for job monitoring

0 commit comments

Comments
 (0)