Skip to content

Commit 9dec7b2

Browse files
Alex Richeyalexrichey
authored andcommitted
Add ProcessingSummary for all processors
1 parent 9be17c9 commit 9dec7b2

File tree

4 files changed

+331
-98
lines changed

4 files changed

+331
-98
lines changed

dcpy/lifecycle/ingest/run.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,7 @@ def ingest(
7878
output_filename=init_parquet,
7979
)
8080

81-
transform.process(
81+
config.ingestion.processing_steps_summaries = transform.process(
8282
config.id,
8383
config.ingestion.processing_steps,
8484
config.columns,
@@ -87,6 +87,9 @@ def ingest(
8787
output_csv=output_csv,
8888
)
8989

90+
with open(dataset_staging_dir / CONFIG_FILENAME, "w") as f:
91+
json.dump(config.model_dump(mode="json"), f, indent=4)
92+
9093
dataset_output_dir = ingest_output_dir / dataset_id / config.version
9194
dataset_output_dir.mkdir(parents=True, exist_ok=True)
9295
logger.info(f"Copying {config.filename} to {dataset_output_dir}")

0 commit comments

Comments
 (0)