Ak/spec #372

anupkalburgi · 2025-11-04T14:28:33Z

Changes

Linked issues

Resolves #..

Requirements

manually tested
updated documentation
updated demos
updated tests

codecov · 2025-11-06T16:14:20Z

Codecov Report

❌ Patch coverage is 53.44262% with 142 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (master@3aca7f3). Learn more about missing BASE report.
⚠️ Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
dbldatagen/spec/generator_spec_impl.py	0.00%	111 Missing ⚠️
dbldatagen/spec/generator_spec.py	84.21%	13 Missing and 2 partials ⚠️
dbldatagen/spec/__init__.py	28.57%	10 Missing ⚠️
dbldatagen/spec/output_targets.py	85.18%	3 Missing and 1 partial ⚠️
dbldatagen/spec/compat.py	60.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##             master     #372   +/-   ##
=========================================
  Coverage          ?   89.41%           
=========================================
  Files             ?       53           
  Lines             ?     4450           
  Branches          ?      809           
=========================================
  Hits              ?     3979           
  Misses            ?      325           
  Partials          ?      146

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

dbldatagen/spec/generator_spec.py

dbldatagen/spec/generator_spec_impl.py

Copilot

Pull request overview

This PR introduces a new Pydantic-based specification API for dbldatagen, providing a declarative, type-safe approach to synthetic data generation. The changes add comprehensive validation, test coverage, and example specifications while updating documentation and build configuration to support both Pydantic V1 and V2.

Key Changes:

New spec-based API with Pydantic models for defining data generation configurations
Comprehensive validation framework with error collection and reporting
Pydantic V1/V2 compatibility layer for broad environment support

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_specs.py	Comprehensive test suite for ValidationResult, ColumnDefinition, DatagenSpec validation, and target configurations
tests/test_datasets_with_specs.py	Tests for Pydantic model validation with BasicUser and BasicStockTicker specifications
tests/test_datagen_specs.py	Tests for DatagenSpec creation, validation, and generator options
pyproject.toml	Added ipython dependency, test matrix for Pydantic versions, and disabled warn_unused_ignores
makefile	Updated to use Pydantic version-specific test environments and removed .venv target
examples/datagen_from_specs/basic_user_datagen_spec.py	Example DatagenSpec factory for generating basic user data with pre-configured specs
examples/datagen_from_specs/basic_stock_ticker_datagen_spec.py	Complex example with OHLC stock data generation including time-series and volatility modeling
examples/datagen_from_specs/README.md	Documentation for Pydantic-based dataset specifications with usage examples
dbldatagen/spec/validation.py	ValidationResult class for collecting and reporting validation errors and warnings
dbldatagen/spec/output_targets.py	Pydantic models for UCSchemaTarget and FilePathTarget output destinations
dbldatagen/spec/generator_spec_impl.py	Generator class implementing the spec-to-DataFrame conversion logic
dbldatagen/spec/generator_spec.py	Core DatagenSpec and TableDefinition models with comprehensive validation
dbldatagen/spec/compat.py	Pydantic V1/V2 compatibility layer enabling cross-version support
dbldatagen/spec/column_spec.py	ColumnDefinition model with validation for primary keys and constraints
dbldatagen/spec/init.py	Module initialization with lazy imports to avoid heavy dependencies
README.md	Updated feature list and formatting to mention new Pydantic-based API
CHANGELOG.md	Added entry for Pydantic-based specification API feature

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/test_specs.py

examples/datagen_from_specs/basic_stock_ticker_datagen_spec.py

dbldatagen/spec/generator_spec_impl.py

makefile

dbldatagen/spec/__init__.py

dbldatagen/spec/column_spec.py

ghanse · 2025-12-04T19:40:25Z

dbldatagen/spec/generator_spec_impl.py

+            conflicting_opts_for_pk = [
+                "distribution", "template", "dataRange", "random", "omit",
+                "min", "max", "uniqueValues", "values", "expr"
+            ]
+
+            for opt_key in conflicting_opts_for_pk:
+                if opt_key in kwargs:
+                    logger.warning(
+                        f"Primary key '{col_name}': Option '{opt_key}' may be ignored")


Why do we disallow min, max, and dataRange for primary keys?

If the primary key has a min, max it might end up conflicting with num_of_rows. Eg if the number of rows is 100, and min is 10, max is 50.

ghanse · 2025-12-04T19:48:31Z

dbldatagen/spec/generator_spec_impl.py

+
+                # Process each column
+                for col_def in table_spec.columns:
+                    kwargs = self._columnSpecToDatagenColumnSpec(col_def)


Maybe use column_options instead to avoid any conflicts.

What is conflict you are referring to here?

dbldatagen/spec/generator_spec_impl.py

ghanse · 2025-12-04T19:58:35Z

dbldatagen/spec/generator_spec_impl.py

+                # Write data based on destination type
+                if isinstance(output_destination, FilePathTarget):
+                    output_path = posixpath.join(output_destination.base_path, table_name)
+                    df.write.format(output_destination.output_format).mode("overwrite").save(output_path)
+                    logger.info(f"Wrote table '{table_name}' to file path: {output_path}")
+
+                elif isinstance(output_destination, UCSchemaTarget):
+                    output_table = f"{output_destination.catalog}.{output_destination.schema_}.{table_name}"
+                    df.write.mode("overwrite").saveAsTable(output_table)
+                    logger.info(f"Wrote table '{table_name}' to Unity Catalog: {output_table}")


We should use utils.write_data_to_output for this.

The method in utils relies on OutputDataset. See the other comments above.

ghanse · 2025-12-04T20:00:33Z

dbldatagen/spec/output_targets.py

We have OutputDataset in config.py. I think we can reuse it here instead of creating new classes?

imo, The output target in spec would be better suited for the spec as it does validations for UC and file paths volumes. This config.OutputDataset is generic, and getting specific errors would mean essientially copying this over to the new spec folder or other way round. Is there something I am missing here ?

* added use of ABC to mark TextGenerator as abstract * Lint text generators module * Add persistence methods * Add tests and docs; Update PR template * Update hatch installation for push action * Refactor * Update method names and signatures --------- Co-authored-by: ronanstokes-db <[email protected]> Co-authored-by: Ronan Stokes <[email protected]>

* Add files via upload Adding an example notebook on how to leverage DBLDATAGEN in the creation of oil and gas datasets/. Currently placing this in the examples/notebooks file * Update and rename [DBLDATAGEN]Oil&GasWellHeaderDailyProductionTypeCurve.py to oil_gas_data_generation.py implemented changes Greg suggested aorund function formatting * Update oil_gas_data_generation.py added documentaiton link and install clarification --------- Co-authored-by: Greg Hansen <[email protected]>

* Add files via upload A notebook that generates synthetic log in data for various gamers. The gamers can have multiple devices that are consistent across them and are located in different areas. * Update VideoGameLoginSyntheticDataGeneration.py Fixed code with comments from PR * Update and rename VideoGameLoginSyntheticDataGeneration.py to gaming_data_generation.py

* Format data_analyzer.py and text_generator_plugins.py * Update formatting and fix docstrings * Update inferred max value for DecimalType columns * Update docstrings and inferred max value for DecimalType columns

* Added CPG supply chain dbldatagen notebook * Deleting original CPG_supply_chain_datagen.py notebook * Adding new file with the changes Adding new file in the correct location in the repo. * Cleaned up comments --------- Co-authored-by: Greg Hansen <[email protected]>

akshayamin · 2025-12-10T22:30:13Z

examples/datagen_from_specs/basic_user_datagen_spec.py

 """

-from dbldatagen.spec.generator_spec import DatagenSpec, TableDefinition
+from dbldatagen.spec.generator_spec import DatagenSpec, DatasetDefinition


Looks like the param usage on line# 87 is from a previous version of the class? Would it be "datasets =" instead of "tables ="?

spec = DatagenSpec(
tables={"users": table_def},

akshayamin · 2025-12-10T22:31:29Z

examples/datagen_from_specs/basic_stock_ticker_datagen_spec.py

+import random

-from dbldatagen.spec.generator_spec import DatagenSpec, TableDefinition
+from dbldatagen.spec.generator_spec import DatagenSpec, DatasetDefinition


same here as the other example:

Looks like the param usage on line# 275 is from a previous version of the class? Would it be "datasets =" instead of "tables ="?

spec = DatagenSpec(
tables={"users": table_def},

akshayamin · 2025-12-11T02:32:01Z

dbldatagen/spec/generator_spec_impl.py

+                for col_def in table_spec.columns:
+                    kwargs = _columnSpecToDatagenColumnSpec(col_def)
+                    data_gen = data_gen.withColumn(colName=col_def.name, **kwargs)
+                    # Has performance implications.


convert to TODO? What are the performance implications here ?

anupkalburgi added 4 commits September 23, 2025 12:16

working test wireup

20db964

Initial code, spec and test, pushing for review

4e9cba5

fixing tests

d37de68

changes to make file

f5214ca

anupkalburgi force-pushed the ak/spec branch from 47e80f1 to f5214ca Compare November 6, 2025 15:55

updating docs

0a8fa2a

anupkalburgi commented Nov 10, 2025

View reviewed changes

dbldatagen/spec/generator_spec.py Outdated Show resolved Hide resolved

anupkalburgi commented Nov 10, 2025

View reviewed changes

dbldatagen/spec/generator_spec.py Outdated Show resolved Hide resolved

anupkalburgi commented Nov 10, 2025

View reviewed changes

dbldatagen/spec/generator_spec_impl.py Outdated Show resolved Hide resolved

anupkalburgi commented Nov 10, 2025

View reviewed changes

dbldatagen/spec/generator_spec_impl.py Outdated Show resolved Hide resolved

anupkalburgi added 3 commits November 17, 2025 13:56

fixing tests, removing the solved todos, targets to a diff module

61f676d

converting to camelCase

e139c8b

validation into a diff module

52a4283

anupkalburgi marked this pull request as ready for review December 2, 2025 14:31

anupkalburgi requested review from a team as code owners December 2, 2025 14:31

anupkalburgi requested review from nfx and renardeinside and removed request for a team December 2, 2025 14:31

removing compat/scratch notes

a0ce13b

anupkalburgi force-pushed the ak/spec branch from 1b90b6a to a0ce13b Compare December 2, 2025 14:56

marking the spec module experimental

f6c9a69

ghanse requested a review from Copilot December 4, 2025 18:04

Copilot AI reviewed Dec 4, 2025

View reviewed changes

tests/test_specs.py Outdated Show resolved Hide resolved

examples/datagen_from_specs/basic_stock_ticker_datagen_spec.py Outdated Show resolved Hide resolved

dbldatagen/spec/generator_spec_impl.py Outdated Show resolved Hide resolved

makefile Show resolved Hide resolved

ghanse requested changes Dec 4, 2025

View reviewed changes

anupkalburgi requested a review from akshayamin December 5, 2025 14:00

ghanse and others added 2 commits December 10, 2025 14:36

Format modules (#367)

de1bb75

brettaulbaugh-db and others added 6 commits December 10, 2025 14:36

Format data_analyzer and text_generation_plugins (#370)

7cb3789

* Format data_analyzer.py and text_generator_plugins.py * Update formatting and fix docstrings * Update inferred max value for DecimalType columns * Update docstrings and inferred max value for DecimalType columns

renaming the keys and falling back to absolute imports

da8bc16

making methods static, and fixing tests

99d5526

akshayamin reviewed Dec 10, 2025

View reviewed changes

akshayamin reviewed Dec 11, 2025

View reviewed changes

Ak/spec #372

Are you sure you want to change the base?

Ak/spec #372

Uh oh!

Conversation

anupkalburgi commented Nov 4, 2025

Changes

Linked issues

Requirements

Uh oh!

codecov bot commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

codecov bot commented Nov 6, 2025 •

edited

Loading