refactor: port Dataset to TypeBase. #30918

asukaminato0721 · 2026-01-13T10:43:25Z

Important

Make sure you have read our contribution guidelines
Ensure there is an associated issue and you have been assigned to it
Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

Screenshots

Before	After
...	...

Checklist

This change requires a documentation update, included: Dify Document
I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
I've updated the documentation accordingly.
I ran make lint and make type-check (backend) and cd web && npx lint-staged (frontend) to appease the lint gods

gemini-code-assist · 2026-01-13T10:43:42Z

Summary of Changes

Hello @asukaminato0721, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refactors the Dataset model by updating its base class to TypeBase and introducing comprehensive type annotations for all its attributes. This change aims to improve the maintainability and type safety of the model definition. Additionally, the process of creating new Dataset instances has been streamlined for better code readability.

Highlights

Model Inheritance Update: The Dataset model now inherits from TypeBase instead of Base, aligning with modern SQLAlchemy practices for improved type safety and consistency.
Comprehensive Type Hinting: Explicit type hints using Mapped have been added to all column definitions within the Dataset model, enhancing code readability and enabling static analysis.
Streamlined Default Value Assignment: Default values for several columns in the Dataset model (e.g., keyword_number, built_in_field_enabled, runtime_mode, enable_api) are now defined directly within their mapped_column declarations.
Refactored Dataset Creation: The create_empty_dataset function has been updated to initialize Dataset objects more concisely by passing all relevant attributes directly to the constructor, eliminating subsequent individual assignments.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request refactors the Dataset model to inherit from TypeBase, which is a good improvement for type safety and consistency. The changes in dataset_service.py to use the Dataset constructor with all parameters are clean. However, I've found a few incorrect type hints in the Dataset model in api/models/dataset.py that should be corrected to avoid potential runtime issues and to ensure type consistency.

api/models/dataset.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Copilot

Pull request overview

This pull request refactors the Dataset model to use TypeBase (which includes MappedAsDataclass) instead of Base, as part of a broader migration effort (#26788). The changes include:

Changes:

Migrated the Dataset model from Base to TypeBase to enable dataclass-style instantiation
Added proper type annotations to all Dataset model fields
Refactored Dataset creation in create_empty_dataset to use constructor parameters instead of attribute assignment
Added default values for nullable and optional fields

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.

File	Description
api/models/dataset.py	Migrated Dataset class from Base to TypeBase and updated all field definitions with type annotations and defaults
api/services/dataset_service.py	Refactored create_empty_dataset to use constructor-based initialization instead of post-creation attribute assignment

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-14T14:55:52Z

api/models/dataset.py

+    updated_at: Mapped[datetime] = mapped_column(
        sa.DateTime, nullable=False, server_default=func.current_timestamp(), onupdate=func.current_timestamp()
    )


When using TypeBase with MappedAsDataclass, auto-generated timestamp fields should have init=False to exclude them from the dataclass constructor, as they are updated by the database. This prevents requiring these values in the constructor.

Copilot · 2026-01-14T14:55:52Z

api/models/dataset.py

    INDEXING_TECHNIQUE_LIST = ["high_quality", "economy", None]
    PROVIDER_LIST = ["vendor", "external", None]

    id: Mapped[str] = mapped_column(StringUUID, default=lambda: str(uuid4()))


When using TypeBase with MappedAsDataclass, the id field should include init=False to exclude it from the dataclass constructor, as it's auto-generated. Following the pattern used in other TypeBase models like Pipeline, it should also use default_factory=lambda: str(uuid4()) instead of just default to properly generate unique IDs for each instance.

Copilot · 2026-01-14T14:55:52Z

api/models/dataset.py

    provider: Mapped[str] = mapped_column(String(255), server_default=sa.text("'vendor'"))
    permission: Mapped[str] = mapped_column(String(255), server_default=sa.text("'only_me'"))
-    data_source_type = mapped_column(String(255))
+    data_source_type: Mapped[str] = mapped_column(String(255))


The type annotation for data_source_type is inconsistent with the database schema. The database migration shows this column is nullable (nullable=True), but the type annotation is Mapped[str] which indicates it's required when using TypeBase with MappedAsDataclass. This will cause runtime errors when creating Dataset instances without providing data_source_type (as seen in create_empty_dataset and other methods). The type should be Mapped[str | None] to match the database schema and allow it to be optional in the constructor.

Copilot · 2026-01-14T14:55:53Z

api/models/dataset.py

    name: Mapped[str] = mapped_column(String(255))
-    description = mapped_column(LongText, nullable=True)
+    description: Mapped[str | None] = mapped_column(LongText, nullable=True)
    provider: Mapped[str] = mapped_column(String(255), server_default=sa.text("'vendor'"))


When using TypeBase with MappedAsDataclass, fields with server_default also need a Python-side default parameter for the dataclass constructor. The provider field should include default="vendor" to match the server_default and allow it to be optional in the constructor.

Copilot · 2026-01-14T14:55:53Z

api/models/dataset.py

-    description = mapped_column(LongText, nullable=True)
+    description: Mapped[str | None] = mapped_column(LongText, nullable=True)
    provider: Mapped[str] = mapped_column(String(255), server_default=sa.text("'vendor'"))
    permission: Mapped[str] = mapped_column(String(255), server_default=sa.text("'only_me'"))


When using TypeBase with MappedAsDataclass, fields with server_default also need a Python-side default parameter for the dataclass constructor. The permission field should include default="only_me" to match the server_default and allow it to be optional in the constructor.

Copilot · 2026-01-14T14:55:53Z

api/models/dataset.py

-    created_by = mapped_column(StringUUID, nullable=False)
+    index_struct: Mapped[str | None] = mapped_column(LongText, nullable=True)
+    created_by: Mapped[str] = mapped_column(StringUUID, nullable=False)
    created_at: Mapped[datetime] = mapped_column(DateTime, nullable=False, server_default=func.current_timestamp())


When using TypeBase with MappedAsDataclass, auto-generated timestamp fields should have init=False to exclude them from the dataclass constructor, as they are generated by the database. This prevents requiring these values in the constructor.

.

756b775

[autofix.ci] apply automated fixes

2cdc7c0

gemini-code-assist bot reviewed Jan 13, 2026

View reviewed changes

api/models/dataset.py Outdated Show resolved Hide resolved

api/models/dataset.py Outdated Show resolved Hide resolved

api/models/dataset.py Show resolved Hide resolved

asukaminato0721 and others added 5 commits January 13, 2026 19:53

Update api/models/dataset.py

112b73a

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update api/models/dataset.py

31125bb

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update api/models/dataset.py

e683509

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

.

b9a5f64

Merge branch 'main' into typebase-dataset

24ae11f

asukaminato0721 marked this pull request as ready for review January 14, 2026 14:50

asukaminato0721 requested a review from JohnJyong as a code owner January 14, 2026 14:50

Copilot AI review requested due to automatic review settings January 14, 2026 14:50

dosubot bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Jan 14, 2026

Merge branch 'main' into typebase-dataset

503b186

asukaminato0721 marked this pull request as draft January 14, 2026 14:50

Copilot started reviewing on behalf of asukaminato0721 January 14, 2026 14:50 View session

Copilot AI reviewed Jan 14, 2026

View reviewed changes

asukaminato0721 added 2 commits January 16, 2026 13:40

Merge branch 'main' into typebase-dataset

64e7f1d

Merge branch 'main' into typebase-dataset

0ead1bb

asukaminato0721 mentioned this pull request Jan 16, 2026

Dataset seems missing many fields, or have strange default? #31123

Open

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: port Dataset to TypeBase. #30918

refactor: port Dataset to TypeBase. #30918

Uh oh!

asukaminato0721 commented Jan 13, 2026

Uh oh!

gemini-code-assist bot commented Jan 13, 2026

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 14, 2026

Uh oh!

Copilot AI Jan 14, 2026

Uh oh!

Copilot AI Jan 14, 2026

Uh oh!

Copilot AI Jan 14, 2026

Uh oh!

Copilot AI Jan 14, 2026

Uh oh!

Copilot AI Jan 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

refactor: port Dataset to TypeBase. #30918

Are you sure you want to change the base?

refactor: port Dataset to TypeBase. #30918

Uh oh!

Conversation

asukaminato0721 commented Jan 13, 2026

Summary

Screenshots

Checklist

Uh oh!

gemini-code-assist bot commented Jan 13, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant