[DRAFT] User collections: integer range #393

hadia206 · 2025-07-15T00:25:36Z

No description provided.

knassre-bodo · 2025-07-15T18:49:24Z

pydough/qdag/collections/user_collections.py

+__all__ = ["PyDoughUserGeneratedCollection"]
+
+
+class PyDoughUserGeneratedCollection(ABC):


Let's move this + range_collection.py into the user_collections module since they will be referenced by multiple different IRs, and they don't really "belong" to any of them (kinda like metadata).

knassre-bodo · 2025-07-15T18:50:01Z

pydough/qdag/collections/user_collections.py

+    def __eq__(self, other) -> bool:
+        return isinstance(other, PyDoughUserGeneratedCollection) and repr(self) == repr(
+            other
+        )


Let's just do self.equals(other) and define an equals abstractmethod for each user collection.

knassre-bodo · 2025-07-15T18:51:10Z

pydough/qdag/collections/range_collection.py

+class RangeGeneratedCollection(PyDoughUserGeneratedCollection):
+    """Integer range-based collection."""
+
+    # HA_Q: should start/end/step be int or PyDoughQDAG? Why?


I say int | None since this class will transcend all the IRs

knassre-bodo · 2025-07-15T18:53:42Z

pydough/qdag/collections/range_collection.py

+        self.start = start
+        self.end = end
+        self.step = step


NIT: store all of these as self._start etc. and have @property to access them.

Also, let's add self._range = range(start, end, step) so the to_string can be: RangeCollection({self.name}!r, {self.column_name}={self.range})

knassre-bodo · 2025-07-15T18:54:41Z

pydough/qdag/collections/user_collections.py

+    @abstractmethod
+    def is_empty(self) -> bool:
+        """Check if the collection is empty."""


May want to flip this to always_non_empty, due to the usage

knassre-bodo · 2025-07-15T19:16:47Z

pydough/qdag/collections/user_collection_qdag.py

+        if term_name not in self.collection.columns:
+            raise PyDoughQDAGException(self.name_mismatch_error(term_name))
+
+        return Reference(self._ancestor, term_name)


I think it should be Reference(self, term_name)?

knassre-bodo · 2025-07-15T19:17:03Z

pydough/qdag/collections/user_collection_qdag.py

+    def get_expression_position(self, expr_name: str) -> int:
+        raise PyDoughQDAGException(f"Cannot call get_expression_position on {self!r}")


Yeah you can, use the ordering from the collection to figure out the order of the columns

knassre-bodo · 2025-07-15T19:43:18Z

pydough/qdag/collections/user_collection_qdag.py

+        Returns a string representation of the collection in a standalone form.
+        This is used for debugging and logging purposes.
+        """
+        return f"UserGeneratedCollection[{self.name}, {', '.join(self.collection.columns)}]"


Let's just use a to_string for the collection

knassre-bodo · 2025-07-15T19:43:40Z

pydough/qdag/collections/user_collection_qdag.py

+    def to_string(self) -> str:
+        # Stringify as "name(column_name)
+        return f"{self.name}({', '.join(self.collection.columns)})"


Again lets use the to_string for the collection to help out here

knassre-bodo · 2025-07-15T19:43:50Z

pydough/qdag/collections/user_collection_qdag.py

+    def tree_item_string(self) -> str:
+        return f"UserGeneratedCollection[{self.name}: {', '.join(self.collection.columns)}]"


Same point about the to_string for the collection

…update_reference

for more information, see https://pre-commit.ci

knassre-bodo · 2025-07-18T20:00:48Z

pydough/conversion/hybrid_tree.py

+            case HybridUserGeneratedCollection():
+                # User-generated collections are always guaranteed to exist.
+                pass


No they aren't (what if the range is empty?) This is why we need the always exists field for the generated colleciton.

knassre-bodo · 2025-07-18T20:01:08Z

pydough/conversion/hybrid_tree.py

+            # HA TODO: confirm is that right?
+            case HybridUserGeneratedCollection():
+                # User-generated collections are always guaranteed to be
+                # singular.
+                pass


Definitely not. It is only singular if we can guarantee it has <=1 rows.

knassre-bodo · 2025-07-25T22:11:54Z

pydough/sqlglot/sqlglot_relational_visitor.py

+    def visit_generated_table(self, generated_table: "GeneratedTable") -> None:
+        """convert the `GeneratedTable` to SQL code based on which underlying
+        `PyDoughUserGeneratedCollection` it uses
+
+        Args:
+            generated_table: The generated table node to visit.
+
+        """


None of the logic currently inside this function should be happening inside the function. This function should just call a new method inside self.bindings t deal with generated tables. That method should case on the type of generated table and call an appropriate method (for now, just one helper method for range). Everything currently inside this function should go inside that helper method for range.

Also, there shouldn't be a docstring for this method since there is one for the base class.

knassre-bodo · 2025-07-25T22:13:54Z

pydough/qdag/collections/user_collection_qdag.py

+        """
+        Returns a string representation of the collection in a standalone form.
+        This is used for debugging and logging purposes.
+        """


We shouldn't have this docstring since it is already in the parent classes.

knassre-bodo · 2025-07-25T22:17:24Z

pydough/qdag/collections/user_collection_qdag.py

+    @property
+    def unique_terms(self) -> list[str]:
+        return self.collection.columns


Lets make unique_column_names a property of PyDoughUserGeneratedCollection, that way this property only needs to return self.collection.unique_column_names

knassre-bodo · 2025-07-25T22:21:01Z

pydough/relational/relational_nodes/generated_table.py

+    def node_equals(self, other: RelationalNode) -> bool:
+        return isinstance(other, GeneratedTable) and self.name == other.name


We should be checking if self.collection == other.collection.

knassre-bodo · 2025-07-25T22:22:15Z

pydough/relational/relational_nodes/generated_table.py

+        visitor.visit_generated_table(self)
+
+    def to_string(self, compact=False) -> str:
+        return f"GENERATED_TABLE(table={self.name}, columns={self.make_column_string(self.columns, compact)})"


This is not a very useful string representation for the generated plans. At this point, we don't really care about the names (those are mostly used for QDAG to identify which ancestors you are talking about). Instead, let's stringify it as f"GENERATED_TABLE({self.collection})

knassre-bodo · 2025-07-25T22:23:39Z

pydough/conversion/hybrid_operations.py

+    def __repr__(self):
+        return f"USER_GEN_COLLECTION[{self.user_collection.name}]"


The name doesn't matter here, let's stringify using self.user_collection.to_string()

knassre-bodo · 2025-07-25T22:25:46Z

pydough/qdag/collections/user_collection_qdag.py

+    def to_string(self) -> str:
+        return self.collection.to_string()
+
+    @property
+    def tree_item_string(self) -> str:
+        return self.to_string()


These two need to be a bit different for QDAG:

to_string() should return a string that looks like the PyDough code that would generate this QDAG node once qualified.

tree_item_string should return a more compact/clean form that goes inside the tree diagrams from the qualification tests (so what self.collection.to_string() is currently doing)

knassre-bodo · 2025-07-25T22:27:21Z

pydough/user_collections/range_collection.py

+
+    def to_string(self) -> str:
+        """Return a string representation of the range collection."""
+        return f"RangeCollection({self.name}!r, {self.columns[0]}={self.range})"


The !r is supposed to go inside the {}. Right now if you look at the strings inside test_qualification.py they look a bit off because of this.

knassre-bodo · 2025-07-25T22:27:40Z

pydough/user_collections/range_collection.py

+        if not isinstance(other, RangeGeneratedCollection):
+            return False


Can just make isinstance(other, RangeGeneratedCollection) the first condition in the next line

knassre-bodo · 2025-07-25T22:28:47Z

pydough/user_collections/user_collections.py

+    @property
+    @abstractmethod
+    def data(self) -> Any:
+        """Return information about data in the collection."""


Why is this a method in the base class / at all? The subclasses will have no common idea of what "data" is.

knassre-bodo reviewed Jul 15, 2025

View reviewed changes

hadia206 added 6 commits July 15, 2025 16:49

attempt 1

499c000

add collection back

c192a6e

Merge branch 'main' of https://github.com/bodo-ai/PyDough into Hadia/…

77f63d1

…update_reference

range collections base, initial try

721eecc

add range_collection to pydough top

4a53e01

add test

d0ce87f

hadia206 force-pushed the Hadia/user_collections_range branch from 207f36b to d0ce87f Compare July 16, 2025 17:55

hadia206 changed the base branch from main to Hadia/update_reference July 16, 2025 17:56

pre-commit-ci bot and others added 6 commits July 16, 2025 17:57

[pre-commit.ci] auto fixes from pre-commit.com hooks

3c5ed66

for more information, see https://pre-commit.ci

address comments

8893d79

merge conflict

a7469d3

update test

9e66de2

make range inside UnqualifiedGeneratedCollection

025b556

[run CI] hybrid and execute

29e1e74

knassre-bodo reviewed Jul 18, 2025

View reviewed changes

hadia206 and others added 5 commits July 18, 2025 15:35

fix uniqueness, singular, always exists, and add another test

dc3c686

[run CI] more tests and fix empty table

a626dc1

add other tests (skipped as they're not passing)

8491cc6

Fixing test bugs

6315f3d

Fixing hybrid/qualification/conversion bugs

d4230e4

knassre-bodo reviewed Jul 25, 2025

View reviewed changes

Base automatically changed from Hadia/update_reference to main August 12, 2025 17:06

		__all__ = ["PyDoughUserGeneratedCollection"]


		class PyDoughUserGeneratedCollection(ABC):

		def get_expression_position(self, expr_name: str) -> int:
		raise PyDoughQDAGException(f"Cannot call get_expression_position on {self!r}")

		def tree_item_string(self) -> str:
		return f"UserGeneratedCollection[{self.name}: {', '.join(self.collection.columns)}]"

		def node_equals(self, other: RelationalNode) -> bool:
		return isinstance(other, GeneratedTable) and self.name == other.name

		def __repr__(self):
		return f"USER_GEN_COLLECTION[{self.user_collection.name}]"

		if not isinstance(other, RangeGeneratedCollection):
		return False

[DRAFT] User collections: integer range #393

Are you sure you want to change the base?

[DRAFT] User collections: integer range #393

Uh oh!

Conversation

hadia206 commented Jul 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knassre-bodo Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knassre-bodo Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knassre-bodo Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knassre-bodo Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

knassre-bodo Jul 15, 2025 •

edited

Loading

knassre-bodo Jul 18, 2025 •

edited

Loading

knassre-bodo Jul 25, 2025 •

edited

Loading

knassre-bodo Jul 25, 2025 •

edited

Loading