Skip to content

Conversation

@ion-elgreco
Copy link
Contributor

@ion-elgreco ion-elgreco commented Oct 12, 2024

Which issue does this PR close?

Rationale for this change

Gives more flexibility to the user.

What changes are included in this PR?

Adds a with_columns which is a wrapper around with_column in Rust, the schema_name of the expr is used as the name. However I think we could improve over this in a follow up, since we should keep the original name unless somewhere down the expr path alias was called.

See this example of Polars VS Datafusion now:

original df

+---+---+---+
| a | b | c |
+---+---+---+
| 1 | 4 | 8 |
| 2 | 5 | 5 |
| 3 | 6 | 8 |
+---+---+---+

Polars

df.with_columns(pl.col('a)+5)
┌─────┬─────┬─────┐
│ abc   │
│ --------- │
│ i64i64i64 │
╞═════╪═════╪═════╡
│ 648   │
│ 755   │
│ 868   │
└─────┴─────┴─────┘

Datafusion

df.with_columns(column('a')+5)
+---+---+---+--------------+
| a | b | c | a + Int64(5) |
+---+---+---+--------------+
| 1 | 4 | 8 | 6            |
| 2 | 5 | 5 | 7            |
| 3 | 6 | 8 | 8            |
+---+---+---+--------------+

Are there any user-facing changes?

Adds new with_columns method on DataFrame.

Copy link
Member

@timsaucer timsaucer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great addition!

Comment on lines 171 to 179
Example usage:
The following will add 4 columns labeled a, b, c, and d.
df = df.with_columns(
lit(0).alias('a'),
[lit(1).alias('b'), lit(2).alias('c')],
d=lit(3)
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know if this renders right in the online doc? If not, I can pull your branch and test it out

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@timsaucer I am not sure, is it possible to locally build the docs?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there is a build script in the docs folder and then you can open the generated html files locally.

@timsaucer timsaucer merged commit 7cca028 into apache:main Oct 18, 2024
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace with_column with with_columns

2 participants