V16.3 (#129)

* parsing bugfixes

* - Updated documentation on new preferred method of interpolation using `@eval` and `$`
- Added documentation on using other macros inside of TidierData macros

Author: kdpsingh

NEWS.md

@@ -1,7 +1,10 @@
 # TidierData.jl updates
 
-## v16.3
-- Bugfix: `@summary` no longer errors with non-numeric columns. Instead, it only reports non-numeric summary stats on non-numeric columns. Minor changes to summary column names to be lowercase and snakecase.
+## v0.16.3 - 2024-12-28
+- Bugfix: `@summary` no longer errors with non-numeric columns. Instead, it only reports non-numeric summary stats on non-numeric columns. Minor changes to summary column names to be snake_case.
+- Bugfix: Reverted a bug introduced in v0.13.4, which escaped all macros. Now, string macros remain escaped (i.e., keeping it possible to work with Unitful units, e.g. `u"psi"`), but other macros are *not* escaped to allow for those macros to refer to column names within arguments.
+- Updated documentation on new preferred method of interpolation using `@eval` and `$`
+- Added documentation on using other macros inside of TidierData macros
 
 ## v0.16.2 - 2024-09-03
 - Bugfix: `@slice_min` and `@slice_max` respect the `n` argument

docs/Project.toml

@@ -1,12 +1,13 @@
 [deps]
+BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
 CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
 Chain = "8be319e6-bccf-4806-a6f7-6fae938471bc"
 DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
 DocumenterMarkdown = "997ab1e6-3595-5248-9280-8efb232c3433"
 Literate = "98b081ad-f1c9-55d3-8b20-4c87d4299306"
-Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
 RDatasets = "ce6b1742-4840-55fa-b093-852dadbb1d8b"
+Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
 StableRNGs = "860ef19b-820b-49d6-a774-d7a799459cd3"
 TidierData = "fe2206b3-d496-4ee9-a338-6a095c4ece80"
-BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
+Unitful = "1986cc42-f94f-5a68-af5c-568840ba703d"

docs/examples/UserGuide/interpolation.jl

@@ -1,3 +1,74 @@
+# ## Native (and preferred) method of interpolating using `@eval` and `$`
+
+# TidierData relies on "non-standard evaluation," which has the side effect of making interpolation slightly more complicated. For example, in the expression `@mutate(df, a = b + 1)`, the `df` refers to a data frame, while `a` and `b` refer to column names within the data frame. What would happen if you created a variable `var` that contains the value `:a`. Would this interpolated expression work?
+
+# ```julia
+# using TidierData
+# df = DataFrame(a = 1:5, b = 6:10)
+# 
+# var = :a
+# @mutate(df, $var = b + 1)
+# ```
+
+# Unfortunately, this does not work because it produces `@mutate(df, :a = b + 1)`. Since TidierData uses bare variables (and not symbols) to refer to column names, this will result in an error. However, there is a slight modification we can apply to make this code work: prefixing it with an `@eval`.
+
+using TidierData
+df = DataFrame(a = 1:5, b = 6:10, c = 11:15)
+
+var = :a
+@eval @mutate(df, $var = b + 1)
+
+# ### Why does adding an `@eval` to the beginning of the expression make interpolation work?
+
+# Adding `@eval` to the beginning causes the interpolated expressions to be evaluated prior to be interpolated. So `$var`, which contains the value `:a`, is evaluated to `a`, which produces the desired expression `@mutate(df, a = b + 1)`. The need of `@eval` here then is primarily because TidierData expects an `a` rather than an `:a` to refer to the column "a" in a data frame.
+
+# ### How can I use `@eval` with a chained set of expressions?
+
+# The answer is simple: use `@eval @chain` instead of `@chain`.
+
+var = :a
+
+@eval @chain df begin
+  @select($var)
+  @mutate($var = $var + 1)
+end
+
+# If you want to select multiple variables, just use a `...` to splat the vector (or tuple) of variables.
+
+vars = [:a, :b]
+
+@eval @chain df begin
+  @select($vars...)
+end
+
+# The `@eval`-based interpolation syntax is highly flexible in that it should work anywhere you might need it across the entire package.
+
+@eval @chain df begin
+  @summarize(across($vars..., mean))
+end
+
+# ### Does `@eval` work inside of user-defined functions?
+
+# Yes. Here's an example of how you could roll up a new `select_new` function wrapping the `@select` macros.
+
+function select_new(df, columns...)
+  @eval @select(df, $columns...)
+end
+
+select_new(df, :a, :c)
+
+# Yes. Here's another example of an `add_one()` function that adds one to all numeric columns and returns the result in a new set of columns.
+
+function add_one(df)
+  @eval @mutate(df, across(where(is_number), x -> x .+ 1))
+end
+
+add_one(df)
+
+# ## Note: the below documentation is included here only for historical reasons. It will be removed in the future.
+
+# ## Superseded method of interpolating using the `!!` ("bang bang") operator
+
 # The `!!` ("bang bang") operator can be used to interpolate values of variables from the parent environment into your code. This operator is borrowed from the R `rlang` package. At some point, we may switch to using native Julia interpolation, but for a variety of reasons that introduce some complexity with native interpolation, we plan to continue to support `!!` interpolation.
 
 # To interpolate multiple variables, the `rlang` R package uses the `!!!` "triple bang" operator. However, in `TidierData.jl`, the `!!` "bang bang" operator can be used to interpolate either single or multiple values as shown in the examples below.

docs/examples/UserGuide/macros.jl

@@ -0,0 +1,47 @@
+# While TidierData relies heavily on macros, you may occasionally find yourself needing to use other macros *within* TidierData macros. This can result in errors that may be hard to interpret. This page is intended to demonstrate two common situations when working with macros: string macros and macros with columns as arguments.
+
+# ## String macros
+
+# You may not think of string macros as macros because we often work with them in the form of prefixes (or suffixes) attached to string literals, such as `prefix"string goes here"`. However, string macros are indeed macros, and thankfully, these should work without any modification.
+
+using TidierData
+
+# Let's add the `psi` unit to the `a` column using the Unitful package.
+
+using Unitful
+df = DataFrame(a = 1:5, b = 6:10)
+@chain df @mutate(a = a * u"psi")
+
+# It just works!
+
+# ## Macros with columns as arguments
+
+# Occasionally, you may want to work with macros that operate on columns of a data frame. You may want to apply syntax that looks like this:
+
+# ```julia
+# using Printf
+# df = DataFrame(a = [0.11, 0.21, 0.12, 0.22])
+# 
+# @chain df begin
+#     @mutate(a_label = @sprintf("Var = %.1f", a))
+# end
+# ```
+
+# However, this will not work! Why not? Well, there are a two reasons: it is difficult to escape a macro but not its arguments, and macros cannot be vectorized by adding a period to the end (unlike functions).
+
+# The easiest way to fix both issues is to wrap the macro inside of an anonymous function. Thus, `@example_macro(a)` turns into `(x -> example_macro(x))(a)`. What is happening here is that an anonymous function is being defined, and then that function is immediately being called with an argument `a` referring to the column name `a`. 
+
+# Here is what the looks like for `@sprintf`:
+
+using Printf
+df = DataFrame(a = [0.11, 0.21, 0.12, 0.22])
+
+@chain df begin
+    @mutate(a_label = (x -> @sprintf("Var = %.1f", x))(a))
+end
+
+# This works!
+
+# Even though TidierData cannot dot-vectorize `@sprintf`, it can vectorize the anonymous function in which `@sprintf` is wrapped, converting the expression to `a_label = (x -> @sprintf("Var = %.1f", x)).(a)` before it is run. Notice that TidierData adds a period before the `(a)` to vectorize the function before passing this expression to DataFrames.jl.
+
+# Lastly, one caveat to know is that the above anonymous wrapper function syntax currently only works for **macros** and *not* for functions. It should not be needed for functions, but sharing here for awareness.

docs/mkdocs.yml

@@ -138,6 +138,7 @@ nav:
   - "Column names": "examples/generated/UserGuide/column_names.md"
   - "Interpolation" : "examples/generated/UserGuide/interpolation.md"
   - "Auto-vectorization" : "examples/generated/UserGuide/autovec.md"
+  - "Using macros": "examples/generated/UserGuide/macros.md"
   # - "Benchmarking" : "examples/generated/UserGuide/benchmarking.md"
   - "Comparison to DF.jl" : "examples/generated/UserGuide/comparisons.md"
   - "Contribute" : "examples/generated/Contributors/Howto.md"

src/parsing.jl

@@ -428,7 +428,13 @@ function parse_escape_function(rhs_expr::Union{Expr,Symbol})
         return x
       end
     elseif @capture(x, @mac_(args__))
-      return esc(Expr(:macrocall, mac, LineNumberNode, args...))
+      if endswith(string(mac), "_str")
+        # Macros used inside of string macros are escaped, making it possible to work with Unitful units inside of `@mutate` (e.g. `u"psi"`)
+        return esc(Expr(:macrocall, mac, LineNumberNode, args...))
+      else
+        # Other macros that may reference variables referring to column names should *not* be escaped
+        return x
+      end
     end
     return x
   end