Add support for parameters via pf.Param(data) (#212)

* Main edit

* Fix tests

* Also update constraints

* Introduce Param function

* Shift examples to using Param

* Add support for parameters from files

* Remove unnecessary arguments in Param

* Improve documentation

* Fix annotations in Python 3.9

* Fix tests

* Fix broken references

* Improve testing

* Fix failing tests

* Increase test coverage

* Increase test coverage

Author: staadecker

docs/contribute/index.md

@@ -39,6 +39,9 @@ Pyoframe has several types of tests.
 
 4. Documentation tests (in `docs/`). All Python code blocks in the documentation are run to ensure the documentation doesn't become outdated. This is done using Sybil. Refer to the [Sybil documentation](https://sybil.readthedocs.io/en/latest/markdown.html#code-blocks) to learn how to create setup code or skip code blocks you don't wish to test.
 
+!!! warning "Non-breaking spaces"
+    Be aware that Pyoframe uses non-breaking spaces to improve the formatting of expressions. If your Sybil tests are unexpectedly failing, make sure that the expected output contains all the needed non-breaking spaces.
+
 ## Writing documentation
 
 You can preview the documentation website by running `mkdocs serve` and navigating to [`http://127.0.0.1:8000/pyoframe/`](http://127.0.0.1:8000/pyoframe/).

docs/examples/facility_location.md

@@ -26,12 +26,12 @@ model.customers = model.x_axis * model.y_axis  # (1)!
 
 
 model.facility_position = pf.Variable(model.facilities, model.axis, lb=0, ub=1)
-model.customer_position_x = pd.DataFrame(
+model.customer_position_x = pf.Param(
     {"x": range(G), "x_pos": [step / (G - 1) for step in range(G)]}
-).to_expr()
-model.customer_position_y = pd.DataFrame(
+)
+model.customer_position_y = pf.Param(
     {"y": range(G), "y_pos": [step / (G - 1) for step in range(G)]}
-).to_expr()
+)
 
 model.max_distance = pf.Variable(lb=0)
 

docs/learn/advanced-concepts/internals.md

@@ -1,7 +1,6 @@
 # Internal details
 
-Pyoframe's inner workings involve a few tricks that you should be aware of
-if you wish to modify Pyoframe's internal code.
+Pyoframe's inner workings involve a few tricks that you should be aware of if you wish to contribute to Pyoframe's code base.
 
 ## The zero variable
 
@@ -16,7 +15,9 @@ constant terms and also simplifies the [handling of quadratics](#quadratics).
 
 Internally, [Expression][pyoframe.Expression] is used to represent both linear and quadratic mathematical expressions. When a quadratic expression is formed, column `__quadratic_variable_id` is added to [Expression.data][pyoframe.Expression.data]. If an expression's quadratic terms happen to cancel out (e.g. `(ab + c) - ab`), this column is automatically removed.
 
-Column `__quadratic_variable_id` records the ID of the _second_ variable in a quadratic term (the `b` in `3ab`). For linear terms, which have no second variable, this column contains the [Zero Variable](#the-zero-variable). Quadratic terms are always stored such that the first term's variable ID (in column `__variable_id`) is greater or equal to the second term's variable id (in column `__quadratic_variable_id`). For example, `var_7 * var_8` would be rearranged and stored as `var_8 * var_7`. This helps simplify expressions and provides a useful guarantee: If the variable in the first column (`__variable_id`) is the Zero Variable (`var_0`) we know the variable in the second column must also be the Zero Variable and, thus, the term must be a constant.
+Column `__quadratic_variable_id` records the ID of the _second_ variable in a quadratic term (the `b` in `3ab`). For linear terms, which have no second variable, this column contains the [Zero Variable](#the-zero-variable).
+
+Quadratic terms are always stored such that the first term's variable ID (in column `__variable_id`) is greater or equal to the second term's variable id (in column `__quadratic_variable_id`). For example, `var_7 * var_8` would be rearranged and stored as `var_8 * var_7`. This helps simplify expressions and provides a useful guarantee: If the variable in the first column (`__variable_id`) is the Zero Variable (`var_0`) we know the variable in the second column must also be the Zero Variable and, thus, the term must be a constant.
 
 ## Division
 

docs/learn/concepts/addition.md

@@ -31,14 +31,13 @@ import pyoframe as pf
 import polars as pl
 
 air_data = pl.DataFrame({"flight_no": ["A4543", "K937"], "emissions": [1.4, 2.4]})
-ground_data = pl.DataFrame(
-    {"flight_number": ["A4543", "K937"], "emissions": [0.02, 0.05]}
-)
 
 model = pf.Model()
 model.Fly = pf.Variable(air_data["flight_no"], vtype="binary")
 model.air_emissions = model.Fly * air_data
-model.ground_emissions = ground_data.to_expr()
+model.ground_emissions = pf.Param(
+    {"flight_number": ["A4543", "K937"], "emissions": [0.02, 0.05]}
+)
 -->
 
 ```pycon
@@ -90,7 +89,7 @@ What we'd like to do is effectively 'copy' (aka. 'broadcast') `E_max` _over_ eve
 
 ```pycon
 >>> model.E_max.over("flight_no")
-<Expression terms=1 type=linear>
+<Expression (linear) terms=1>
 ┌───────────┬────────────┐
 │ flight_no ┆ expression │
 ╞═══════════╪════════════╡
@@ -104,7 +103,7 @@ Notice how applying `.over("flight_no")` added a dimension `flight_no` with valu
 ```pycon
 >>> model.emission_constraint = model.E_max.over("flight_no") >= model.flight_emissions
 >>> model.emission_constraint
-<Constraint 'emission_constraint' height=2 terms=6 type=linear>
+<Constraint 'emission_constraint' (linear) height=2 terms=6>
 ┌───────────┬───────────────────────────────┐
 │ flight_no ┆ constraint                    │
 │ (2)       ┆                               │
@@ -127,19 +126,16 @@ If one of the two expressions in an addition has extras labels not present in th
 import pyoframe as pf
 import polars as pl
 
-air_data = pl.DataFrame(
+model = pf.Model()
+model.air_emissions = pf.Param(
     {
         "flight_no": ["A4543", "K937", "D2082", "D8432", "D1206"],
         "emissions": [1.4, 2.4, 4, 7.6, 4],
     }
 )
-ground_data = pl.DataFrame(
+model.ground_emissions = pf.Param(
     {"flight_no": ["A4543", "K937", "B3420"], "emissions": [0.02, 0.05, 0.001]}
 )
-
-model = pf.Model()
-model.air_emissions = air_data.to_expr()
-model.ground_emissions = ground_data.to_expr()
 -->
 
 Consider again [example 1](#example-1-catching-a-mistake) where we added air emissions and ground emissions.
@@ -205,7 +201,7 @@ Option 2 hardly seems reasonable this time considering that air emissions make u
 
 ```pycon
 >>> model.air_emissions.keep_extras() + model.ground_emissions.drop_extras()
-<Expression height=5 terms=5 type=constant>
+<Expression (parameter) height=5 terms=5>
 ┌───────────┬────────────┐
 │ flight_no ┆ expression │
 │ (5)       ┆            │

docs/learn/concepts/special-functions.md

@@ -10,103 +10,3 @@ Pyoframe has a few special functions that make working with dataframes easy and
 
 ## `Expression.map()`
 
-
-## `DataFrame.to_expr()`
-
-!!! abstract "Summary"
-
-    [`pandas.DataFrame.to_expr()`](../../reference/external/pandas.DataFrame.to_expr.md) and [`polars.DataFrame.to_expr()`](../../reference/external/polars.DataFrame.to_expr.md) allow users to manually convert their DataFrames to Pyoframe [Expressions][pyoframe.Expression] when Pyoframe is unable to perform an automatic conversation.
-
-Pyoframe conveniently allows users to use [Polars DataFrames](https://docs.pola.rs/api/python/stable/reference/dataframe/index.html) and [Pandas DataFrames](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) in their mathematical expressions. To do so, Pyoframe automatically detects these DataFrames and converts them to Pyoframe [Expressions][pyoframe.Expression] whenever there is a mathematical operation (e.g., `*`, `-`, `+`) involving at least one Pyoframe object (e.g. [Variable][pyoframe.Variable], [Set][pyoframe.Set], [Expression][pyoframe.Expression], etc.).
-
-However, if **neither** the left or right terms of a mathematical operation is a Pyoframe object, Pyoframe will not automatically convert DataFrames[^2]. In these situations, users can manually convert their DataFrames to Pyoframe expressions using `.to_expr()`.
-
-Additionally, users should use `.to_expr()` whenever they wish to use [over][pyoframe.Expression.over], [drop_extras][pyoframe.Expression.drop_extras], or [keep_extras][pyoframe.Expression.keep_extras] on a DataFrame.
-
-!!! info "Under the hood"
-
-    How is `.to_expr()` a valid Pandas and Polars method? `import pyoframe` causes Pyoframe to [monkey patch](https://stackoverflow.com/questions/5626193/what-is-monkey-patching) the Pandas and Polars libraries. One of the patches adds the `.to_expr()` method to both `pandas.DataFrame` and `polars.DataFrame` (see [`monkey_patch.py`](https://github.com/Bravos-Power/pyoframe/tree/main/src/pyoframe)).
-
-!!! tip "Working with Pandas Series"
-
-    You can call `.to_expr()` on a Pandas Series to produce an expression where the labels will be determined from the Series' index.
-
-[^2]: After all, how could it? If a user decides to write code that adds two DataFrames together, Pyoframe shouldn't interfere.
-
-### Example
-
-Consider the following scenario where we have some population data on yearly births and deaths, as well as an immigration variable.
-
-```python
-import pyoframe as pf
-import pandas as pd
-
-population_data = pd.DataFrame(
-    dict(year=[2025, 2026], births=[1e6, 1.1e6], deaths=[-1.2e6, -1.4e6])
-)
-
-model = pf.Model()
-model.immigration = pf.Variable(dict(year=[2025, 2026]))
-```
-
-Now, saw we wanted an expression representing the total yearly population change. The following works just fine:
-
-```pycon
->>> (
-...     model.immigration
-...     + population_data[["year", "births"]]
-...     + population_data[["year", "deaths"]]
-... )
-<Expression height=2 terms=4 type=linear>
-┌──────┬───────────────────────────┐
-│ year ┆ expression                │
-│ (2)  ┆                           │
-╞══════╪═══════════════════════════╡
-│ 2025 ┆ immigration[2025] -200000 │
-│ 2026 ┆ immigration[2026] -300000 │
-└──────┴───────────────────────────┘
-
-```
-
-But, if we simply change the order of the terms in our addition, we get an error:
-
-```pycon
->>> (
-...     population_data[["year", "births"]]
-...     + population_data[["year", "deaths"]]
-...     + model.immigration
-... )
-Traceback (most recent call last):
-...
-ValueError: Cannot create an expression with duplicate labels:
-┌────────┬────────┬─────────┬───────────────┐
-│ births ┆ deaths ┆ __coeff ┆ __variable_id │
-│ ---    ┆ ---    ┆ ---     ┆ ---           │
-│ f64    ┆ f64    ┆ i64     ┆ i32           │
-╞════════╪════════╪═════════╪═══════════════╡
-│ null   ┆ null   ┆ 4050    ┆ 0             │
-│ null   ┆ null   ┆ 4052    ┆ 0             │
-└────────┴────────┴─────────┴───────────────┘.
-
-```
-
-What happened? Since Python computes additions from left to right, the second re-arranged version failed because, in the first addition, neither operand is a Pyoframe object. As such, the addition is done by Pandas, not Pyoframe, which leads to unexpected results.
-
-How do we avoid these weird behaviors? Users can manually convert their DataFrames to Pyoframe expressions ahead of time with `.to_expr()`. For example:
-
-```pycon
->>> (
-...     population_data[["year", "births"]].to_expr()
-...     + population_data[["year", "deaths"]].to_expr()
-...     + model.immigration
-... )
-<Expression height=2 terms=4 type=linear>
-┌──────┬─────────────────────────────┐
-│ year ┆ expression                  │
-│ (2)  ┆                             │
-╞══════╪═════════════════════════════╡
-│ 2025 ┆ -200000 + immigration[2025] │
-│ 2026 ┆ -300000 + immigration[2026] │
-└──────┴─────────────────────────────┘
-
-```

docs/learn/get-started/basic-example/example-with-dimensions.md

@@ -50,7 +50,7 @@ Load `food_data.csv` using [Polars](https://pola.rs/) or [Pandas](https://pandas
 
     Pyoframe works the same whether you're using [Polars](https://pola.rs/) or [Pandas](https://pandas.pydata.org/), two similar libraries for manipulating data with DataFrames. We prefer using Polars because it is much faster (and generally better), but you can use whichever library you're most comfortable with.
 
-    Note that, internally, Pyoframe always uses Polars during computations to ensure the best performance. If you're using Pandas, your DataFrames will automatically be converted to Polars prior to computations. If needed, you can convert a Polars DataFrame back to Pandas using [`polars.DataFrame.to_pandas()`](https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.to_pandas.html#polars.DataFrame.to_pandas).
+    Note that, internally, Pyoframe always uses Polars during computations to ensure the best performance. If you're using Pandas, your DataFrames will automatically be converted to Polars prior to computations.
 
 ## Step 2: Create the model
 
@@ -101,7 +101,7 @@ First, multiply the variable by the protein amount.
 
 ```pycon
 >>> data[["food", "cost"]] * m.Buy
-<Expression height=2 terms=2 type=linear>
+<Expression (linear) height=2 terms=2>
 ┌──────────────┬─────────────────────┐
 │ food         ┆ expression          │
 │ (2)          ┆                     │
@@ -120,7 +120,7 @@ Second, notice that the `Expression` still has the `food` dimension—it really
 
 ```pycon
 >>> (data[["food", "cost"]] * m.Buy).sum("food")
-<Expression terms=2 type=linear>
+<Expression (linear) terms=2>
 4 Buy[tofu_block] +3 Buy[chickpea_can]
 
 ```
@@ -146,6 +146,8 @@ assert m.Buy.solution["solution"].to_list() == [2, 1]
 
 ## Put it all together
 
+If you've followed the steps above your code should look like:
+
 <!-- clear-namespace -->
 
 ```python
@@ -162,7 +164,7 @@ m.protein_constraint = (data[["food", "protein"]] * m.Buy).sum() >= 50
 m.optimize()
 ```
 
-So you should buy:
+And you can retrieve the problem's solution as follows:
 
 ```pycon
 >>> m.Buy.solution
@@ -177,6 +179,6 @@ So you should buy:
 
 ```
 
-Notice that since `m.Buy` is dimensioned, `m.Buy.solution` returned a DataFrame with the solution for each of the labels.
+Since `m.Buy` is dimensioned, `m.Buy.solution` returned a DataFrame with the solution for each of the labels!
 
 <!--  -->

docs/learn/get-started/basics.md

@@ -200,7 +200,7 @@ Notice how the order of the days in `Hours_Sleep` is reversed. This is no proble
 
 ```pycon
 >>> m.hours_remaining
-<Expression height=5 terms=15 type=linear>
+<Expression (linear) height=5 terms=15>
 ┌─────┬───────────────────────────────────────────┐
 │ day ┆ expression                                │
 │ (5) ┆                                           │
@@ -214,7 +214,15 @@ Notice how the order of the days in `Hours_Sleep` is reversed. This is no proble
 
 ```
 
-Expressions can also be formed by combining variables with DataFrames. For example:
+### Using parameters
+
+Often, our models need to incorporate external data. To do this, we need to use **parameters**.
+
+In Pyoframe, a parameter is actually just an [Expression][pyoframe.Expression] that does not contain any Variables (aka. a constant).
+
+You can convert your data to a parameter by passing it to [`pf.Param(data)`][pyoframe.Param]. The last column of the data is always treated as the parameter value, and all other columns are treated as labels. (See [Param][pyoframe.Param] for other ways to create parameters.)
+
+For example, consider the following code that integrates the `holidays` DataFrame into a pay calculation:
 
 ```python
 import pandas as pd
@@ -227,15 +235,34 @@ base_pay = 20
 holiday_bonus = 10
 
 m = pf.Model()
+m.is_holiday = pf.Param(holidays)
 m.Hours_Worked = pf.Variable(holidays["day"], lb=0)
-m.pay = m.Hours_Worked * base_pay + m.Hours_Worked * holidays * holiday_bonus
+m.pay = m.Hours_Worked * (base_pay + m.is_holiday * holiday_bonus)
+```
+
+Here, `m.is_holiday` is a parameter Expression:
+
+```pycon
+>>> m.is_holiday
+<Expression (parameter) height=5 terms=5>
+┌─────┬────────────┐
+│ day ┆ expression │
+│ (5) ┆            │
+╞═════╪════════════╡
+│ Mon ┆ 0          │
+│ Tue ┆ 0          │
+│ Wed ┆ 0          │
+│ Thu ┆ 0          │
+│ Fri ┆ 1          │
+└─────┴────────────┘
+
 ```
 
-Here the `holidays` DataFrame is converted into an Expression and then multiplied by the `Hours_Worked` variable. When DataFrames are converted to Expressions, the last column is always treated as the value column and all previous columns are treated as dimension columns. The result is:
+And the resulting `m.pay` Expression correctly incorporates the holiday bonus only on Fridays:
 
 ```pycon
 >>> m.pay
-<Expression height=5 terms=5 type=linear>
+<Expression (linear) height=5 terms=5>
 ┌─────┬──────────────────────┐
 │ day ┆ expression           │
 │ (5) ┆                      │
@@ -249,38 +276,29 @@ Here the `holidays` DataFrame is converted into an Expression and then multiplie
 
 ```
 
-The page on [Transforms](../concepts/special-functions.md) describes additional ways to manipulate Expressions (e.g. `.sum(…)`, `.map(…)`, `.next(…)`).
-
-### Using `.to_expr()`
-
-Note that previous example would not have worked if we had instead written:
+Note that often, you can skip defining parameters because whenever a Pyoframe object is combined with a DataFrame, Pyoframe will automatically convert the DataFrame to a parameter Expression. For example, the following works just fine:
 
 ```pycon
->>> m.pay = m.Hours_Worked * (base_pay + holidays * holiday_bonus)
-Traceback (most recent call last):
-...
-TypeError: unsupported operand type(s) for +: 'int' and 'str'
-
-```
-
-The error occurs because you cannot simply multiply a DataFrame (`holidays`) by a number (`holiday_bonus`). In these cases, you need to explicitly convert the DataFrame to an Expression using `.to_expr()`:
-
-```pycon
->>> m.Hours_Worked * (base_pay + holidays.to_expr() * holiday_bonus)
-<Expression height=5 terms=5 type=linear>
+>>> m.bonus_pay = m.Hours_Worked * holidays * holiday_bonus
+>>> m.bonus_pay
+<Expression (linear) height=5 terms=5>
 ┌─────┬──────────────────────┐
 │ day ┆ expression           │
 │ (5) ┆                      │
 ╞═════╪══════════════════════╡
-│ Mon ┆ 20 Hours_Worked[Mon] │
-│ Tue ┆ 20 Hours_Worked[Tue] │
-│ Wed ┆ 20 Hours_Worked[Wed] │
-│ Thu ┆ 20 Hours_Worked[Thu] │
-│ Fri ┆ 30 Hours_Worked[Fri] │
+│ Mon ┆ 0                    │
+│ Tue ┆ 0                    │
+│ Wed ┆ 0                    │
+│ Thu ┆ 0                    │
+│ Fri ┆ 10 Hours_Worked[Fri] │
 └─────┴──────────────────────┘
 
 ```
 
+### Transforms
+
+The page on [Transforms](../concepts/special-functions.md) describes additional ways to formulate Expressions (e.g. using `.sum(…)`, `.map(…)`, `.next(…)`).
+
 ## Add constraints
 
 Create constraints by using the `<=`, `>=`, and `==` operators between Expressions. For example:

docs/reference/.nav.yml

@@ -2,5 +2,4 @@ nav:
   - Overview: index.md
   - Public API: public/
   - Base Classes: bases/
-  - External methods: external/
   - Types: types/