|
| 1 | +# DecisionFocusedLearningBenchmarks.jl |
| 2 | + |
| 3 | +[](https://JuliaDecisionFocusedLearning.github.io/DecisionFocusedLearningBenchmarks.jl/stable/) |
| 4 | +[](https://JuliaDecisionFocusedLearning.github.io/DecisionFocusedLearningBenchmarks.jl/dev/) |
| 5 | +[](https://github.com/JuliaDecisionFocusedLearning/DecisionFocusedLearningBenchmarks.jl/actions/workflows/Test.yml?query=branch%3Amain) |
| 6 | +[](https://app.codecov.io/gh/JuliaDecisionFocusedLearning/DecisionFocusedLearningBenchmarks.jl) |
| 7 | +[](https://github.com/JuliaDiff/BlueStyle) |
| 8 | + |
| 9 | +!!! warning |
| 10 | + This package is currently under active development. The API may change in future releases. |
| 11 | + Please refer to the [documentation](https://JuliaDecisionFocusedLearning.github.io/DecisionFocusedLearningBenchmarks.jl/stable/) for the latest updates. |
| 12 | + |
| 13 | +## What is Decision-Focused Learning? |
| 14 | + |
| 15 | +Decision-focused learning (DFL) is a paradigm that integrates machine learning prediction with combinatorial optimization to make better decisions under uncertainty. Unlike traditional "predict-then-optimize" approaches that optimize prediction accuracy independently of downstream decision quality, DFL directly optimizes end-to-end decision performance. |
| 16 | + |
| 17 | +A typical DFL algorithm involves training a parametrized policy that combines a statistical predictor with an optimization component: |
| 18 | + |
| 19 | +```math |
| 20 | +\xrightarrow{x} \boxed{\text{Statistical model } \varphi_w} \xrightarrow{\theta} \boxed{\text{CO algorithm } f} \xrightarrow{ y} |
| 21 | +``` |
| 22 | + |
| 23 | +Where: |
| 24 | +- **Instance** $x$: input data (e.g., features, context) |
| 25 | +- **Statistical model** $\varphi_w$: machine learning predictor (e.g., neural network) |
| 26 | +- **Parameters** $\theta$: predicted parameters for the optimization problem |
| 27 | +- **CO algorithm** $f$: combinatorial optimization solver |
| 28 | +- **Solution** $y$: final decision/solution |
| 29 | + |
| 30 | +## Package Overview |
| 31 | + |
| 32 | +**DecisionFocusedLearningBenchmarks.jl** provides a comprehensive collection of benchmark problems for evaluating decision-focused learning algorithms. The package offers: |
| 33 | + |
| 34 | +- **Standardized benchmark problems** spanning diverse application domains |
| 35 | +- **Common interfaces** for datasets, statistical models, and optimization components |
| 36 | +- **Ready-to-use pipelines** compatible with [InferOpt.jl](https://github.com/JuliaDecisionFocusedLearning/InferOpt.jl) and the whole [JuliaDecisionFocusedLearning](https://github.com/JuliaDecisionFocusedLearning) ecosystem |
| 37 | +- **Evaluation tools** for comparing algorithm performance |
| 38 | + |
| 39 | +## Benchmark Categories |
| 40 | + |
| 41 | +The package organizes benchmarks into three main categories based on their problem structure: |
| 42 | + |
| 43 | +### Static Benchmarks (`AbstractBenchmark`) |
| 44 | +Single-stage optimization problems with no randomness involved: |
| 45 | +- [`ArgmaxBenchmark`](@ref): argmax toy problem |
| 46 | +- [`Argmax2DBenchmark`](@ref): 2D argmax toy problem |
| 47 | +- [`RankingBenchmark`](@ref): ranking problem |
| 48 | +- [`SubsetSelectionBenchmark`](@ref): select optimal subset of items |
| 49 | +- [`PortfolioOptimizationBenchmark`](@ref): portfolio optimization problem |
| 50 | +- [`FixedSizeShortestPathBenchmark`](@ref): find shortest path on grid graphs with fixed size |
| 51 | +- [`WarcraftBenchmark`](@ref): shortest path on image maps |
| 52 | + |
| 53 | +### Stochastic Benchmarks (`AbstractStochasticBenchmark`) |
| 54 | +Single-stage problems with random noise affecting the objective: |
| 55 | +- [`StochasticVehicleSchedulingBenchmark`](@ref): stochastic vehicle scheduling under delay uncertainty |
| 56 | + |
| 57 | +### Dynamic Benchmarks (`AbstractDynamicBenchmark`) |
| 58 | +Multi-stage sequential decision-making problems: |
| 59 | +- [`DynamicVehicleSchedulingBenchmark`](@ref): multi-stage vehicle scheduling under customer uncertainty |
| 60 | +- [`DynamicAssortmentBenchmark`](@ref): sequential product assortment selection |
| 61 | + |
| 62 | +## Getting Started |
| 63 | + |
| 64 | +In a few lines of code, you can create benchmark instances, generate datasets, initialize learning components, and evaluate performance, using the same syntax across all benchmarks: |
| 65 | + |
| 66 | +```julia |
| 67 | +using DecisionFocusedLearningBenchmarks |
| 68 | + |
| 69 | +# Create a benchmark instance for the argmax problem |
| 70 | +benchmark = ArgmaxBenchmark() |
| 71 | + |
| 72 | +# Generate training data |
| 73 | +dataset = generate_dataset(benchmark, 100) |
| 74 | + |
| 75 | +# Initialize policy components |
| 76 | +model = generate_statistical_model(benchmark) |
| 77 | +maximizer = generate_maximizer(benchmark) |
| 78 | + |
| 79 | +# Training algorithm you want to use |
| 80 | +# ... your training code here ... |
| 81 | + |
| 82 | +# Evaluate performance |
| 83 | +gap = compute_gap(benchmark, dataset, model, maximizer) |
| 84 | +``` |
| 85 | + |
| 86 | +## Related Packages |
| 87 | + |
| 88 | +This package is part of the [JuliaDecisionFocusedLearning](https://github.com/JuliaDecisionFocusedLearning) organization, and built to be compatible with other packages in the ecosystem: |
| 89 | +- **[InferOpt.jl](https://github.com/JuliaDecisionFocusedLearning/InferOpt.jl)**: differentiable optimization layers and losses for decision-focused learning |
| 90 | +- **[DecisionFocusedLearningAlgorithms.jl](https://github.com/JuliaDecisionFocusedLearning/DecisionFocusedLearningAlgorithms.jl)**: collection of generic black-box implementations of decision-focused learning algorithms |
0 commit comments