From 484de7b0b3d3a372e5211a06d75d29caad1b4a6f Mon Sep 17 00:00:00 2001 From: "Anthony D. Blaom" Date: Sat, 25 Jan 2025 09:36:51 +1300 Subject: [PATCH 01/12] Integrate documentation for LearnTestAPI --- docs/Project.toml | 1 + docs/make.jl | 3 +- docs/src/index.md | 15 +++++--- docs/src/testing_an_implementation.md | 52 +++++++++++++++++++++++++-- 4 files changed, 62 insertions(+), 9 deletions(-) diff --git a/docs/Project.toml b/docs/Project.toml index 4d8ef094..9c183588 100644 --- a/docs/Project.toml +++ b/docs/Project.toml @@ -2,6 +2,7 @@ Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4" DocumenterInterLinks = "d12716ef-a0f6-4df4-a9f1-a5a34e75c656" LearnAPI = "92ad9a40-7767-427a-9ee6-6e577f1266cb" +LearnTestAPI = "3111ed91-c4f2-40e7-bb19-7f6c618409b8" MLUtils = "f1d291b0-491e-4a28-83b9-f70985020b54" ScientificTypesBase = "30f210dd-8aff-4c5f-94ba-8e64358c1161" Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c" diff --git a/docs/make.jl b/docs/make.jl index 95e0480a..5bc8fdbb 100644 --- a/docs/make.jl +++ b/docs/make.jl @@ -2,11 +2,12 @@ using Documenter using LearnAPI using ScientificTypesBase using DocumenterInterLinks +using LearnTestAPI const REPO = Remotes.GitHub("JuliaAI", "LearnAPI.jl") makedocs( - modules=[LearnAPI,], + modules=[LearnAPI, LearnTestAPI], format=Documenter.HTML( prettyurls = true,#get(ENV, "CI", nothing) == "true", collapselevel = 1, diff --git a/docs/src/index.md b/docs/src/index.md index 0d10db0f..35248255 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -84,11 +84,13 @@ appear as an input to training but not to prediction. ## Data interfaces Algorithms are free to consume data in any format. However, a method called [`obs`](@ref -data_interface) (read as "observations") gives users and meta-algorithms access to an -algorithm-specific representation of input data, which is also guaranteed to implement a -standard interface for accessing individual observations, unless the algorithm explicitly -opts out. Moreover, the `fit` and `predict` methods will also be able to consume these -alternative data representations, for performance benefits in some situations. +data_interface) (read as "observations") gives developers the option of providing a +separate data front end for their algorithms. In this case `obs` gives users and +meta-algorithms access to an algorithm-specific representation of input data, which is +also guaranteed to implement a standard interface for accessing individual observations, +unless the algorithm explicitly opts out. Moreover, the `fit` and `predict` methods will +also be able to consume these alternative data representations, for performance benefits +in some situations. The fallback data interface is the [MLUtils.jl](https://github.com/JuliaML/MLUtils.jl) `getobs/numobs` interface, here tagged as [`LearnAPI.RandomAccess()`](@ref), and if the @@ -97,6 +99,9 @@ then overloading `obs` is completely optional. Plain iteration interfaces, with knowledge of the number of observations, can also be specified, to support, e.g., data loaders reading images from disk. +Some canned data front ends are provided by the +[LearnDataFrontEnds.jl](https://juliaai.github.io/LearnAPI.jl/stable/) package. + ## Learning more - [Anatomy of an Implementation](@ref): informal introduction to the main actors in a new diff --git a/docs/src/testing_an_implementation.md b/docs/src/testing_an_implementation.md index 449a031c..ae971889 100644 --- a/docs/src/testing_an_implementation.md +++ b/docs/src/testing_an_implementation.md @@ -1,9 +1,55 @@ # Testing an Implementation -```@raw html -🚧 +Testing is provided by the LearnTestAPI.jl package documented below. + +## Quick start + +```@docs +LearnTestAPI ``` +LearnAPI.jl and LearnTestAPI.jl have synchronized releases. For example, LearnTestAPI.jl +version 0.2.3 will generally support all LearnAPI.jl versions 0.2.*. + !!! warning - Under construction + New releases of LearnTestAPI.jl may add tests to `@testapi`, and this may result in + new failures in client package test suites. Nevertheless, adding a test to `@testapi` + is not considered a breaking change to LearnTestAPI, unless the addition supports a + breaking release of LearnAPI.jl. + + +## The @testapi macro + +```@docs +LearnTestAPI.@testapi +``` + +## Learners for testing + +LearnTestAPI.jl provides some simple, tested, LearnAPI.jl implementations, which may be +useful for testing learner wrappers and meta-algorithms. + +```@docs +LearnTestAPI.Ridge +LearnTestAPI.BabyRidge +LearnTestAPI.TruncatedSVD +LearnTestAPI.Selector +LearnTestAPI.FancySelector +LearnTestAPI.NormalEstimator +LearnTestAPI.Ensemble +``` + +## Private methods + +For LearnTestAPI.jl developers only, and subject to breaking changes: + +```@docs +LearnTestAPI.@logged_testset +LearnTestAPI.@nearly +LearnTestAPI.isnear +LearnTestAPI.learner_get +LearnTestAPI.model_get +LearnTestAPI.verb +LearnTestAPI.filter_out_verbosity +``` From 6e2d8c84aa45234e9b166b647a78c7283d7cd732 Mon Sep 17 00:00:00 2001 From: "Anthony D. Blaom" Date: Tue, 11 Feb 2025 17:21:43 +1300 Subject: [PATCH 02/12] apply readme simplifications & other tweaks --- README.md | 35 +++++++++------------------ docs/src/index.md | 10 ++++---- docs/src/testing_an_implementation.md | 1 + 3 files changed, 17 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index 824cc957..7df15c2d 100644 --- a/README.md +++ b/README.md @@ -11,40 +11,27 @@ Comprehensive documentation is [here](https://juliaai.github.io/LearnAPI.jl/dev/ New contributions welcome. See the [road map](ROADMAP.md). -## Code snippet +## Synopsis -Configure a machine learning algorithm: +Package provides for variations and elaborations on the following basic pattern in machine +learning and statistics: ```julia -julia> ridge = Ridge(lambda=0.1) +model = fit(learner, data) +predict(model, newdata) ``` -Inspect available functionality: +Here `learner` specifies the configuration the algorithm (the hyperparameters) while +`model` stores learned parameters and any byproducts of algorithm execution. -``` -julia> @functions ridge -(fit, LearnAPI.learner, LearnAPI.strip, obs, LearnAPI.features, LearnAPI.target, predict, LearnAPI.coefficients) -``` - -Train: - -```julia -julia> model = fit(ridge, data) -``` +## Related packages -Predict: +- [MLCore.jl](https://github.com/JuliaML/MLCore.jl) ([docs](https://juliaml.github.io/MLUtils.jl/stable/api/#Core-API)) -```julia -julia> predict(model, newdata)[1] -"virginica" -``` +- [LearnTestAPI.jl](https://github.com/JuliaAI/LearnTestAPI.jl): Package to test implementations of LearnAPI.jl (but documented here) -Predict a probability distribution ([proxy](https://juliaai.github.io/LearnAPI.jl/dev/kinds_of_target_proxy/#proxy_types) for the target): +- [LearnDataFrontEnds.jl](https://github.com/JuliaAI/LearnDataFrontEnds.jl): for including flexible, user-friendly, data front ends for LearnAPI.jl implementations ([docs]((https://juliaai.github.io/stable/)) -```julia -julia> predict(model, Distribution(), newdata)[1] -UnivariateFinite{Multiclass{3}}(setosa=>0.0, versicolor=>0.25, virginica=>0.75) -``` ## Credits diff --git a/docs/src/index.md b/docs/src/index.md index 35248255..1da20859 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -4,10 +4,10 @@
Tutorial  |  - Reference  |  Patterns + Reference  | 
@@ -87,7 +87,7 @@ Algorithms are free to consume data in any format. However, a method called [`ob data_interface) (read as "observations") gives developers the option of providing a separate data front end for their algorithms. In this case `obs` gives users and meta-algorithms access to an algorithm-specific representation of input data, which is -also guaranteed to implement a standard interface for accessing individual observations, +additionally guaranteed to implement a standard interface for accessing individual observations, unless the algorithm explicitly opts out. Moreover, the `fit` and `predict` methods will also be able to consume these alternative data representations, for performance benefits in some situations. @@ -99,8 +99,8 @@ then overloading `obs` is completely optional. Plain iteration interfaces, with knowledge of the number of observations, can also be specified, to support, e.g., data loaders reading images from disk. -Some canned data front ends are provided by the -[LearnDataFrontEnds.jl](https://juliaai.github.io/LearnAPI.jl/stable/) package. +Some canned data front ends (implementations of [`obs`](@ref)) are provided by the +[LearnDataFrontEnds.jl](https://juliaai.github.io/LearnDataFrontEnds.jl/stable/) package. ## Learning more diff --git a/docs/src/testing_an_implementation.md b/docs/src/testing_an_implementation.md index ae971889..f1153e39 100644 --- a/docs/src/testing_an_implementation.md +++ b/docs/src/testing_an_implementation.md @@ -38,6 +38,7 @@ LearnTestAPI.Selector LearnTestAPI.FancySelector LearnTestAPI.NormalEstimator LearnTestAPI.Ensemble +LearnTestAPI.StumpRegressor ``` ## Private methods From c8d50ca322bd11d36470a306c1bcc104f346fdeb Mon Sep 17 00:00:00 2001 From: "Anthony D. Blaom" Date: Tue, 11 Feb 2025 17:34:49 +1300 Subject: [PATCH 03/12] replace MLUtils wiith MLCore --- README.md | 2 +- docs/Project.toml | 2 +- docs/src/anatomy_of_an_implementation.md | 22 +++++++++++----------- docs/src/index.md | 13 +++++++------ docs/src/obs.md | 14 +++++++------- docs/src/predict_transform.md | 4 ++-- docs/src/reference.md | 4 ++-- docs/src/traits.md | 2 +- src/accessor_functions.jl | 2 +- src/obs.jl | 10 +++++----- src/traits.jl | 4 ++-- src/types.jl | 12 ++++++------ 12 files changed, 46 insertions(+), 45 deletions(-) diff --git a/README.md b/README.md index 7df15c2d..8c893c4b 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ Here `learner` specifies the configuration the algorithm (the hyperparameters) w ## Related packages -- [MLCore.jl](https://github.com/JuliaML/MLCore.jl) ([docs](https://juliaml.github.io/MLUtils.jl/stable/api/#Core-API)) +- [MLCore.jl](https://github.com/JuliaML/MLCore.jl) ([docs](https://juliaml.github.io/MLCore.jl/stable/api/#Core-API)) - [LearnTestAPI.jl](https://github.com/JuliaAI/LearnTestAPI.jl): Package to test implementations of LearnAPI.jl (but documented here) diff --git a/docs/Project.toml b/docs/Project.toml index 9c183588..4ffd503c 100644 --- a/docs/Project.toml +++ b/docs/Project.toml @@ -3,7 +3,7 @@ Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4" DocumenterInterLinks = "d12716ef-a0f6-4df4-a9f1-a5a34e75c656" LearnAPI = "92ad9a40-7767-427a-9ee6-6e577f1266cb" LearnTestAPI = "3111ed91-c4f2-40e7-bb19-7f6c618409b8" -MLUtils = "f1d291b0-491e-4a28-83b9-f70985020b54" +MLCore = "c2834f40-e789-41da-a90e-33b280584a8c" ScientificTypesBase = "30f210dd-8aff-4c5f-94ba-8e64358c1161" Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c" diff --git a/docs/src/anatomy_of_an_implementation.md b/docs/src/anatomy_of_an_implementation.md index e6dba45a..4eafe265 100644 --- a/docs/src/anatomy_of_an_implementation.md +++ b/docs/src/anatomy_of_an_implementation.md @@ -334,7 +334,7 @@ assumptions about data from those made above. - If the `data` object consumed by `fit`, `predict`, or `transform` is not not a suitable table¹, array³, tuple of tables and arrays, or some other object implementing the - [MLUtils.jl](https://juliaml.github.io/MLUtils.jl/dev/) `getobs`/`numobs` interface, + [MLCore.jl](https://juliaml.github.io/MLCore.jl/dev/) `getobs`/`numobs` interface, then an implementation must: (i) overload [`obs`](@ref) to articulate how provided data can be transformed into a form that does support this interface, as illustrated below under [Providing a separate data front end](@ref) below; or (ii) overload the trait @@ -419,7 +419,7 @@ The [`obs`](@ref) methods exist to: how it works. In the typical case, where [`LearnAPI.data_interface`](@ref) is not overloaded, the -alternative data representations must implement the MLUtils.jl `getobs/numobs` interface +alternative data representations must implement the MLCore.jl `getobs/numobs` interface for observation subsampling, which is generally all a user or meta-algorithm will need, before passing the data on to `fit`/`predict`, as you would the original data. @@ -436,14 +436,14 @@ one enables the following alternative: observations = obs(learner, data) # preprocessed training data # optional subsampling: -observations = MLUtils.getobs(observations, train_indices) +observations = MLCore.getobs(observations, train_indices) model = fit(learner, observations) newobservations = obs(model, newdata) # optional subsampling: -newobservations = MLUtils.getobs(observations, test_indices) +newobservations = MLCore.getobs(observations, test_indices) predict(model, newobservations) ``` @@ -568,7 +568,7 @@ LearnAPI.target(learner::Ridge, data) = LearnAPI.target(learner, obs(learner, da are generally different. - We need the adjoint operator, `'`, because the last dimension in arrays is the - observation dimension, according to the MLUtils.jl convention. Remember, `Xnew` is a + observation dimension, according to the MLCore.jl convention. Remember, `Xnew` is a table here. Since LearnAPI.jl provides fallbacks for `obs` that simply return the unadulterated data @@ -576,7 +576,7 @@ argument, overloading `obs` is optional. This is provided data in publicized `fit`/`predict` signatures already consists only of objects implement the [`LearnAPI.RandomAccess`](@ref) interface (most tables¹, arrays³, and tuples thereof). -To opt out of supporting the MLUtils.jl interface altogether, an implementation must +To opt out of supporting the MLCore.jl interface altogether, an implementation must overload the trait, [`LearnAPI.data_interface(learner)`](@ref). See [Data interfaces](@ref data_interfaces) for details. @@ -593,15 +593,15 @@ LearnAPI.fit(learner::Ridge, X, y; kwargs...) = fit(learner, (X, y); kwargs...) ## [Demonstration of an advanced `obs` workflow](@id advanced_demo) We now can train and predict using internal data representations, resampled using the -generic MLUtils.jl interface: +generic MLCore.jl interface: ```@example anatomy2 -import MLUtils +import MLCore learner = Ridge() observations_for_fit = obs(learner, (X, y)) -model = fit(learner, MLUtils.getobs(observations_for_fit, train)) +model = fit(learner, MLCore.getobs(observations_for_fit, train)) observations_for_predict = obs(model, X) -ẑ = predict(model, MLUtils.getobs(observations_for_predict, test)) +ẑ = predict(model, MLCore.getobs(observations_for_predict, test)) ``` ```julia @@ -616,7 +616,7 @@ obs_workflows). ¹ In LearnAPI.jl a *table* is any object `X` implementing the [Tables.jl](https://tables.juliadata.org/dev/) interface, additionally satisfying `Tables.istable(X) == true` and implementing `DataAPI.nrow` (and whence -`MLUtils.numobs`). Tables that are also (unnamed) tuples are disallowed. +`MLCore.numobs`). Tables that are also (unnamed) tuples are disallowed. ² An implementation can provide further accessor functions, if necessary, but like the native ones, they must be included in the [`LearnAPI.functions`](@ref) diff --git a/docs/src/index.md b/docs/src/index.md index 1da20859..a862aee7 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -92,12 +92,13 @@ unless the algorithm explicitly opts out. Moreover, the `fit` and `predict` meth also be able to consume these alternative data representations, for performance benefits in some situations. -The fallback data interface is the [MLUtils.jl](https://github.com/JuliaML/MLUtils.jl) -`getobs/numobs` interface, here tagged as [`LearnAPI.RandomAccess()`](@ref), and if the -input consumed by the algorithm already implements that interface (tables, arrays, etc.) -then overloading `obs` is completely optional. Plain iteration interfaces, with or without -knowledge of the number of observations, can also be specified, to support, e.g., data -loaders reading images from disk. +The fallback data interface is the [MLCore.jl](https://github.com/JuliaML/MLCore.jl) +`getobs/numobs` interface (previously provided by MLUtils.jl) here tagged as +[`LearnAPI.RandomAccess()`](@ref). However, if the input consumed by the algorithm already +implements that interface (tables, arrays, etc.) then overloading `obs` is completely +optional. Plain iteration interfaces, with or without knowledge of the number of +observations, can also be specified, to support, e.g., data loaders reading images from +disk. Some canned data front ends (implementations of [`obs`](@ref)) are provided by the [LearnDataFrontEnds.jl](https://juliaai.github.io/LearnDataFrontEnds.jl/stable/) package. diff --git a/docs/src/obs.md b/docs/src/obs.md index 70b6eb46..c36f4e3d 100644 --- a/docs/src/obs.md +++ b/docs/src/obs.md @@ -26,26 +26,26 @@ observations = obs(learner, data) then, assuming the typical case that `LearnAPI.data_interface(learner) == LearnAPI.RandomAccess()`, `observations` implements the -[MLUtils.jl](https://juliaml.github.io/MLUtils.jl/dev/) `getobs`/`numobs` interface, for +[MLCore.jl](https://juliaml.github.io/MLCore.jl/dev/) `getobs`/`numobs` interface, for grabbing and counting observations. Moreover, we can pass `observations` to `fit` in place -of the original data, or first resample it using `MLUtils.getobs`: +of the original data, or first resample it using `MLCore.getobs`: ```julia # equivalent to `model = fit(learner, data)` model = fit(learner, observations) # with resampling: -resampled_observations = MLUtils.getobs(observations, 1:10) +resampled_observations = MLCore.getobs(observations, 1:10) model = fit(learner, resampled_observations) ``` In some implementations, the alternative pattern above can be used to avoid repeating unnecessary internal data preprocessing, or inefficient resampling. For example, here's -how a user might call `obs` and `MLUtils.getobs` to perform efficient cross-validation: +how a user might call `obs` and `MLCore.getobs` to perform efficient cross-validation: ```julia using LearnAPI -import MLUtils +import MLCore learner = @@ -61,7 +61,7 @@ never_trained = true scores = map(train_test_folds) do (train, test) # train using model-specific representation of data: - fitobs_subset = MLUtils.getobs(fitobs, train) + fitobs_subset = MLCore.getobs(fitobs, train) model = fit(learner, fitobs_subset) # predict on the fold complement: @@ -70,7 +70,7 @@ scores = map(train_test_folds) do (train, test) global predictobs = obs(model, X) global never_trained = false end - predictobs_subset = MLUtils.getobs(predictobs, test) + predictobs_subset = MLCore.getobs(predictobs, test) ŷ = predict(model, Point(), predictobs_subset) y = LearnAPI.target(learner, data) diff --git a/docs/src/predict_transform.md b/docs/src/predict_transform.md index d6ab8f25..2733fc64 100644 --- a/docs/src/predict_transform.md +++ b/docs/src/predict_transform.md @@ -52,8 +52,8 @@ transform(learner, data) # `fit` implied ```julia fitobs = obs(learner, (X, y)) # learner-specific repr. of data -model = fit(learner, MLUtils.getobs(fitobs, 1:100)) -predictobs = obs(model, MLUtils.getobs(X, 101:150)) +model = fit(learner, MLCore.getobs(fitobs, 1:100)) +predictobs = obs(model, MLCore.getobs(X, 101:150)) ŷ = predict(model, Point(), predictobs) ``` diff --git a/docs/src/reference.md b/docs/src/reference.md index 18fb92df..8a18bf6f 100644 --- a/docs/src/reference.md +++ b/docs/src/reference.md @@ -21,13 +21,13 @@ individual observations. A `DataFrame` instance, from [DataFrames.jl](https://dataframes.juliadata.org/stable/), is an example of data, the observations being the rows. Typically, data provided to LearnAPI.jl algorithms, will implement the -[MLUtils.jl](https://juliaml.github.io/MLUtils.jl/stable) `getobs/numobs` interface for +[MLCore.jl](https://juliaml.github.io/MLCore.jl/stable) `getobs/numobs` interface for accessing individual observations, but implementations can opt out of this requirement; see [`obs`](@ref) and [`LearnAPI.data_interface`](@ref) for details. !!! note - In the MLUtils.jl + In the MLCore.jl convention, observations in tables are the rows but observations in a matrix are the columns. diff --git a/docs/src/traits.md b/docs/src/traits.md index eebb9e8e..890a53f9 100644 --- a/docs/src/traits.md +++ b/docs/src/traits.md @@ -27,7 +27,7 @@ In the examples column of the table below, `Continuous` is a name owned the pack | [`LearnAPI.nonlearners`](@ref)`(learner)` | properties *not* corresponding to other learners | all properties | `(:K, :leafsize, :metric,)` | | [`LearnAPI.human_name`](@ref)`(learner)` | human name for the learner; should be a noun | type name with spaces | "elastic net regressor" | | [`LearnAPI.iteration_parameter`](@ref)`(learner)` | symbolic name of an iteration parameter | `nothing` | :epochs | -| [`LearnAPI.data_interface`](@ref)`(learner)` | Interface implemented by objects returned by [`obs`](@ref) | `Base.HasLength()` (supports `MLUtils.getobs/numobs`) | `Base.SizeUnknown()` (supports `iterate`) | +| [`LearnAPI.data_interface`](@ref)`(learner)` | Interface implemented by objects returned by [`obs`](@ref) | `Base.HasLength()` (supports `MLCore.getobs/numobs`) | `Base.SizeUnknown()` (supports `iterate`) | | [`LearnAPI.fit_scitype`](@ref)`(learner)` | upper bound on `scitype(data)` ensuring `fit(learner, data)` works | `Union{}` | `Tuple{AbstractVector{Continuous}, Continuous}` | | [`LearnAPI.target_observation_scitype`](@ref)`(learner)` | upper bound on the scitype of each observation of the targget | `Any` | `Continuous` | | [`LearnAPI.is_static`](@ref)`(learner)` | `true` if `fit` consumes no data | `false` | `true` | diff --git a/src/accessor_functions.jl b/src/accessor_functions.jl index b5f487b4..84cbab05 100644 --- a/src/accessor_functions.jl +++ b/src/accessor_functions.jl @@ -319,7 +319,7 @@ Here's a sample workflow for some such `learner`, with training data, `(X, y)`, is the training target, here assumed to be a vector. ```julia -import MLUtils.getobs +import MLCore.getobs model = fit(learner, (X, y)) yhat = LearnAPI.predictions(model) test_indices = LearnAPI.out_of_sample_indices(model) diff --git a/src/obs.jl b/src/obs.jl index 2e631d30..ea9024d0 100644 --- a/src/obs.jl +++ b/src/obs.jl @@ -25,15 +25,15 @@ model = fit(learner, data_train) ŷ = predict(model, Point(), X[101:150]) ``` -Alternative workflow using `obs` and the MLUtils.jl method `getobs` to carry out +Alternative workflow using `obs` and the MLCore.jl method `getobs` to carry out subsampling (assumes `LearnAPI.data_interface(learner) == RandomAccess()`): ```julia -import MLUtils +import MLCore fit_observations = obs(learner, data) -model = fit(learner, MLUtils.getobs(fit_observations, 1:100)) +model = fit(learner, MLCore.getobs(fit_observations, 1:100)) predict_observations = obs(model, X) -ẑ = predict(model, Point(), MLUtils.getobs(predict_observations, 101:150)) +ẑ = predict(model, Point(), MLCore.getobs(predict_observations, 101:150)) @assert ẑ == ŷ ``` @@ -54,7 +54,7 @@ alternatives with the same output, whenever `observations = obs(model, data)`. If `LearnAPI.data_interface(learner) == RandomAccess()` (the default), then `fit`, `predict` and `transform` must additionally accept `obs` output that has been *subsampled* -using `MLUtils.getobs`, with the obvious interpretation applying to the outcomes of such +using `MLCore.getobs`, with the obvious interpretation applying to the outcomes of such calls (e.g., if *all* observations are subsampled, then outcomes should be the same as if using the original data). diff --git a/src/traits.jl b/src/traits.jl index 0a99aaff..e85af904 100644 --- a/src/traits.jl +++ b/src/traits.jl @@ -12,7 +12,7 @@ const DOC_ON_TYPE = "The value of the trait must depend only on the type of `lea # Here, "for each `o` in `observations`" is understood in the sense of the data # interface specified for the learner, [`LearnAPI.data_interface(learner)`](@ref). For # example, if this is `LearnAPI.RandomAccess()`, then this means "for `o` in -# `MLUtils.eachobs(observations)`". +# `MLCore.eachobs(observations)`". # """ @@ -470,7 +470,7 @@ Specifically, both of the following are always true: `target_observations`" is understood in the sense of the data interface specified for the learner, [`LearnAPI.data_interface(learner)`](@ref). For example, if this is `LearnAPI.RandomAccess()`, then this means "for each `o in - MLUtils.eachobs(target_observations)`". + MLCore.eachobs(target_observations)`". - `S` is an upper bound on the `scitype` of (point) observations that might normally be extracted from the output of [`predict`](@ref). diff --git a/src/types.jl b/src/types.jl index 3212c3f2..810c345e 100644 --- a/src/types.jl +++ b/src/types.jl @@ -212,9 +212,9 @@ abstract type Finite <: DataInterface end LearnAPI.RandomAccess A data interface type. We say that `data` implements the `RandomAccess` interface if -`data` implements the methods `getobs` and `numobs` from MLUtils.jl. The first method +`data` implements the methods `getobs` and `numobs` from MLCore.jl. The first method allows one to grab observations specified by an arbitrary index set, as in -`MLUtils.getobs(data, [2, 3, 5])`, while the second method returns the total number of +`MLCore.getobs(data, [2, 3, 5])`, while the second method returns the total number of available observations, which is assumed to be known and finite. All arrays implement `RandomAccess`, with the last index being the observation index @@ -234,8 +234,8 @@ If [`LearnAPI.data_interface(learner)`](@ref) takes the value `RandomAccess()`, # Implementing `RandomAccess` for new data types Typically, to implement `RandomAccess` for a new data type requires only implementing -`Base.getindex` and `Base.length`, which are the fallbacks for `MLUtils.getobs` and -`MLUtils.numobs`, and this avoids making MLUtils.jl a package dependency. +`Base.getindex` and `Base.length`, which are the fallbacks for `MLCore.getobs` and +`MLCore.numobs`, and this avoids making MLCore.jl a package dependency. See also [`LearnAPI.FiniteIterable`](@ref), [`LearnAPI.Iterable`](@ref). """ @@ -251,7 +251,7 @@ it implements Julia's `iterate` interface, including `Base.length`, and if - `data` implements the [`LearnAPI.RandomAccess`](@ref) interface (arrays and most tables); or -- `data isa MLUtils.DataLoader`, which includes output from `MLUtils.eachobs`. +- `data isa MLCore.DataLoader`, which includes output from `MLCore.eachobs`. If [`LearnAPI.data_interface(learner)`](@ref) takes the value `FiniteIterable()`, then [`obs`](@ref)`(learner, ...)` is guaranteed to return objects implementing the @@ -267,7 +267,7 @@ struct FiniteIterable <: Finite end A data interface type. We say that `data` implements the `Iterable` interface if it implements Julia's basic `iterate` interface. (Such objects may not implement -`MLUtils.numobs` or `Base.length`.) +`MLCore.numobs` or `Base.length`.) If [`LearnAPI.data_interface(learner)`](@ref) takes the value `Iterable()`, then [`obs`](@ref)`(learner, ...)` is guaranteed to return objects implementing `Iterable`, From 05c466b37c4b42cefe84eb7be54453b287502899 Mon Sep 17 00:00:00 2001 From: "Anthony D. Blaom" Date: Fri, 14 Feb 2025 21:58:41 +1300 Subject: [PATCH 04/12] add links to classifier exemplar; fix some links --- README.md | 4 ++-- docs/src/patterns/classification.md | 4 +++- docs/src/patterns/density_estimation.md | 2 +- docs/src/patterns/dimension_reduction.md | 2 +- docs/src/patterns/ensembling.md | 4 ++-- docs/src/patterns/feature_engineering.md | 2 +- docs/src/patterns/gradient_descent.md | 2 +- docs/src/patterns/incremental_algorithms.md | 2 +- docs/src/patterns/iterative_algorithms.md | 6 +++--- docs/src/patterns/meta_algorithms.md | 2 +- docs/src/patterns/regression.md | 4 ++-- docs/src/patterns/static_algorithms.md | 2 +- docs/src/patterns/transformers.md | 2 +- docs/src/testing_an_implementation.md | 1 + 14 files changed, 21 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 8c893c4b..5c3a259a 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ New contributions welcome. See the [road map](ROADMAP.md). ## Synopsis -Package provides for variations and elaborations on the following basic pattern in machine +LearnAPI.jl provides for variations and elaborations on the following basic pattern in machine learning and statistics: ```julia @@ -30,7 +30,7 @@ Here `learner` specifies the configuration the algorithm (the hyperparameters) w - [LearnTestAPI.jl](https://github.com/JuliaAI/LearnTestAPI.jl): Package to test implementations of LearnAPI.jl (but documented here) -- [LearnDataFrontEnds.jl](https://github.com/JuliaAI/LearnDataFrontEnds.jl): for including flexible, user-friendly, data front ends for LearnAPI.jl implementations ([docs]((https://juliaai.github.io/stable/)) +- [LearnDataFrontEnds.jl](https://github.com/JuliaAI/LearnDataFrontEnds.jl): for including flexible, user-friendly, data front ends for LearnAPI.jl implementations ([docs](https://juliaai.github.io/stable/)) ## Credits diff --git a/docs/src/patterns/classification.md b/docs/src/patterns/classification.md index fd278478..41fc522e 100644 --- a/docs/src/patterns/classification.md +++ b/docs/src/patterns/classification.md @@ -2,4 +2,6 @@ See these examples from the JuliaTestAPI.jl test suite: -- [perceptron classifier](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/gradient_descent.jl) +- [perceptron classifier](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/gradient_descent.jl) + +- [constant classifier](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/classification.jl) (including `Sage` data front end) diff --git a/docs/src/patterns/density_estimation.md b/docs/src/patterns/density_estimation.md index 9fc0144a..74cad18f 100644 --- a/docs/src/patterns/density_estimation.md +++ b/docs/src/patterns/density_estimation.md @@ -2,4 +2,4 @@ See these examples from the JuliaTestAPI.jl test suite: -- [normal distribution estimator](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/incremental_algorithms.jl) +- [normal distribution estimator](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/incremental_algorithms.jl) diff --git a/docs/src/patterns/dimension_reduction.md b/docs/src/patterns/dimension_reduction.md index e886dd15..e780559d 100644 --- a/docs/src/patterns/dimension_reduction.md +++ b/docs/src/patterns/dimension_reduction.md @@ -2,5 +2,5 @@ See these examples from the JuliaTestAPI.jl test suite: -- [Truncated SVD](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/dimension_reduction.jl) +- [Truncated SVD](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/dimension_reduction.jl) (including `Tarragon` data front end) diff --git a/docs/src/patterns/ensembling.md b/docs/src/patterns/ensembling.md index 8d774f5e..474d32e1 100644 --- a/docs/src/patterns/ensembling.md +++ b/docs/src/patterns/ensembling.md @@ -2,6 +2,6 @@ See these examples from the JuliaTestAPI.jl test suite: -- [bagged ensembling of a regression model](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/ensembling.jl) +- [bagged ensembling of a regression model](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/ensembling.jl) -- [extremely randomized ensemble of decision stumps (regression)](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/ensembling.jl) +- [extremely randomized ensemble of decision stumps (regression)](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/ensembling.jl) diff --git a/docs/src/patterns/feature_engineering.md b/docs/src/patterns/feature_engineering.md index 6e3c656c..1b098475 100644 --- a/docs/src/patterns/feature_engineering.md +++ b/docs/src/patterns/feature_engineering.md @@ -3,5 +3,5 @@ See these examples from the JuliaTestAI.jl test suite: - [feature - selectors](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/static_algorithms.jl) + selectors](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/static_algorithms.jl) from tests. diff --git a/docs/src/patterns/gradient_descent.md b/docs/src/patterns/gradient_descent.md index 9dc5401a..95cd7d00 100644 --- a/docs/src/patterns/gradient_descent.md +++ b/docs/src/patterns/gradient_descent.md @@ -2,4 +2,4 @@ See these examples from the JuliaTestAI.jl test suite: -- [perceptron classifier](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/gradient_descent.jl) +- [perceptron classifier](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/gradient_descent.jl) diff --git a/docs/src/patterns/incremental_algorithms.md b/docs/src/patterns/incremental_algorithms.md index d2855a55..fea6566b 100644 --- a/docs/src/patterns/incremental_algorithms.md +++ b/docs/src/patterns/incremental_algorithms.md @@ -2,4 +2,4 @@ See these examples from the JuliaTestAI.jl test suite: -- [normal distribution estimator](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/incremental_algorithms.jl) +- [normal distribution estimator](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/incremental_algorithms.jl) diff --git a/docs/src/patterns/iterative_algorithms.md b/docs/src/patterns/iterative_algorithms.md index 265dddf7..d7e81ad4 100644 --- a/docs/src/patterns/iterative_algorithms.md +++ b/docs/src/patterns/iterative_algorithms.md @@ -2,8 +2,8 @@ See these examples from the JuliaTestAI.jl test suite: -- [bagged ensembling](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/ensembling.jl) +- [bagged ensembling](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/ensembling.jl) -- [perceptron classifier](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/gradient_descent.jl) +- [perceptron classifier](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/gradient_descent.jl) -- [extremely randomized ensemble of decision stumps (regression)](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/ensembling.jl) +- [extremely randomized ensemble of decision stumps (regression)](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/ensembling.jl) diff --git a/docs/src/patterns/meta_algorithms.md b/docs/src/patterns/meta_algorithms.md index 6a9e7300..d86d7966 100644 --- a/docs/src/patterns/meta_algorithms.md +++ b/docs/src/patterns/meta_algorithms.md @@ -2,6 +2,6 @@ Many meta-algorithms are can be implemented as wrappers. An example is [this bagged ensemble -algorithm](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/ensembling.jl) +algorithm](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/ensembling.jl) from tests. diff --git a/docs/src/patterns/regression.md b/docs/src/patterns/regression.md index a6de5b10..52ae5477 100644 --- a/docs/src/patterns/regression.md +++ b/docs/src/patterns/regression.md @@ -2,6 +2,6 @@ See these examples from the JuliaTestAPI.jl test suite: -- [ridge regression](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/regression.jl) +- [ridge regression](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/regression.jl) (including `Saffron` data front end) -- [extremely randomized ensemble of decision stumps](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/ensembling.jl) +- [extremely randomized ensemble of decision stumps](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/ensembling.jl) diff --git a/docs/src/patterns/static_algorithms.md b/docs/src/patterns/static_algorithms.md index 4724006f..d62ea43d 100644 --- a/docs/src/patterns/static_algorithms.md +++ b/docs/src/patterns/static_algorithms.md @@ -3,7 +3,7 @@ See these examples from the JuliaTestAI.jl test suite: - [feature - selection](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/static_algorithms.jl) + selection](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/static_algorithms.jl) diff --git a/docs/src/patterns/transformers.md b/docs/src/patterns/transformers.md index c27f9682..b207dd2d 100644 --- a/docs/src/patterns/transformers.md +++ b/docs/src/patterns/transformers.md @@ -2,4 +2,4 @@ Check out the following examples from the TestLearnAPI.jl test suite: -- [Truncated SVD](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/dimension_reduction.jl) +- [Truncated SVD](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/dimension_reduction.jl) diff --git a/docs/src/testing_an_implementation.md b/docs/src/testing_an_implementation.md index f1153e39..24f78701 100644 --- a/docs/src/testing_an_implementation.md +++ b/docs/src/testing_an_implementation.md @@ -33,6 +33,7 @@ useful for testing learner wrappers and meta-algorithms. ```@docs LearnTestAPI.Ridge LearnTestAPI.BabyRidge +LearnTestAPI.ConstantClassifier LearnTestAPI.TruncatedSVD LearnTestAPI.Selector LearnTestAPI.FancySelector From 143f97a99d0aeec122ff6fd001098956fdca37cd Mon Sep 17 00:00:00 2001 From: "Anthony D. Blaom" Date: Tue, 18 Feb 2025 10:26:24 +1300 Subject: [PATCH 05/12] docstring tweaks around parameter updates --- docs/src/fit_update.md | 6 +++--- src/fit_update.jl | 8 ++++---- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/src/fit_update.md b/docs/src/fit_update.md index 8e27126c..2329d494 100644 --- a/docs/src/fit_update.md +++ b/docs/src/fit_update.md @@ -15,9 +15,9 @@ clustering algorithms); there is no training data and heavy lifting is carried o ### Updating ``` -update(model, data; verbosity=..., param1=new_value1, param2=new_value2, ...) -> updated_model -update_observations(model, new_data; verbosity=..., param1=new_value1, ...) -> updated_model -update_features(model, new_data; verbosity=..., param1=new_value1, ...) -> updated_model +update(model, data; verbosity=..., :param1=new_value1, :param2=new_value2, ...) -> updated_model +update_observations(model, new_data; verbosity=..., :param1=new_value1, ...) -> updated_model +update_features(model, new_data; verbosity=..., :param1=new_value1, ...) -> updated_model ``` ## Typical workflows diff --git a/src/fit_update.jl b/src/fit_update.jl index c33e40b8..39f78273 100644 --- a/src/fit_update.jl +++ b/src/fit_update.jl @@ -61,8 +61,8 @@ function fit end update(model, data, param_replacements...; verbosity=1) Return an updated version of the `model` object returned by a previous [`fit`](@ref) or -`update` call, but with the specified hyperparameter replacements, in the form `p1 => -value1, p2 => value2, ...`. +`update` call, but with the specified hyperparameter replacements, in the form `:p1 => +value1, :p2 => value2, ...`. ```julia learner = MyForest(ntrees=100) @@ -105,7 +105,7 @@ function update end Return an updated version of the `model` object returned by a previous [`fit`](@ref) or `update` call given the new observations present in `new_data`. One may additionally -specify hyperparameter replacements in the form `p1 => value1, p2 => value2, ...`. +specify hyperparameter replacements in the form `:p1 => value1, :p2 => value2, ...`. ```julia-repl learner = MyNeuralNetwork(epochs=10, learning_rate => 0.01) @@ -145,7 +145,7 @@ function update_observations end Return an updated version of the `model` object returned by a previous [`fit`](@ref) or `update` call given the new features encapsulated in `new_data`. One may additionally -specify hyperparameter replacements in the form `p1 => value1, p2 => value2, ...`. +specify hyperparameter replacements in the form `:p1 => value1, :p2 => value2, ...`. When following the call `fit(learner, data)`, the `update` call is semantically equivalent to retraining ab initio using a concatenation of `data` and `new_data`, From f8b3b94f8e122517a46af2808be0877a4c893da6 Mon Sep 17 00:00:00 2001 From: "Anthony D. Blaom" Date: Tue, 18 Feb 2025 13:38:15 +1300 Subject: [PATCH 06/12] add more doc improvements --- README.md | 34 ++++++++- docs/make.jl | 1 + docs/src/anatomy_of_an_implementation.md | 2 +- docs/src/examples.md | 93 +++++++++++++++++++++++- docs/src/index.md | 25 ++----- docs/src/list_of_public_names.md | 49 +++++++++++++ docs/src/reference.md | 8 +- docs/src/testing_an_implementation.md | 18 ++--- src/traits.jl | 2 +- 9 files changed, 193 insertions(+), 39 deletions(-) create mode 100644 docs/src/list_of_public_names.md diff --git a/README.md b/README.md index 5c3a259a..5414dfd4 100644 --- a/README.md +++ b/README.md @@ -6,8 +6,7 @@ A base Julia interface for machine learning and statistics [![Build Status](https://github.com/JuliaAI/LearnAPI.jl/workflows/CI/badge.svg)](https://github.com/JuliaAI/LearnAPI.jl/actions) [![codecov](https://codecov.io/gh/JuliaAI/LearnAPI.jl/graph/badge.svg?token=9IWT9KYINZ)](https://codecov.io/gh/JuliaAI/LearnAPI.jl?branch=dev) [![Docs](https://img.shields.io/badge/docs-dev-blue.svg)](https://juliaai.github.io/LearnAPI.jl/dev/) - -Comprehensive documentation is [here](https://juliaai.github.io/LearnAPI.jl/dev/). +[![Docs](https://img.shields.io/badge/docs-stable-blue.svg)](https://juliaai.github.io/LearnAPI.jl/stable/) New contributions welcome. See the [road map](ROADMAP.md). @@ -24,13 +23,40 @@ predict(model, newdata) Here `learner` specifies the configuration the algorithm (the hyperparameters) while `model` stores learned parameters and any byproducts of algorithm execution. +LearnAPI.jl mostly a few method stubs and lots of documentation. It does not provide +meta-algorithms, such as cross-validation or hyperparameter optimization, but does aim to +support such algorithms. + ## Related packages -- [MLCore.jl](https://github.com/JuliaML/MLCore.jl) ([docs](https://juliaml.github.io/MLCore.jl/stable/api/#Core-API)) +- [MLCore.jl](https://github.com/JuliaML/MLCore.jl): The default sub-sampling API (`getobs`/`numbobs`) for LearnAPI.jl implementations, which supports tables and arrays. - [LearnTestAPI.jl](https://github.com/JuliaAI/LearnTestAPI.jl): Package to test implementations of LearnAPI.jl (but documented here) -- [LearnDataFrontEnds.jl](https://github.com/JuliaAI/LearnDataFrontEnds.jl): for including flexible, user-friendly, data front ends for LearnAPI.jl implementations ([docs](https://juliaai.github.io/stable/)) +- [LearnDataFrontEnds.jl](https://github.com/JuliaAI/LearnDataFrontEnds.jl): For including flexible, user-friendly, data front ends for LearnAPI.jl implementations ([docs](https://juliaai.github.io/stable/)) + +- [StatisticalMeasures.jl](https://github.com/JuliaAI/StatisticalMeasures.jl): Package providing metrics, compatible with LearnAPI.jl + +### Selected packages providing alternative API's + +The following alphabetical list of packages provide public base API's. Some provide +additional functionality. PR's to add missing items very welcome. + +- [AutoMLPipeline.jl](https://github.com/IBM/AutoMLPipeline.jl) + +- [BetaML.jl](https://github.com/sylvaticus/BetaML.jl) + +- [FastAI.jl](https://github.com/FluxML/FastAI.jl) (focused on deep learning) + +- [LearnBase.jl](https://github.com/JuliaML/LearnBase.jl) (now archived but of historical interest) + +- [MLJModelInterface.jl](https://github.com/JuliaAI/MLJModelInterface.jl) + +- [ScikitLearn.jl](https://github.com/cstjean/ScikitLearn.jl) (an API in addition to being a wrapper for [scikit-learn](https://scikit-learn.org/stable/) + +- [StatsAPI.jl](https://github.com/JuliaStats/StatsAPI.jl/blob/main/src/regressionmodel.jl) (specialized to needs of traditional statistical models) + +- [MLUtils.jl](https://github.com/JuliaML/MLUtils.jl) (more than a base API, and focused on deep learning) ## Credits diff --git a/docs/make.jl b/docs/make.jl index 5bc8fdbb..66e71113 100644 --- a/docs/make.jl +++ b/docs/make.jl @@ -17,6 +17,7 @@ makedocs( "Anatomy of an Implementation" => "anatomy_of_an_implementation.md", "Reference" => [ "Overview" => "reference.md", + "Public Names" => "list_of_public_names.md", "fit/update" => "fit_update.md", "predict/transform" => "predict_transform.md", "Kinds of Target Proxy" => "kinds_of_target_proxy.md", diff --git a/docs/src/anatomy_of_an_implementation.md b/docs/src/anatomy_of_an_implementation.md index 4eafe265..b621344b 100644 --- a/docs/src/anatomy_of_an_implementation.md +++ b/docs/src/anatomy_of_an_implementation.md @@ -555,8 +555,8 @@ above. Here we must explicitly overload them, so that they also handle the outpu ```@example anatomy2 LearnAPI.features(::Ridge, observations::RidgeFitObs) = observations.A -LearnAPI.target(::Ridge, observations::RidgeFitObs) = observations.y LearnAPI.features(learner::Ridge, data) = LearnAPI.features(learner, obs(learner, data)) +LearnAPI.target(::Ridge, observations::RidgeFitObs) = observations.y LearnAPI.target(learner::Ridge, data) = LearnAPI.target(learner, obs(learner, data)) ``` diff --git a/docs/src/examples.md b/docs/src/examples.md index dea9bc56..49932084 100644 --- a/docs/src/examples.md +++ b/docs/src/examples.md @@ -4,7 +4,8 @@ Below is the complete source code for the ridge implementations described in the [Anatomy of an Implementation](@ref). - [Basic implementation](@ref) -- [Implementation with data front end](@ref) +- [Implementation with a data front end](@ref) +- [Implementation with a canned data front end](@ref) ## Basic implementation @@ -85,7 +86,7 @@ LearnAPI.strip(model::RidgeFitted) = LearnAPI.fit(learner::Ridge, X, y; kwargs...) = fit(learner, (X, y); kwargs...) ``` -# Implementation with data front end +# Implementation with a data front end ```julia using LearnAPI @@ -190,3 +191,91 @@ LearnAPI.strip(model::RidgeFitted) = ) ``` + +# Implementation with a canned data front end + +The following implements the `Saffron` data front end from +[LearnDataFrontEnds.jl](https://juliaai.github.io/LearnDataFrontEnds.jl/stable/), which +allows for a greater variety of forms of input to `fit` and `predict`. Refer to that +package's [documentation](https://juliaai.github.io/LearnDataFrontEnds.jl/stable/) for details. + +```julia +using LearnAPI +import LearnDataFrontEnds as FrontEnds +using LinearAlgebra, Tables + +struct Ridge{T<:Real} + lambda::T +end + +Ridge(; lambda=0.1) = Ridge(lambda) + +# struct for output of `fit`: +struct RidgeFitted{T,F} + learner::Ridge + coefficients::Vector{T} + named_coefficients::F +end + +frontend = FrontEnds.Saffron() + +# these will return objects of type `FrontEnds.Obs`: +LearnAPI.obs(learner::Ridge, data) = FrontEnds.fitobs(learner, data, frontend) +LearnAPI.obs(model::RidgeFitted, data) = obs(model, data, frontend) + +function LearnAPI.fit(learner::Ridge, observations::FrontEnds.Obs; verbosity=1) + + lambda = learner.lambda + + A = observations.features + names = observations.names + y = observations.target + + # apply core learner: + coefficients = (A*A' + learner.lambda*I)\(A*y) # 1 x p matrix + + # determine named coefficients: + named_coefficients = [names[j] => coefficients[j] for j in eachindex(names)] + + # make some noise, if allowed: + verbosity > 0 && @info "Coefficients: $named_coefficients" + + return RidgeFitted(learner, coefficients, named_coefficients) + +end +LearnAPI.fit(learner::Ridge, data; kwargs...) = + fit(learner, obs(learner, data); kwargs...) + +LearnAPI.predict(model::RidgeFitted, ::Point, observations::FrontEnds.Obs) = + (observations.features)'*model.coefficients +LearnAPI.predict(model::RidgeFitted, ::Point, Xnew) = + predict(model, Point(), obs(model, Xnew)) + +# training data deconstructors: +LearnAPI.features(learner::Ridge, data) = LearnAPI.features(learner, data, frontend) +LearnAPI.target(learner::Ridge, data) = LearnAPI.target(learner, data, frontend) + +# accessor functions: +LearnAPI.learner(model::RidgeFitted) = model.learner +LearnAPI.coefficients(model::RidgeFitted) = model.named_coefficients +LearnAPI.strip(model::RidgeFitted) = + RidgeFitted(model.learner, model.coefficients, nothing) + +@trait( + Ridge, + constructor = Ridge, + kinds_of_proxy=(Point(),), + tags = ("regression",), + functions = ( + :(LearnAPI.fit), + :(LearnAPI.learner), + :(LearnAPI.clone), + :(LearnAPI.strip), + :(LearnAPI.obs), + :(LearnAPI.features), + :(LearnAPI.target), + :(LearnAPI.predict), + :(LearnAPI.coefficients), + ) +) +``` diff --git a/docs/src/index.md b/docs/src/index.md index a862aee7..87237d86 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -29,17 +29,6 @@ includes a number of Julia [traits](@ref traits) for promising specific behavior LearnAPI.jl's has no package dependencies. -```@raw html -🚧 -``` - -!!! warning - - The API described here is under active development and not ready for adoption. - Join an ongoing design discussion at - [this](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048) - Julia Discourse thread. - ## Sample workflow @@ -47,7 +36,7 @@ Suppose `forest` is some object encapsulating the hyperparameters of the [random algorithm](https://en.wikipedia.org/wiki/Random_forest) (the number of trees, etc.). Then, a LearnAPI.jl interface can be implemented, for objects with the type of `forest`, to enable the basic workflow below. In this case data is presented following the -"scikit-learn" `X, y` pattern, although LearnAPI.jl supports other data pattern. +"scikit-learn" `X, y` pattern, although LearnAPI.jl supports other data patterns. ```julia # `X` is some training features @@ -58,7 +47,7 @@ enable the basic workflow below. In this case data is presented following the @functions forest # Train: -model = fit(forest, X, y) +model = fit(forest, (X, y)) # Generate point predictions: ŷ = predict(model, Xnew) # or `predict(model, Point(), Xnew)` @@ -81,16 +70,16 @@ on the usual supervised/unsupervised learning dichotomy. From this point of view supervised learner is simply one in which a target variable exists, and happens to appear as an input to training but not to prediction. -## Data interfaces +## Data interfaces and front ends Algorithms are free to consume data in any format. However, a method called [`obs`](@ref data_interface) (read as "observations") gives developers the option of providing a separate data front end for their algorithms. In this case `obs` gives users and meta-algorithms access to an algorithm-specific representation of input data, which is -additionally guaranteed to implement a standard interface for accessing individual observations, -unless the algorithm explicitly opts out. Moreover, the `fit` and `predict` methods will -also be able to consume these alternative data representations, for performance benefits -in some situations. +additionally guaranteed to implement a standard interface for accessing individual +observations, unless the algorithm explicitly opts out. Moreover, the `fit` and `predict` +methods can directly consume these alternative data representations, for performance +benefits in some situations, such as cross-validation. The fallback data interface is the [MLCore.jl](https://github.com/JuliaML/MLCore.jl) `getobs/numobs` interface (previously provided by MLUtils.jl) here tagged as diff --git a/docs/src/list_of_public_names.md b/docs/src/list_of_public_names.md new file mode 100644 index 00000000..0e224fe2 --- /dev/null +++ b/docs/src/list_of_public_names.md @@ -0,0 +1,49 @@ +# List of Public Names + +## Core methods + +- [`fit`](@ref) + +- [`update`](@ref) + +- [`update_observations`](@ref) + +- [`predict`](@ref) + +- [`transform`](@ref) + +- [`inverse_transform`](@ref) + +- [`obs`](@ref) + +## Training data deconstructors + +- [`LearnAPI.features`](@ref) + +- [`LearnAPI.target`](@ref) + +- [`LearnAPI.weights`](@ref) + + +## Accessor functions + +See [here](@ref accessor_functions). + + +## Learner traits + +See [here](@ref traits). + + +## Kinds of target proxy + +See [here](@ref proxy_types). + + +## Utilities (never overloaded) + +- [`clone`](@ref): for cloning a learner with specified hyperparameter replacements. + +- [`@trait`](@ref): for simultaneously declaring multiple traits + +- [`@functions`](@ref): for listing functions available for use with a learner diff --git a/docs/src/reference.md b/docs/src/reference.md index 8a18bf6f..a4499e55 100644 --- a/docs/src/reference.md +++ b/docs/src/reference.md @@ -1,8 +1,10 @@ # [Reference](@id reference) Here we give the definitive specification of the LearnAPI.jl interface. For informal -guides see [Anatomy of an Implementation](@ref) and [Common Implementation -Patterns](@ref patterns). +guides see [Anatomy of an Implementation](@ref) and [Common Implementation Patterns](@ref +patterns). + + - [List of Public Names](@ref) ## [Important terms and concepts](@id scope) @@ -190,7 +192,7 @@ Most learners will also implement [`predict`](@ref) and/or [`transform`](@ref). - [`LearnAPI.features`](@ref input), [`LearnAPI.target`](@ref input), [`LearnAPI.weights`](@ref input): for extracting relevant parts of training data, where - defined. + defined. Also called *training data deconstructors*. - [Accessor functions](@ref accessor_functions): these include functions like `LearnAPI.feature_importances` and `LearnAPI.training_losses`, for extracting, from diff --git a/docs/src/testing_an_implementation.md b/docs/src/testing_an_implementation.md index 24f78701..cc0d58f6 100644 --- a/docs/src/testing_an_implementation.md +++ b/docs/src/testing_an_implementation.md @@ -1,6 +1,6 @@ # Testing an Implementation -Testing is provided by the LearnTestAPI.jl package documented below. +Testing is provided by the LearnTestAPI.jl package documented below. ## Quick start @@ -8,15 +8,13 @@ Testing is provided by the LearnTestAPI.jl package documented below. LearnTestAPI ``` -LearnAPI.jl and LearnTestAPI.jl have synchronized releases. For example, LearnTestAPI.jl -version 0.2.3 will generally support all LearnAPI.jl versions 0.2.*. - !!! warning - New releases of LearnTestAPI.jl may add tests to `@testapi`, and this may result in - new failures in client package test suites. Nevertheless, adding a test to `@testapi` - is not considered a breaking change to LearnTestAPI, unless the addition supports a - breaking release of LearnAPI.jl. + New releases of LearnTestAPI.jl may add tests to `@testapi`, and + this may result in new failures in client package test suites, because + of previously undetected broken contracts. Adding a test to `@testapi` + is not considered a breaking change + to LearnTestAPI, unless it supports a breaking change to LearnAPI.jl. ## The @testapi macro @@ -28,7 +26,7 @@ LearnTestAPI.@testapi ## Learners for testing LearnTestAPI.jl provides some simple, tested, LearnAPI.jl implementations, which may be -useful for testing learner wrappers and meta-algorithms. +useful for testing learner wrappers and meta-algorithms. ```@docs LearnTestAPI.Ridge @@ -44,7 +42,7 @@ LearnTestAPI.StumpRegressor ## Private methods -For LearnTestAPI.jl developers only, and subject to breaking changes: +For LearnTestAPI.jl developers only, and subject to breaking changes at any time: ```@docs LearnTestAPI.@logged_testset diff --git a/src/traits.jl b/src/traits.jl index e85af904..54cdacfc 100644 --- a/src/traits.jl +++ b/src/traits.jl @@ -65,7 +65,7 @@ Return a tuple of expressions representing functions that can be meaningfully ap argument. Learner traits (methods for which `learner` is the *only* argument) are excluded. -To return actual functions, instead of symbols, use [`@functions`](@ref)` learner` +To return actual functions, instead of symbols, use [`@functions`](@ref) `learner` instead. The returned tuple may include expressions like `:(DecisionTree.print_tree)`, which From 594e737fd730f9794f94055670ee47feb3209efa Mon Sep 17 00:00:00 2001 From: "Anthony D. Blaom" Date: Tue, 18 Feb 2025 13:38:39 +1300 Subject: [PATCH 07/12] bump 1.0 --- Project.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Project.toml b/Project.toml index d791a46e..831e208e 100644 --- a/Project.toml +++ b/Project.toml @@ -1,7 +1,7 @@ name = "LearnAPI" uuid = "92ad9a40-7767-427a-9ee6-6e577f1266cb" authors = ["Anthony D. Blaom "] -version = "0.2.0" +version = "1.0.0" [compat] julia = "1.10" From e57d5a4cbcc53f87cc4f2ff31d9baf46b4a5e98a Mon Sep 17 00:00:00 2001 From: "Anthony D. Blaom" Date: Tue, 18 Feb 2025 16:38:47 +1300 Subject: [PATCH 08/12] tweak readme --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 5414dfd4..4969197c 100644 --- a/README.md +++ b/README.md @@ -52,11 +52,11 @@ additional functionality. PR's to add missing items very welcome. - [MLJModelInterface.jl](https://github.com/JuliaAI/MLJModelInterface.jl) -- [ScikitLearn.jl](https://github.com/cstjean/ScikitLearn.jl) (an API in addition to being a wrapper for [scikit-learn](https://scikit-learn.org/stable/) +- [ScikitLearn.jl](https://github.com/cstjean/ScikitLearn.jl) (an API in addition to being a wrapper for [scikit-learn](https://scikit-learn.org/stable/)) - [StatsAPI.jl](https://github.com/JuliaStats/StatsAPI.jl/blob/main/src/regressionmodel.jl) (specialized to needs of traditional statistical models) -- [MLUtils.jl](https://github.com/JuliaML/MLUtils.jl) (more than a base API, and focused on deep learning) +- [MLUtils.jl](https://github.com/JuliaML/MLUtils.jl) (more than a base API, focused on deep learning) ## Credits From 646aef3cbcc4a0649154a2a7c190bc91f35415c0 Mon Sep 17 00:00:00 2001 From: "Anthony D. Blaom" Date: Tue, 18 Feb 2025 16:41:05 +1300 Subject: [PATCH 09/12] fix order --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 4969197c..80979383 100644 --- a/README.md +++ b/README.md @@ -52,12 +52,12 @@ additional functionality. PR's to add missing items very welcome. - [MLJModelInterface.jl](https://github.com/JuliaAI/MLJModelInterface.jl) +- [MLUtils.jl](https://github.com/JuliaML/MLUtils.jl) (more than a base API, focused on deep learning) + - [ScikitLearn.jl](https://github.com/cstjean/ScikitLearn.jl) (an API in addition to being a wrapper for [scikit-learn](https://scikit-learn.org/stable/)) - [StatsAPI.jl](https://github.com/JuliaStats/StatsAPI.jl/blob/main/src/regressionmodel.jl) (specialized to needs of traditional statistical models) -- [MLUtils.jl](https://github.com/JuliaML/MLUtils.jl) (more than a base API, focused on deep learning) - ## Credits From b1341eb403cb0e5fe733b83088cc88421ca641f4 Mon Sep 17 00:00:00 2001 From: "Anthony D. Blaom" Date: Tue, 18 Feb 2025 16:53:21 +1300 Subject: [PATCH 10/12] typo --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 80979383..ec6d4d27 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ predict(model, newdata) Here `learner` specifies the configuration the algorithm (the hyperparameters) while `model` stores learned parameters and any byproducts of algorithm execution. -LearnAPI.jl mostly a few method stubs and lots of documentation. It does not provide +LearnAPI.jl is mostly a few method stubs and lots of documentation. It does not provide meta-algorithms, such as cross-validation or hyperparameter optimization, but does aim to support such algorithms. From 720038bfacce847c70b0f32cc62c97e7516b6e45 Mon Sep 17 00:00:00 2001 From: "Anthony D. Blaom" Date: Tue, 18 Feb 2025 16:54:18 +1300 Subject: [PATCH 11/12] doc tweak --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index ec6d4d27..624fd063 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ predict(model, newdata) Here `learner` specifies the configuration the algorithm (the hyperparameters) while `model` stores learned parameters and any byproducts of algorithm execution. -LearnAPI.jl is mostly a few method stubs and lots of documentation. It does not provide +LearnAPI.jl is mostly method stubs and lots of documentation. It does not provide meta-algorithms, such as cross-validation or hyperparameter optimization, but does aim to support such algorithms. @@ -40,7 +40,7 @@ support such algorithms. ### Selected packages providing alternative API's The following alphabetical list of packages provide public base API's. Some provide -additional functionality. PR's to add missing items very welcome. +additional functionality. PR's to add missing items welcome. - [AutoMLPipeline.jl](https://github.com/IBM/AutoMLPipeline.jl) From ed60baf13c378714e0484d124c2aca686796b966 Mon Sep 17 00:00:00 2001 From: "Anthony D. Blaom" Date: Tue, 18 Feb 2025 17:35:34 +1300 Subject: [PATCH 12/12] fix a ref --- docs/src/anatomy_of_an_implementation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/src/anatomy_of_an_implementation.md b/docs/src/anatomy_of_an_implementation.md index b621344b..338f61b8 100644 --- a/docs/src/anatomy_of_an_implementation.md +++ b/docs/src/anatomy_of_an_implementation.md @@ -105,7 +105,7 @@ nothing # hide ``` Note that we also include `learner` in the struct, for it must be possible to recover -`learner` from the output of `fit`; see [Accessor functions](@ref) below. +`learner` from the output of `fit`; see [Accessor functions](@ref af) below. The implementation of `fit` looks like this: @@ -159,7 +159,7 @@ first element of the tuple returned by [`LearnAPI.kinds_of_proxy(learner)`](@ref we overload appropriately below. -### Accessor functions +### [Accessor functions](@id af) An [accessor function](@ref accessor_functions) has the output of [`fit`](@ref) as it's sole argument. Every new implementation must implement the accessor function