diff --git a/Project.toml b/Project.toml index d791a46e..831e208e 100644 --- a/Project.toml +++ b/Project.toml @@ -1,7 +1,7 @@ name = "LearnAPI" uuid = "92ad9a40-7767-427a-9ee6-6e577f1266cb" authors = ["Anthony D. Blaom "] -version = "0.2.0" +version = "1.0.0" [compat] julia = "1.10" diff --git a/README.md b/README.md index 7ef89a62..2554142e 100644 --- a/README.md +++ b/README.md @@ -6,45 +6,58 @@ A base Julia interface for machine learning and statistics [![Build Status](https://github.com/JuliaAI/LearnAPI.jl/workflows/CI/badge.svg)](https://github.com/JuliaAI/LearnAPI.jl/actions) [![codecov](https://codecov.io/gh/JuliaAI/LearnAPI.jl/graph/badge.svg?token=9IWT9KYINZ)](https://codecov.io/gh/JuliaAI/LearnAPI.jl?branch=dev) [![Docs](https://img.shields.io/badge/docs-dev-blue.svg)](https://juliaai.github.io/LearnAPI.jl/dev/) - -Comprehensive documentation is [here](https://juliaai.github.io/LearnAPI.jl/dev/). +[![Docs](https://img.shields.io/badge/docs-stable-blue.svg)](https://juliaai.github.io/LearnAPI.jl/stable/) New contributions welcome. See the [road map](ROADMAP.md). -## Code snippet +## Synopsis -Configure a machine learning algorithm: +LearnAPI.jl provides for variations and elaborations on the following basic pattern in machine +learning and statistics: ```julia -julia> ridge = Ridge(lambda=0.1) +model = fit(learner, data) +predict(model, newdata) ``` -Inspect available functionality: +Here `learner` specifies the configuration the algorithm (the hyperparameters) while +`model` stores learned parameters and any byproducts of algorithm execution. -``` -julia> @functions ridge -(fit, LearnAPI.learner, LearnAPI.strip, obs, LearnAPI.features, LearnAPI.target, predict, LearnAPI.coefficients) -``` +LearnAPI.jl is mostly method stubs and lots of documentation. It does not provide +meta-algorithms, such as cross-validation or hyperparameter optimization, but does aim to +support such algorithms. -Train: +## Related packages -```julia -julia> model = fit(ridge, data) -``` +- [MLCore.jl](https://github.com/JuliaML/MLCore.jl): The default sub-sampling API (`getobs`/`numbobs`) for LearnAPI.jl implementations, which supports tables and arrays. -Predict: +- [LearnTestAPI.jl](https://github.com/JuliaAI/LearnTestAPI.jl): Package to test implementations of LearnAPI.jl (but documented here) -```julia -julia> predict(model, data)[1] -"virginica" -``` +- [LearnDataFrontEnds.jl](https://github.com/JuliaAI/LearnDataFrontEnds.jl): For including flexible, user-friendly, data front ends for LearnAPI.jl implementations ([docs](https://juliaai.github.io/LearnDataFrontEnds.jl/stable/)) -Predict a probability distribution ([proxy](https://juliaai.github.io/LearnAPI.jl/dev/kinds_of_target_proxy/#proxy_types) for the target): +- [StatisticalMeasures.jl](https://github.com/JuliaAI/StatisticalMeasures.jl): Package providing metrics, compatible with LearnAPI.jl + +### Selected packages providing alternative API's + +The following alphabetical list of packages provide public base API's. Some provide +additional functionality. PR's to add missing items welcome. + +- [AutoMLPipeline.jl](https://github.com/IBM/AutoMLPipeline.jl) + +- [BetaML.jl](https://github.com/sylvaticus/BetaML.jl) + +- [FastAI.jl](https://github.com/FluxML/FastAI.jl) (focused on deep learning) + +- [LearnBase.jl](https://github.com/JuliaML/LearnBase.jl) (now archived but of historical interest) + +- [MLJModelInterface.jl](https://github.com/JuliaAI/MLJModelInterface.jl) + +- [MLUtils.jl](https://github.com/JuliaML/MLUtils.jl) (more than a base API, focused on deep learning) + +- [ScikitLearn.jl](https://github.com/cstjean/ScikitLearn.jl) (an API in addition to being a wrapper for [scikit-learn](https://scikit-learn.org/stable/)) + +- [StatsAPI.jl](https://github.com/JuliaStats/StatsAPI.jl/tree/main) (specialized to needs of traditional statistical models) -```julia -julia> predict(model, Distribution(), data)[1] -UnivariateFinite{Multiclass{3}}(setosa=>0.0, versicolor=>0.25, virginica=>0.75) -``` ## Credits diff --git a/ROADMAP.md b/ROADMAP.md index 7da578ae..98de84fe 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,12 +1,12 @@ # Road map -- [ ] Mock up a challenging `update` use-case: controlling an iterative algorithm that +- [x] Mock up a challenging `update` use-case: controlling an iterative algorithm that wants, for efficiency, to internally compute the out-of-sample predictions that will be for used to *externally* determined early stopping cc: @jeremiedb - [ ] Get code coverage to 100% (see next item) -- [ ] Add to this repo or a utility repo methods to test a valid implementation of +- [x] Add to this repo or a utility repo methods to test a valid implementation of LearnAPI.jl - [ ] Flush out "Common Implementation Patterns". The current plan is to mock up example @@ -18,17 +18,17 @@ - [ ] clustering - [x] gradient descent - [x] iterative algorithms - - [ ] incremental algorithms - - [ ] dimension reduction + - [x] incremental algorithms + - [x] dimension reduction - [x] feature engineering - [x] static algorithms - [ ] missing value imputation - - [ ] transformers + - [x] transformers - [x] ensemble algorithms - [ ] time series forecasting - [ ] time series classification - [ ] survival analysis - - [ ] density estimation + - [x] density estimation - [ ] Bayesian algorithms - [ ] outlier detection - [ ] collaborative filtering @@ -36,10 +36,10 @@ - [ ] audio analysis - [ ] natural language processing - [ ] image processing - - [ ] meta-algorithms + - [x] meta-algorithms -- [ ] In a utility package provide: - - [ ] Methods to facilitate common-use case data interfaces: support simultaneously +- [x] In a utility package provide: + - [x] Methods to facilitate common-use case data interfaces: support simultaneously `fit` data of the form `data = (X, y)` where `X` is table *or* matrix, and `data` a table with target specified by hyperparameter; here `obs` will return a thin wrapping of the matrix of `X`, the target `y`, and the names of all fields. We can have diff --git a/docs/Project.toml b/docs/Project.toml index 4d8ef094..4ffd503c 100644 --- a/docs/Project.toml +++ b/docs/Project.toml @@ -2,7 +2,8 @@ Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4" DocumenterInterLinks = "d12716ef-a0f6-4df4-a9f1-a5a34e75c656" LearnAPI = "92ad9a40-7767-427a-9ee6-6e577f1266cb" -MLUtils = "f1d291b0-491e-4a28-83b9-f70985020b54" +LearnTestAPI = "3111ed91-c4f2-40e7-bb19-7f6c618409b8" +MLCore = "c2834f40-e789-41da-a90e-33b280584a8c" ScientificTypesBase = "30f210dd-8aff-4c5f-94ba-8e64358c1161" Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c" diff --git a/docs/make.jl b/docs/make.jl index 95e0480a..66e71113 100644 --- a/docs/make.jl +++ b/docs/make.jl @@ -2,11 +2,12 @@ using Documenter using LearnAPI using ScientificTypesBase using DocumenterInterLinks +using LearnTestAPI const REPO = Remotes.GitHub("JuliaAI", "LearnAPI.jl") makedocs( - modules=[LearnAPI,], + modules=[LearnAPI, LearnTestAPI], format=Documenter.HTML( prettyurls = true,#get(ENV, "CI", nothing) == "true", collapselevel = 1, @@ -16,6 +17,7 @@ makedocs( "Anatomy of an Implementation" => "anatomy_of_an_implementation.md", "Reference" => [ "Overview" => "reference.md", + "Public Names" => "list_of_public_names.md", "fit/update" => "fit_update.md", "predict/transform" => "predict_transform.md", "Kinds of Target Proxy" => "kinds_of_target_proxy.md", diff --git a/docs/src/anatomy_of_an_implementation.md b/docs/src/anatomy_of_an_implementation.md index e6dba45a..338f61b8 100644 --- a/docs/src/anatomy_of_an_implementation.md +++ b/docs/src/anatomy_of_an_implementation.md @@ -105,7 +105,7 @@ nothing # hide ``` Note that we also include `learner` in the struct, for it must be possible to recover -`learner` from the output of `fit`; see [Accessor functions](@ref) below. +`learner` from the output of `fit`; see [Accessor functions](@ref af) below. The implementation of `fit` looks like this: @@ -159,7 +159,7 @@ first element of the tuple returned by [`LearnAPI.kinds_of_proxy(learner)`](@ref we overload appropriately below. -### Accessor functions +### [Accessor functions](@id af) An [accessor function](@ref accessor_functions) has the output of [`fit`](@ref) as it's sole argument. Every new implementation must implement the accessor function @@ -334,7 +334,7 @@ assumptions about data from those made above. - If the `data` object consumed by `fit`, `predict`, or `transform` is not not a suitable table¹, array³, tuple of tables and arrays, or some other object implementing the - [MLUtils.jl](https://juliaml.github.io/MLUtils.jl/dev/) `getobs`/`numobs` interface, + [MLCore.jl](https://juliaml.github.io/MLCore.jl/dev/) `getobs`/`numobs` interface, then an implementation must: (i) overload [`obs`](@ref) to articulate how provided data can be transformed into a form that does support this interface, as illustrated below under [Providing a separate data front end](@ref) below; or (ii) overload the trait @@ -419,7 +419,7 @@ The [`obs`](@ref) methods exist to: how it works. In the typical case, where [`LearnAPI.data_interface`](@ref) is not overloaded, the -alternative data representations must implement the MLUtils.jl `getobs/numobs` interface +alternative data representations must implement the MLCore.jl `getobs/numobs` interface for observation subsampling, which is generally all a user or meta-algorithm will need, before passing the data on to `fit`/`predict`, as you would the original data. @@ -436,14 +436,14 @@ one enables the following alternative: observations = obs(learner, data) # preprocessed training data # optional subsampling: -observations = MLUtils.getobs(observations, train_indices) +observations = MLCore.getobs(observations, train_indices) model = fit(learner, observations) newobservations = obs(model, newdata) # optional subsampling: -newobservations = MLUtils.getobs(observations, test_indices) +newobservations = MLCore.getobs(observations, test_indices) predict(model, newobservations) ``` @@ -555,8 +555,8 @@ above. Here we must explicitly overload them, so that they also handle the outpu ```@example anatomy2 LearnAPI.features(::Ridge, observations::RidgeFitObs) = observations.A -LearnAPI.target(::Ridge, observations::RidgeFitObs) = observations.y LearnAPI.features(learner::Ridge, data) = LearnAPI.features(learner, obs(learner, data)) +LearnAPI.target(::Ridge, observations::RidgeFitObs) = observations.y LearnAPI.target(learner::Ridge, data) = LearnAPI.target(learner, obs(learner, data)) ``` @@ -568,7 +568,7 @@ LearnAPI.target(learner::Ridge, data) = LearnAPI.target(learner, obs(learner, da are generally different. - We need the adjoint operator, `'`, because the last dimension in arrays is the - observation dimension, according to the MLUtils.jl convention. Remember, `Xnew` is a + observation dimension, according to the MLCore.jl convention. Remember, `Xnew` is a table here. Since LearnAPI.jl provides fallbacks for `obs` that simply return the unadulterated data @@ -576,7 +576,7 @@ argument, overloading `obs` is optional. This is provided data in publicized `fit`/`predict` signatures already consists only of objects implement the [`LearnAPI.RandomAccess`](@ref) interface (most tables¹, arrays³, and tuples thereof). -To opt out of supporting the MLUtils.jl interface altogether, an implementation must +To opt out of supporting the MLCore.jl interface altogether, an implementation must overload the trait, [`LearnAPI.data_interface(learner)`](@ref). See [Data interfaces](@ref data_interfaces) for details. @@ -593,15 +593,15 @@ LearnAPI.fit(learner::Ridge, X, y; kwargs...) = fit(learner, (X, y); kwargs...) ## [Demonstration of an advanced `obs` workflow](@id advanced_demo) We now can train and predict using internal data representations, resampled using the -generic MLUtils.jl interface: +generic MLCore.jl interface: ```@example anatomy2 -import MLUtils +import MLCore learner = Ridge() observations_for_fit = obs(learner, (X, y)) -model = fit(learner, MLUtils.getobs(observations_for_fit, train)) +model = fit(learner, MLCore.getobs(observations_for_fit, train)) observations_for_predict = obs(model, X) -ẑ = predict(model, MLUtils.getobs(observations_for_predict, test)) +ẑ = predict(model, MLCore.getobs(observations_for_predict, test)) ``` ```julia @@ -616,7 +616,7 @@ obs_workflows). ¹ In LearnAPI.jl a *table* is any object `X` implementing the [Tables.jl](https://tables.juliadata.org/dev/) interface, additionally satisfying `Tables.istable(X) == true` and implementing `DataAPI.nrow` (and whence -`MLUtils.numobs`). Tables that are also (unnamed) tuples are disallowed. +`MLCore.numobs`). Tables that are also (unnamed) tuples are disallowed. ² An implementation can provide further accessor functions, if necessary, but like the native ones, they must be included in the [`LearnAPI.functions`](@ref) diff --git a/docs/src/examples.md b/docs/src/examples.md index dea9bc56..49932084 100644 --- a/docs/src/examples.md +++ b/docs/src/examples.md @@ -4,7 +4,8 @@ Below is the complete source code for the ridge implementations described in the [Anatomy of an Implementation](@ref). - [Basic implementation](@ref) -- [Implementation with data front end](@ref) +- [Implementation with a data front end](@ref) +- [Implementation with a canned data front end](@ref) ## Basic implementation @@ -85,7 +86,7 @@ LearnAPI.strip(model::RidgeFitted) = LearnAPI.fit(learner::Ridge, X, y; kwargs...) = fit(learner, (X, y); kwargs...) ``` -# Implementation with data front end +# Implementation with a data front end ```julia using LearnAPI @@ -190,3 +191,91 @@ LearnAPI.strip(model::RidgeFitted) = ) ``` + +# Implementation with a canned data front end + +The following implements the `Saffron` data front end from +[LearnDataFrontEnds.jl](https://juliaai.github.io/LearnDataFrontEnds.jl/stable/), which +allows for a greater variety of forms of input to `fit` and `predict`. Refer to that +package's [documentation](https://juliaai.github.io/LearnDataFrontEnds.jl/stable/) for details. + +```julia +using LearnAPI +import LearnDataFrontEnds as FrontEnds +using LinearAlgebra, Tables + +struct Ridge{T<:Real} + lambda::T +end + +Ridge(; lambda=0.1) = Ridge(lambda) + +# struct for output of `fit`: +struct RidgeFitted{T,F} + learner::Ridge + coefficients::Vector{T} + named_coefficients::F +end + +frontend = FrontEnds.Saffron() + +# these will return objects of type `FrontEnds.Obs`: +LearnAPI.obs(learner::Ridge, data) = FrontEnds.fitobs(learner, data, frontend) +LearnAPI.obs(model::RidgeFitted, data) = obs(model, data, frontend) + +function LearnAPI.fit(learner::Ridge, observations::FrontEnds.Obs; verbosity=1) + + lambda = learner.lambda + + A = observations.features + names = observations.names + y = observations.target + + # apply core learner: + coefficients = (A*A' + learner.lambda*I)\(A*y) # 1 x p matrix + + # determine named coefficients: + named_coefficients = [names[j] => coefficients[j] for j in eachindex(names)] + + # make some noise, if allowed: + verbosity > 0 && @info "Coefficients: $named_coefficients" + + return RidgeFitted(learner, coefficients, named_coefficients) + +end +LearnAPI.fit(learner::Ridge, data; kwargs...) = + fit(learner, obs(learner, data); kwargs...) + +LearnAPI.predict(model::RidgeFitted, ::Point, observations::FrontEnds.Obs) = + (observations.features)'*model.coefficients +LearnAPI.predict(model::RidgeFitted, ::Point, Xnew) = + predict(model, Point(), obs(model, Xnew)) + +# training data deconstructors: +LearnAPI.features(learner::Ridge, data) = LearnAPI.features(learner, data, frontend) +LearnAPI.target(learner::Ridge, data) = LearnAPI.target(learner, data, frontend) + +# accessor functions: +LearnAPI.learner(model::RidgeFitted) = model.learner +LearnAPI.coefficients(model::RidgeFitted) = model.named_coefficients +LearnAPI.strip(model::RidgeFitted) = + RidgeFitted(model.learner, model.coefficients, nothing) + +@trait( + Ridge, + constructor = Ridge, + kinds_of_proxy=(Point(),), + tags = ("regression",), + functions = ( + :(LearnAPI.fit), + :(LearnAPI.learner), + :(LearnAPI.clone), + :(LearnAPI.strip), + :(LearnAPI.obs), + :(LearnAPI.features), + :(LearnAPI.target), + :(LearnAPI.predict), + :(LearnAPI.coefficients), + ) +) +``` diff --git a/docs/src/fit_update.md b/docs/src/fit_update.md index 8e27126c..2329d494 100644 --- a/docs/src/fit_update.md +++ b/docs/src/fit_update.md @@ -15,9 +15,9 @@ clustering algorithms); there is no training data and heavy lifting is carried o ### Updating ``` -update(model, data; verbosity=..., param1=new_value1, param2=new_value2, ...) -> updated_model -update_observations(model, new_data; verbosity=..., param1=new_value1, ...) -> updated_model -update_features(model, new_data; verbosity=..., param1=new_value1, ...) -> updated_model +update(model, data; verbosity=..., :param1=new_value1, :param2=new_value2, ...) -> updated_model +update_observations(model, new_data; verbosity=..., :param1=new_value1, ...) -> updated_model +update_features(model, new_data; verbosity=..., :param1=new_value1, ...) -> updated_model ``` ## Typical workflows diff --git a/docs/src/index.md b/docs/src/index.md index 0d10db0f..87237d86 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -4,10 +4,10 @@
Tutorial  |  - Reference  |  Patterns + Reference  | 
@@ -29,17 +29,6 @@ includes a number of Julia [traits](@ref traits) for promising specific behavior LearnAPI.jl's has no package dependencies. -```@raw html -🚧 -``` - -!!! warning - - The API described here is under active development and not ready for adoption. - Join an ongoing design discussion at - [this](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048) - Julia Discourse thread. - ## Sample workflow @@ -47,7 +36,7 @@ Suppose `forest` is some object encapsulating the hyperparameters of the [random algorithm](https://en.wikipedia.org/wiki/Random_forest) (the number of trees, etc.). Then, a LearnAPI.jl interface can be implemented, for objects with the type of `forest`, to enable the basic workflow below. In this case data is presented following the -"scikit-learn" `X, y` pattern, although LearnAPI.jl supports other data pattern. +"scikit-learn" `X, y` pattern, although LearnAPI.jl supports other data patterns. ```julia # `X` is some training features @@ -58,7 +47,7 @@ enable the basic workflow below. In this case data is presented following the @functions forest # Train: -model = fit(forest, X, y) +model = fit(forest, (X, y)) # Generate point predictions: ŷ = predict(model, Xnew) # or `predict(model, Point(), Xnew)` @@ -81,21 +70,27 @@ on the usual supervised/unsupervised learning dichotomy. From this point of view supervised learner is simply one in which a target variable exists, and happens to appear as an input to training but not to prediction. -## Data interfaces +## Data interfaces and front ends Algorithms are free to consume data in any format. However, a method called [`obs`](@ref -data_interface) (read as "observations") gives users and meta-algorithms access to an -algorithm-specific representation of input data, which is also guaranteed to implement a -standard interface for accessing individual observations, unless the algorithm explicitly -opts out. Moreover, the `fit` and `predict` methods will also be able to consume these -alternative data representations, for performance benefits in some situations. - -The fallback data interface is the [MLUtils.jl](https://github.com/JuliaML/MLUtils.jl) -`getobs/numobs` interface, here tagged as [`LearnAPI.RandomAccess()`](@ref), and if the -input consumed by the algorithm already implements that interface (tables, arrays, etc.) -then overloading `obs` is completely optional. Plain iteration interfaces, with or without -knowledge of the number of observations, can also be specified, to support, e.g., data -loaders reading images from disk. +data_interface) (read as "observations") gives developers the option of providing a +separate data front end for their algorithms. In this case `obs` gives users and +meta-algorithms access to an algorithm-specific representation of input data, which is +additionally guaranteed to implement a standard interface for accessing individual +observations, unless the algorithm explicitly opts out. Moreover, the `fit` and `predict` +methods can directly consume these alternative data representations, for performance +benefits in some situations, such as cross-validation. + +The fallback data interface is the [MLCore.jl](https://github.com/JuliaML/MLCore.jl) +`getobs/numobs` interface (previously provided by MLUtils.jl) here tagged as +[`LearnAPI.RandomAccess()`](@ref). However, if the input consumed by the algorithm already +implements that interface (tables, arrays, etc.) then overloading `obs` is completely +optional. Plain iteration interfaces, with or without knowledge of the number of +observations, can also be specified, to support, e.g., data loaders reading images from +disk. + +Some canned data front ends (implementations of [`obs`](@ref)) are provided by the +[LearnDataFrontEnds.jl](https://juliaai.github.io/LearnDataFrontEnds.jl/stable/) package. ## Learning more diff --git a/docs/src/list_of_public_names.md b/docs/src/list_of_public_names.md new file mode 100644 index 00000000..0e224fe2 --- /dev/null +++ b/docs/src/list_of_public_names.md @@ -0,0 +1,49 @@ +# List of Public Names + +## Core methods + +- [`fit`](@ref) + +- [`update`](@ref) + +- [`update_observations`](@ref) + +- [`predict`](@ref) + +- [`transform`](@ref) + +- [`inverse_transform`](@ref) + +- [`obs`](@ref) + +## Training data deconstructors + +- [`LearnAPI.features`](@ref) + +- [`LearnAPI.target`](@ref) + +- [`LearnAPI.weights`](@ref) + + +## Accessor functions + +See [here](@ref accessor_functions). + + +## Learner traits + +See [here](@ref traits). + + +## Kinds of target proxy + +See [here](@ref proxy_types). + + +## Utilities (never overloaded) + +- [`clone`](@ref): for cloning a learner with specified hyperparameter replacements. + +- [`@trait`](@ref): for simultaneously declaring multiple traits + +- [`@functions`](@ref): for listing functions available for use with a learner diff --git a/docs/src/obs.md b/docs/src/obs.md index 70b6eb46..c36f4e3d 100644 --- a/docs/src/obs.md +++ b/docs/src/obs.md @@ -26,26 +26,26 @@ observations = obs(learner, data) then, assuming the typical case that `LearnAPI.data_interface(learner) == LearnAPI.RandomAccess()`, `observations` implements the -[MLUtils.jl](https://juliaml.github.io/MLUtils.jl/dev/) `getobs`/`numobs` interface, for +[MLCore.jl](https://juliaml.github.io/MLCore.jl/dev/) `getobs`/`numobs` interface, for grabbing and counting observations. Moreover, we can pass `observations` to `fit` in place -of the original data, or first resample it using `MLUtils.getobs`: +of the original data, or first resample it using `MLCore.getobs`: ```julia # equivalent to `model = fit(learner, data)` model = fit(learner, observations) # with resampling: -resampled_observations = MLUtils.getobs(observations, 1:10) +resampled_observations = MLCore.getobs(observations, 1:10) model = fit(learner, resampled_observations) ``` In some implementations, the alternative pattern above can be used to avoid repeating unnecessary internal data preprocessing, or inefficient resampling. For example, here's -how a user might call `obs` and `MLUtils.getobs` to perform efficient cross-validation: +how a user might call `obs` and `MLCore.getobs` to perform efficient cross-validation: ```julia using LearnAPI -import MLUtils +import MLCore learner = @@ -61,7 +61,7 @@ never_trained = true scores = map(train_test_folds) do (train, test) # train using model-specific representation of data: - fitobs_subset = MLUtils.getobs(fitobs, train) + fitobs_subset = MLCore.getobs(fitobs, train) model = fit(learner, fitobs_subset) # predict on the fold complement: @@ -70,7 +70,7 @@ scores = map(train_test_folds) do (train, test) global predictobs = obs(model, X) global never_trained = false end - predictobs_subset = MLUtils.getobs(predictobs, test) + predictobs_subset = MLCore.getobs(predictobs, test) ŷ = predict(model, Point(), predictobs_subset) y = LearnAPI.target(learner, data) diff --git a/docs/src/patterns/classification.md b/docs/src/patterns/classification.md index fd278478..41fc522e 100644 --- a/docs/src/patterns/classification.md +++ b/docs/src/patterns/classification.md @@ -2,4 +2,6 @@ See these examples from the JuliaTestAPI.jl test suite: -- [perceptron classifier](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/gradient_descent.jl) +- [perceptron classifier](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/gradient_descent.jl) + +- [constant classifier](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/classification.jl) (including `Sage` data front end) diff --git a/docs/src/patterns/density_estimation.md b/docs/src/patterns/density_estimation.md index 9fc0144a..74cad18f 100644 --- a/docs/src/patterns/density_estimation.md +++ b/docs/src/patterns/density_estimation.md @@ -2,4 +2,4 @@ See these examples from the JuliaTestAPI.jl test suite: -- [normal distribution estimator](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/incremental_algorithms.jl) +- [normal distribution estimator](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/incremental_algorithms.jl) diff --git a/docs/src/patterns/dimension_reduction.md b/docs/src/patterns/dimension_reduction.md index e886dd15..e780559d 100644 --- a/docs/src/patterns/dimension_reduction.md +++ b/docs/src/patterns/dimension_reduction.md @@ -2,5 +2,5 @@ See these examples from the JuliaTestAPI.jl test suite: -- [Truncated SVD](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/dimension_reduction.jl) +- [Truncated SVD](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/dimension_reduction.jl) (including `Tarragon` data front end) diff --git a/docs/src/patterns/ensembling.md b/docs/src/patterns/ensembling.md index 8d774f5e..474d32e1 100644 --- a/docs/src/patterns/ensembling.md +++ b/docs/src/patterns/ensembling.md @@ -2,6 +2,6 @@ See these examples from the JuliaTestAPI.jl test suite: -- [bagged ensembling of a regression model](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/ensembling.jl) +- [bagged ensembling of a regression model](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/ensembling.jl) -- [extremely randomized ensemble of decision stumps (regression)](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/ensembling.jl) +- [extremely randomized ensemble of decision stumps (regression)](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/ensembling.jl) diff --git a/docs/src/patterns/feature_engineering.md b/docs/src/patterns/feature_engineering.md index 6e3c656c..1b098475 100644 --- a/docs/src/patterns/feature_engineering.md +++ b/docs/src/patterns/feature_engineering.md @@ -3,5 +3,5 @@ See these examples from the JuliaTestAI.jl test suite: - [feature - selectors](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/static_algorithms.jl) + selectors](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/static_algorithms.jl) from tests. diff --git a/docs/src/patterns/gradient_descent.md b/docs/src/patterns/gradient_descent.md index 9dc5401a..95cd7d00 100644 --- a/docs/src/patterns/gradient_descent.md +++ b/docs/src/patterns/gradient_descent.md @@ -2,4 +2,4 @@ See these examples from the JuliaTestAI.jl test suite: -- [perceptron classifier](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/gradient_descent.jl) +- [perceptron classifier](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/gradient_descent.jl) diff --git a/docs/src/patterns/incremental_algorithms.md b/docs/src/patterns/incremental_algorithms.md index d2855a55..fea6566b 100644 --- a/docs/src/patterns/incremental_algorithms.md +++ b/docs/src/patterns/incremental_algorithms.md @@ -2,4 +2,4 @@ See these examples from the JuliaTestAI.jl test suite: -- [normal distribution estimator](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/incremental_algorithms.jl) +- [normal distribution estimator](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/incremental_algorithms.jl) diff --git a/docs/src/patterns/iterative_algorithms.md b/docs/src/patterns/iterative_algorithms.md index 265dddf7..d7e81ad4 100644 --- a/docs/src/patterns/iterative_algorithms.md +++ b/docs/src/patterns/iterative_algorithms.md @@ -2,8 +2,8 @@ See these examples from the JuliaTestAI.jl test suite: -- [bagged ensembling](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/ensembling.jl) +- [bagged ensembling](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/ensembling.jl) -- [perceptron classifier](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/gradient_descent.jl) +- [perceptron classifier](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/gradient_descent.jl) -- [extremely randomized ensemble of decision stumps (regression)](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/ensembling.jl) +- [extremely randomized ensemble of decision stumps (regression)](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/ensembling.jl) diff --git a/docs/src/patterns/meta_algorithms.md b/docs/src/patterns/meta_algorithms.md index 6a9e7300..d86d7966 100644 --- a/docs/src/patterns/meta_algorithms.md +++ b/docs/src/patterns/meta_algorithms.md @@ -2,6 +2,6 @@ Many meta-algorithms are can be implemented as wrappers. An example is [this bagged ensemble -algorithm](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/ensembling.jl) +algorithm](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/ensembling.jl) from tests. diff --git a/docs/src/patterns/regression.md b/docs/src/patterns/regression.md index a6de5b10..52ae5477 100644 --- a/docs/src/patterns/regression.md +++ b/docs/src/patterns/regression.md @@ -2,6 +2,6 @@ See these examples from the JuliaTestAPI.jl test suite: -- [ridge regression](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/regression.jl) +- [ridge regression](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/regression.jl) (including `Saffron` data front end) -- [extremely randomized ensemble of decision stumps](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/ensembling.jl) +- [extremely randomized ensemble of decision stumps](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/ensembling.jl) diff --git a/docs/src/patterns/static_algorithms.md b/docs/src/patterns/static_algorithms.md index 4724006f..d62ea43d 100644 --- a/docs/src/patterns/static_algorithms.md +++ b/docs/src/patterns/static_algorithms.md @@ -3,7 +3,7 @@ See these examples from the JuliaTestAI.jl test suite: - [feature - selection](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/static_algorithms.jl) + selection](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/static_algorithms.jl) diff --git a/docs/src/patterns/transformers.md b/docs/src/patterns/transformers.md index c27f9682..b207dd2d 100644 --- a/docs/src/patterns/transformers.md +++ b/docs/src/patterns/transformers.md @@ -2,4 +2,4 @@ Check out the following examples from the TestLearnAPI.jl test suite: -- [Truncated SVD](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/test/patterns/dimension_reduction.jl) +- [Truncated SVD](https://github.com/JuliaAI/LearnTestAPI.jl/blob/dev/src/learners/dimension_reduction.jl) diff --git a/docs/src/predict_transform.md b/docs/src/predict_transform.md index d6ab8f25..2733fc64 100644 --- a/docs/src/predict_transform.md +++ b/docs/src/predict_transform.md @@ -52,8 +52,8 @@ transform(learner, data) # `fit` implied ```julia fitobs = obs(learner, (X, y)) # learner-specific repr. of data -model = fit(learner, MLUtils.getobs(fitobs, 1:100)) -predictobs = obs(model, MLUtils.getobs(X, 101:150)) +model = fit(learner, MLCore.getobs(fitobs, 1:100)) +predictobs = obs(model, MLCore.getobs(X, 101:150)) ŷ = predict(model, Point(), predictobs) ``` diff --git a/docs/src/reference.md b/docs/src/reference.md index 18fb92df..a4499e55 100644 --- a/docs/src/reference.md +++ b/docs/src/reference.md @@ -1,8 +1,10 @@ # [Reference](@id reference) Here we give the definitive specification of the LearnAPI.jl interface. For informal -guides see [Anatomy of an Implementation](@ref) and [Common Implementation -Patterns](@ref patterns). +guides see [Anatomy of an Implementation](@ref) and [Common Implementation Patterns](@ref +patterns). + + - [List of Public Names](@ref) ## [Important terms and concepts](@id scope) @@ -21,13 +23,13 @@ individual observations. A `DataFrame` instance, from [DataFrames.jl](https://dataframes.juliadata.org/stable/), is an example of data, the observations being the rows. Typically, data provided to LearnAPI.jl algorithms, will implement the -[MLUtils.jl](https://juliaml.github.io/MLUtils.jl/stable) `getobs/numobs` interface for +[MLCore.jl](https://juliaml.github.io/MLCore.jl/stable) `getobs/numobs` interface for accessing individual observations, but implementations can opt out of this requirement; see [`obs`](@ref) and [`LearnAPI.data_interface`](@ref) for details. !!! note - In the MLUtils.jl + In the MLCore.jl convention, observations in tables are the rows but observations in a matrix are the columns. @@ -190,7 +192,7 @@ Most learners will also implement [`predict`](@ref) and/or [`transform`](@ref). - [`LearnAPI.features`](@ref input), [`LearnAPI.target`](@ref input), [`LearnAPI.weights`](@ref input): for extracting relevant parts of training data, where - defined. + defined. Also called *training data deconstructors*. - [Accessor functions](@ref accessor_functions): these include functions like `LearnAPI.feature_importances` and `LearnAPI.training_losses`, for extracting, from diff --git a/docs/src/testing_an_implementation.md b/docs/src/testing_an_implementation.md index 449a031c..cc0d58f6 100644 --- a/docs/src/testing_an_implementation.md +++ b/docs/src/testing_an_implementation.md @@ -1,9 +1,55 @@ # Testing an Implementation -```@raw html -🚧 +Testing is provided by the LearnTestAPI.jl package documented below. + +## Quick start + +```@docs +LearnTestAPI ``` !!! warning - Under construction + New releases of LearnTestAPI.jl may add tests to `@testapi`, and + this may result in new failures in client package test suites, because + of previously undetected broken contracts. Adding a test to `@testapi` + is not considered a breaking change + to LearnTestAPI, unless it supports a breaking change to LearnAPI.jl. + + +## The @testapi macro + +```@docs +LearnTestAPI.@testapi +``` + +## Learners for testing + +LearnTestAPI.jl provides some simple, tested, LearnAPI.jl implementations, which may be +useful for testing learner wrappers and meta-algorithms. + +```@docs +LearnTestAPI.Ridge +LearnTestAPI.BabyRidge +LearnTestAPI.ConstantClassifier +LearnTestAPI.TruncatedSVD +LearnTestAPI.Selector +LearnTestAPI.FancySelector +LearnTestAPI.NormalEstimator +LearnTestAPI.Ensemble +LearnTestAPI.StumpRegressor +``` + +## Private methods + +For LearnTestAPI.jl developers only, and subject to breaking changes at any time: + +```@docs +LearnTestAPI.@logged_testset +LearnTestAPI.@nearly +LearnTestAPI.isnear +LearnTestAPI.learner_get +LearnTestAPI.model_get +LearnTestAPI.verb +LearnTestAPI.filter_out_verbosity +``` diff --git a/docs/src/traits.md b/docs/src/traits.md index eebb9e8e..890a53f9 100644 --- a/docs/src/traits.md +++ b/docs/src/traits.md @@ -27,7 +27,7 @@ In the examples column of the table below, `Continuous` is a name owned the pack | [`LearnAPI.nonlearners`](@ref)`(learner)` | properties *not* corresponding to other learners | all properties | `(:K, :leafsize, :metric,)` | | [`LearnAPI.human_name`](@ref)`(learner)` | human name for the learner; should be a noun | type name with spaces | "elastic net regressor" | | [`LearnAPI.iteration_parameter`](@ref)`(learner)` | symbolic name of an iteration parameter | `nothing` | :epochs | -| [`LearnAPI.data_interface`](@ref)`(learner)` | Interface implemented by objects returned by [`obs`](@ref) | `Base.HasLength()` (supports `MLUtils.getobs/numobs`) | `Base.SizeUnknown()` (supports `iterate`) | +| [`LearnAPI.data_interface`](@ref)`(learner)` | Interface implemented by objects returned by [`obs`](@ref) | `Base.HasLength()` (supports `MLCore.getobs/numobs`) | `Base.SizeUnknown()` (supports `iterate`) | | [`LearnAPI.fit_scitype`](@ref)`(learner)` | upper bound on `scitype(data)` ensuring `fit(learner, data)` works | `Union{}` | `Tuple{AbstractVector{Continuous}, Continuous}` | | [`LearnAPI.target_observation_scitype`](@ref)`(learner)` | upper bound on the scitype of each observation of the targget | `Any` | `Continuous` | | [`LearnAPI.is_static`](@ref)`(learner)` | `true` if `fit` consumes no data | `false` | `true` | diff --git a/src/accessor_functions.jl b/src/accessor_functions.jl index b5f487b4..84cbab05 100644 --- a/src/accessor_functions.jl +++ b/src/accessor_functions.jl @@ -319,7 +319,7 @@ Here's a sample workflow for some such `learner`, with training data, `(X, y)`, is the training target, here assumed to be a vector. ```julia -import MLUtils.getobs +import MLCore.getobs model = fit(learner, (X, y)) yhat = LearnAPI.predictions(model) test_indices = LearnAPI.out_of_sample_indices(model) diff --git a/src/fit_update.jl b/src/fit_update.jl index c33e40b8..39f78273 100644 --- a/src/fit_update.jl +++ b/src/fit_update.jl @@ -61,8 +61,8 @@ function fit end update(model, data, param_replacements...; verbosity=1) Return an updated version of the `model` object returned by a previous [`fit`](@ref) or -`update` call, but with the specified hyperparameter replacements, in the form `p1 => -value1, p2 => value2, ...`. +`update` call, but with the specified hyperparameter replacements, in the form `:p1 => +value1, :p2 => value2, ...`. ```julia learner = MyForest(ntrees=100) @@ -105,7 +105,7 @@ function update end Return an updated version of the `model` object returned by a previous [`fit`](@ref) or `update` call given the new observations present in `new_data`. One may additionally -specify hyperparameter replacements in the form `p1 => value1, p2 => value2, ...`. +specify hyperparameter replacements in the form `:p1 => value1, :p2 => value2, ...`. ```julia-repl learner = MyNeuralNetwork(epochs=10, learning_rate => 0.01) @@ -145,7 +145,7 @@ function update_observations end Return an updated version of the `model` object returned by a previous [`fit`](@ref) or `update` call given the new features encapsulated in `new_data`. One may additionally -specify hyperparameter replacements in the form `p1 => value1, p2 => value2, ...`. +specify hyperparameter replacements in the form `:p1 => value1, :p2 => value2, ...`. When following the call `fit(learner, data)`, the `update` call is semantically equivalent to retraining ab initio using a concatenation of `data` and `new_data`, diff --git a/src/obs.jl b/src/obs.jl index 2e631d30..ea9024d0 100644 --- a/src/obs.jl +++ b/src/obs.jl @@ -25,15 +25,15 @@ model = fit(learner, data_train) ŷ = predict(model, Point(), X[101:150]) ``` -Alternative workflow using `obs` and the MLUtils.jl method `getobs` to carry out +Alternative workflow using `obs` and the MLCore.jl method `getobs` to carry out subsampling (assumes `LearnAPI.data_interface(learner) == RandomAccess()`): ```julia -import MLUtils +import MLCore fit_observations = obs(learner, data) -model = fit(learner, MLUtils.getobs(fit_observations, 1:100)) +model = fit(learner, MLCore.getobs(fit_observations, 1:100)) predict_observations = obs(model, X) -ẑ = predict(model, Point(), MLUtils.getobs(predict_observations, 101:150)) +ẑ = predict(model, Point(), MLCore.getobs(predict_observations, 101:150)) @assert ẑ == ŷ ``` @@ -54,7 +54,7 @@ alternatives with the same output, whenever `observations = obs(model, data)`. If `LearnAPI.data_interface(learner) == RandomAccess()` (the default), then `fit`, `predict` and `transform` must additionally accept `obs` output that has been *subsampled* -using `MLUtils.getobs`, with the obvious interpretation applying to the outcomes of such +using `MLCore.getobs`, with the obvious interpretation applying to the outcomes of such calls (e.g., if *all* observations are subsampled, then outcomes should be the same as if using the original data). diff --git a/src/traits.jl b/src/traits.jl index 0a99aaff..54cdacfc 100644 --- a/src/traits.jl +++ b/src/traits.jl @@ -12,7 +12,7 @@ const DOC_ON_TYPE = "The value of the trait must depend only on the type of `lea # Here, "for each `o` in `observations`" is understood in the sense of the data # interface specified for the learner, [`LearnAPI.data_interface(learner)`](@ref). For # example, if this is `LearnAPI.RandomAccess()`, then this means "for `o` in -# `MLUtils.eachobs(observations)`". +# `MLCore.eachobs(observations)`". # """ @@ -65,7 +65,7 @@ Return a tuple of expressions representing functions that can be meaningfully ap argument. Learner traits (methods for which `learner` is the *only* argument) are excluded. -To return actual functions, instead of symbols, use [`@functions`](@ref)` learner` +To return actual functions, instead of symbols, use [`@functions`](@ref) `learner` instead. The returned tuple may include expressions like `:(DecisionTree.print_tree)`, which @@ -470,7 +470,7 @@ Specifically, both of the following are always true: `target_observations`" is understood in the sense of the data interface specified for the learner, [`LearnAPI.data_interface(learner)`](@ref). For example, if this is `LearnAPI.RandomAccess()`, then this means "for each `o in - MLUtils.eachobs(target_observations)`". + MLCore.eachobs(target_observations)`". - `S` is an upper bound on the `scitype` of (point) observations that might normally be extracted from the output of [`predict`](@ref). diff --git a/src/types.jl b/src/types.jl index 3212c3f2..810c345e 100644 --- a/src/types.jl +++ b/src/types.jl @@ -212,9 +212,9 @@ abstract type Finite <: DataInterface end LearnAPI.RandomAccess A data interface type. We say that `data` implements the `RandomAccess` interface if -`data` implements the methods `getobs` and `numobs` from MLUtils.jl. The first method +`data` implements the methods `getobs` and `numobs` from MLCore.jl. The first method allows one to grab observations specified by an arbitrary index set, as in -`MLUtils.getobs(data, [2, 3, 5])`, while the second method returns the total number of +`MLCore.getobs(data, [2, 3, 5])`, while the second method returns the total number of available observations, which is assumed to be known and finite. All arrays implement `RandomAccess`, with the last index being the observation index @@ -234,8 +234,8 @@ If [`LearnAPI.data_interface(learner)`](@ref) takes the value `RandomAccess()`, # Implementing `RandomAccess` for new data types Typically, to implement `RandomAccess` for a new data type requires only implementing -`Base.getindex` and `Base.length`, which are the fallbacks for `MLUtils.getobs` and -`MLUtils.numobs`, and this avoids making MLUtils.jl a package dependency. +`Base.getindex` and `Base.length`, which are the fallbacks for `MLCore.getobs` and +`MLCore.numobs`, and this avoids making MLCore.jl a package dependency. See also [`LearnAPI.FiniteIterable`](@ref), [`LearnAPI.Iterable`](@ref). """ @@ -251,7 +251,7 @@ it implements Julia's `iterate` interface, including `Base.length`, and if - `data` implements the [`LearnAPI.RandomAccess`](@ref) interface (arrays and most tables); or -- `data isa MLUtils.DataLoader`, which includes output from `MLUtils.eachobs`. +- `data isa MLCore.DataLoader`, which includes output from `MLCore.eachobs`. If [`LearnAPI.data_interface(learner)`](@ref) takes the value `FiniteIterable()`, then [`obs`](@ref)`(learner, ...)` is guaranteed to return objects implementing the @@ -267,7 +267,7 @@ struct FiniteIterable <: Finite end A data interface type. We say that `data` implements the `Iterable` interface if it implements Julia's basic `iterate` interface. (Such objects may not implement -`MLUtils.numobs` or `Base.length`.) +`MLCore.numobs` or `Base.length`.) If [`LearnAPI.data_interface(learner)`](@ref) takes the value `Iterable()`, then [`obs`](@ref)`(learner, ...)` is guaranteed to return objects implementing `Iterable`,