Add fastai upstream and downstream capacities for fastai>=2.4 and fastcore>=1.3.27 versions (#678)

* added a fastai mixin for  upstream and downstream tasks with fastai library

* Added check for right fastai version

* Added aditional documentation

* Add support for fastai>=2.4 versions and fastcore>=1.3.27

* Update docstrings, save_fastai_learner, push_to_hub_fastai, and change name from fastai_mixin

* trim triling whitespaces

* Import fastai_utils.py in huggingface_hub/__init__py

* Add check_fastai_fastcore_versions function in fastai_utils.py

* Eliminate imports of libraries not used in fastai_utils.py

* Add isort and black format to fastai_utils.py

* Change pickle_protocol argument from kwargs to explicit

* Change kwargs arguments in the function from_pretrained_fastai to explicit

* Simplify push_to_hub_fastai function, particularly the repo_id argument

* Eliminate search for a pickle document in from_pretrained_fastai function

* Simplify push_to_hub_fastai and correct bug in from_pretrained_fastai

* Add pickle.DEFAULT_PROTOCOL for get adequate protocol when exporting the Learners

* Allow to load only models in the Hub when using from_pretrained_fastai

* Eliminate cache_dir from from_pretrained_fastai for simplification

* Correct nit picks in push_to_hub_fastai

* Update with nits

* Apply isort

* Replace config.json for pyproject.toml to check for fastai and fastcore versions

* Isort imports

* Make pyproject.toml automatically filled with fastai, fastcore, and python versions used.

* add check_fastai_fastcore_pyproject_versions function to know the fastai and fastcore versions of the pretrained model to load from the Hub

* Change library tomlkit for toml

* Add extras[fastai] with the toml library

* Change the way the token is asked in def push_to_hub_fastai(

* Eliminate logger from imports

* Fix nits

* Remove typing.Union from imports

* add fastai integration tests

* Import fastai in setup.py

* Import toml inside check_fastai_fastcore_pyproject_versions function

* Nits in fastai_utils.py

* Add build_fastai to python-tests.yml

* Add fastcore import to setup.py

* Add require_fastai_fastcore() to skip tests

* Nits and documentation of raised errors improved

* add strategy for fastai in python-tests.yml

* Eliminate organization from push_to_hub_fastai

* Add python 3.7-3.10 to python-tests.yml

* Change python version in tests to 3.9

* Fix tests

* Fix conflict in config.py due to order of tf packages

* Fix delete_repo function in test_fastai_integration

* Isort test_fastai_integration

* Replace the repo_id name for model_id

* Keep the names consistent

* Make fastai and fastcore versions flexible in setup.py

* Would be "fastai>=2.4" and "fastcore>=1.3.27"

* Confirm fastai supports python 3.10 in python-tests-yml

* Version 2.5.6 of fastai supports python 3.10

* Fix docs in fastai_utils.py

* Fix functions' arguments documentation for a proper rendering

* Change the name of DummyModel for dummy_model

* Handle pickling errors when exporting a fastai.Learner

* Guide user in how to deal with a PicklingError

* Change name of internal functions in fastai_utils.py

* Start functions that are not push_to_hub_fastai and from_pretrained_fastai with a "_" to indicate that they are internal functions. They would not be necessary for the user.

* black style to fastai_utils.py

* Eliminate unnecessary comments from fastai_utils.py

* Add capacity to load a local fastai.Learner to from_pretrained_keras

* In fastai_utils.py.
* This would be in addition to being able to load a pretrained model from the Hub

* black format fastai_utils.py

* Come back to Python 3.9 instead of 3.10

* fastai was made compatible with Python 3.10 just in the most recent version 3.5.6 released on April 1st. I would prefer to wait for the next release of fastai.

* Change name name of save_fastai_learner to _save_pretrained_fastai in test_fastai_integration.py

* Change the name to  _save_pretrained_fastai in __init__.py

* Additionally, isort test_fastai_integration.py
* Change the name from save_fastai_learner to _save_pretrained_fastai
* isort __init__.py

* Fix nits in test_fastai_integration.py

* Add fastai integration to docs

* functions from_pretrained_fastai and push_to_hub_fastai were added to mixins.mdx

* Fix nits

* Fix wording

* Allow _save_pretrained_fastai to directly export the model in save_directory

* By default learner.export saves learner to learner.path
* Before we where saving the model in learner.path and them moving it to save_directory
* Fix by changing learner.path to being equal to save_directory

* black fastai_utils.py

* Make the requirement of having a pyproject.toml optional

* Additionally, makes optional that the pyproject.toml contains the fastai and fastcore versions
* Will continue to throw an  error if the toml library is not available
* Will continue to thrown an error if the pyproject.toml specifies fastai or fastcore versions that are not supported by from_pretrained_fastai
* This will allow to load fastai models from the Hub that were not necessarily uploaded using push_to_hub_fastai.

* Add warnings if the pyproject.toml does not contain a "build-system" and "requires" section

* Change try-excepts for ifs in the warnings checking the fastai and fastcore versions

* Fix nits in documentation

* Move errors in _check_fastai_fastcore_pyproject_versions to conditions after warnings

* Misc improvements

* Move the versions checks for fastai and fastcore to "else"'s

* This allows to eliminate the returns when checking for the versions of fastai and fastcore in the pyproject.toml

* Move the versions checks for fastai and fastcore to "else"'s

* This allows to eliminate the returns when checking for the versions of fastai and fastcore in the pyproject.toml

* Black reformat

Co-authored-by: Omar Espejel <[email protected]>
Co-authored-by: osanseviero <[email protected]>

Author: 3 people

.github/workflows/python-tests.yml

@@ -90,6 +90,27 @@ jobs:
         pip install .[testing,tensorflow]
 
     - run: pytest -Werror::FutureWarning -sv ./tests/test_keras*
+  
+  build_fastai:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.7", "3.9"]
+
+    steps:
+    - uses: actions/checkout@v2
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Install dependencies
+      run: |
+        pip install --upgrade pip
+        pip install .[testing,fastai]
+
+    - run: pytest -Werror::FutureWarning -sv ./tests/test_fastai*
 
   tests_lfs:
     runs-on: ubuntu-latest

docs/source/package_reference/mixins.mdx

@@ -17,4 +17,13 @@ objects, in order to provide simple uploading and downloading functions.
 
 [[autodoc]] save_pretrained_keras
 
-[[autodoc]] KerasModelHubMixin
+[[autodoc]] KerasModelHubMixin
+
+### Fastai
+
+[[autodoc]] from_pretrained_fastai
+
+[[autodoc]] push_to_hub_fastai
+
+
+

setup.py

@@ -27,6 +27,12 @@ def get_version() -> str:
     "torch",
 ]
 
+extras["fastai"] = [
+    "toml",
+    "fastai>=2.4",
+    "fastcore>=1.3.27",
+]
+
 extras["tensorflow"] = ["tensorflow", "pydot", "graphviz"]
 
 extras["testing"] = [

src/huggingface_hub/__init__.py

@@ -31,6 +31,11 @@
     TF2_WEIGHTS_NAME,
     TF_WEIGHTS_NAME,
 )
+from .fastai_utils import (
+    _save_pretrained_fastai,
+    from_pretrained_fastai,
+    push_to_hub_fastai,
+)
 from .file_download import cached_download, hf_hub_download, hf_hub_url
 from .hf_api import (
     DatasetSearchArguments,