Refresh Model Zoo

[ToucheSir]

This issue will track long term progress for the model-zoo. Here are the steps to get there (in order):

• Test all models: run through all the tutorials to check whether they work on the latest release and document problems
• Address outstanding external PRs: take care of PRs such as
  • Metalhead.jl
  • Flux.jl outdims
  • LearnBase.jl
  • yet to be filed MLDataPattern.jl
• Issue/PR clean-up: close stale issues and PRs
• Update models: get models working with latest release (no new tutorials!)
• Add new tutorials: add new tutorials (especially ones that make use of the new ecosystem packages)
• Trim old tutorials: remove tutorials that are superseded by new tutorials
• Set up benchmarking CI: convert tutorials to Weave.jl like SciML
• Translate to benchmarks: convert as many tutorials as possible to use the benchmark model

---

Original Text

Collecting these issues and PRs here for now. Eventually we may want to split some out for triage.

• [WIP] updates for Flux 0.11 model-zoo#241 update to Flux 0.11 (note: this is already done for the tutorial)
• Many examples use HWCN order instead of WHCN recommended by document model-zoo#235 incorrect dimension order for some CNNs
• Install TagBot as a GitHub Action model-zoo#199 use the non-deprecated TagBot Action
• sde and dde need names swapped model-zoo#154 rm bitrotted diffeq examples since they appear to be covered under SciML
• Boston example isn't very good (diverging loss) model-zoo#244 fix model non-convergence
• How about we don't use actively terrible julia code as a common demo? model-zoo#203 clean up unnecessary use of global scope
• Many models are missing README files model-zoo#218, Added Readme.md files for Char-RNN and MNIST  model-zoo#192 many models are missing READMEs
• [TODO issue] MNIST batch size is too aggressive and causes frequent OOMs
• [TODO issue] add CI (and possibly convergence testing?)

[adinhobl]

What is the overall vision for the model zoo? Like what attributes would make a good zoo model? I'm going to throw out some things and let me know if they are what you have in mind:

• single .jl file + Project.toml: self-contained, no need to have a bunch of files, easier setup (just ]instantiate, run Julia file). Not easy to document what different pieces are doing without long comments
• jupyter notebook: example like this which gives a full tutorial to the ecosystem (I'm working on this right now to learn myself, but it could be useful to know the whole pipeline and options for other newcomers). But this is harder to get up and running and hard to build a benchmark pipeline around
• optimized to be extremely fast: CuIterators and carefully thought out optimizations, no global scope. Harder for a Flux/Julia newbie to do
• Everything up to date: keep the examples relevant, identify bugs and holes before other people try them, actively prevent bitrot
• What to do about the data? Does every model need to implement its own method for downloading data? is there a best practice?

I read someone somewhere else said that they thought getting a working model zoo up to best practices would set the stage for growing the ML/DL/Flux ecosystem because then it's easier to benchmark and progressively improve.

[adinhobl]

As someone newer to the ecosystem and Julia, it would probably be good to have a list of best practices when implementing a model - a checklist with explanations. It would promote consistency for the model zoo models and be a good jumping point for new people in the community. There's a PR for something about putting things in the global scope, but someone new from python won't necessarily know not to do that for their model.

I think there is also likely an opportunity to improve these pages as well, specifically the performance one:
https://fluxml.ai/Flux.jl/stable/ecosystem/
https://fluxml.ai/Flux.jl/stable/performance/

[adinhobl]

As for my first post above, I just found Literate.jl, which I think could be a solution to having to maintain 1 .jl file but still being able to take advantage of the richness of notebooks. If they were all done in that style, there could be unit tests and CI but it could be easily transferable to notebooks. Just a thought.

[darsnack]

These are great comments. I'll try and give some insight on some of them.

• Julia files vs. Jupyter notebooks: I believe Chris has given us some references on how SciML uses Literate.jl (or maybe it was Weave.jl) to generate Julia + notebooks as well as benchmark with CI. (link)
• Staying up to date: This is absolutely crucial, and I think that using the zoo for benchmarking will help ensure this.
• Data management: All zoo models should use MLDatasets.jl, or implement the custom dataset interface.

[ChrisRackauckas]

It's all done by https://github.com/SciML/RebuildAction

[darsnack]

As someone newer to the ecosystem and Julia, it would probably be good to have a list of best practices when implementing a model - a checklist with explanations.

Totally agree with this. Once the Metalhead.jl PR is resolved, I can write up a section on this in the docs for Flux.jl.

[ToucheSir]

RE documenting best practices, there's an old list here that might be a good jumping-off point. I think it's also worth a look at the GSOD PRs for overlap and/or opportunities for collaboration.

[adinhobl]

Thanks for all the responses. It sounds like there are a handful of lists that could be put together generally on the topics of performance and best practices, and I think putting them in the Flux Docs probably makes sense.

@darsnack , when you said "the custom dataset interface" it sounds like you were referring to something in particular?

It also sounds like I need to familiarize myself with RebuildAction, CI, and the SciML Benchmarks, likely after this semester ends.

[ChrisRackauckas]

Feel free to ask for help, and feel free to ping me to join the next ML Fast AI coordination call. I am getting Dhariya involved as well.

[darsnack]

@darsnack , when you said "the custom dataset interface" it sounds like you were referring to something in particular?

Yes, since we've adopted MLDataPattern.jl for iterating datasets, any custom dataset needs to implement the getobs interface. Here is a good overview of what that entails.