Best practices libraries, documenting, version info

[mishal]

Hello community,

firstly, I would like to thank all the people behing micropython for their hard work. I'm really excited what can be done using micropython!

To my questions:

Does micropython have some guides/advices/standards about documenting and versioning libraries and drivers? I'm working on my (first hw) project and I see different authors use different methods.

I would like to document my project including vendor libraries which I use, but i'm struggling with a kind of inconsistency. I'm used to document python using standard docstrings with reStructuredText format and generate them using sphinx.

1. Authors document the lib/driver outside the code (in readmes and tutorials) or document it using comments

   Is there any reason for this? file size?

2. What about the library versions?

In the standard library the version is written only in the manifest.py file. Some libraries defines the version as two digits (0.1), some uses three (0.6.0), others different (3.4.2-4). Does this follow semantic versioning? Can I use the libraries with 0 as major version? Is there any changelog or I have to dig in the git log?

./python-stdlib/os-path/manifest.py:metadata(version="0.1.4")
./python-stdlib/collections/manifest.py:metadata(version="0.1.2")
./python-stdlib/pickle/manifest.py:metadata(version="0.1")
./python-stdlib/logging/manifest.py:metadata(version="0.5")
./python-stdlib/heapq/manifest.py:metadata(version="0.9.3")
./python-stdlib/pathlib/manifest.py:metadata(version="0.0.1")
./python-stdlib/shutil/manifest.py:metadata(version="0.0.5")
./python-stdlib/base64/manifest.py:metadata(version="3.3.3-4")
./python-stdlib/copy/manifest.py:metadata(version="3.3.3-2")
./python-stdlib/unittest/manifest.py:metadata(version="0.10.3")
./python-stdlib/inspect/manifest.py:metadata(version="0.1.2")
./python-stdlib/argparse/manifest.py:metadata(version="0.4")
./python-stdlib/html/manifest.py:metadata(version="3.3.3-2")
./python-stdlib/quopri/manifest.py:metadata(version="0.5.1")
./python-stdlib/traceback/manifest.py:metadata(version="0.3")
./python-stdlib/pkgutil/manifest.py:metadata(version="0.1.1")
./python-stdlib/pprint/manifest.py:metadata(version="0.0.4")
./python-stdlib/hashlib/manifest.py:metadata(version="2.4.0-4")
./python-stdlib/itertools/manifest.py:metadata(version="0.2.3")
./python-stdlib/gzip/manifest.py:metadata(version="0.1.1")
./python-stdlib/string/manifest.py:metadata(version="0.1.1")
./python-stdlib/os/manifest.py:metadata(version="0.6")
./python-stdlib/binascii/manifest.py:metadata(version="2.4.0-5")
./python-stdlib/cmd/manifest.py:metadata(version="3.4.0-2")
./python-stdlib/uu/manifest.py:metadata(version="0.5.1")
./python-stdlib/venv/manifest.py:    version="0.1.0",
./python-stdlib/operator/manifest.py:metadata(version="0.1.1")
./python-stdlib/functools/manifest.py:metadata(version="0.0.7")
./python-stdlib/struct/manifest.py:metadata(version="0.1.1")
./python-stdlib/ssl/manifest.py:metadata(version="0.1")
./python-stdlib/stat/manifest.py:metadata(version="0.5.1")
./python-stdlib/tempfile/manifest.py:metadata(version="0.0.1")
./python-stdlib/curses.ascii/manifest.py:metadata(version="3.4.2-1")
./python-stdlib/__future__/manifest.py:metadata(version="0.0.3")
./python-stdlib/errno/manifest.py:metadata(version="0.1.4")
./python-stdlib/time/manifest.py:metadata(version="0.1")
./python-stdlib/textwrap/manifest.py:metadata(version="3.4.2-1")
./python-stdlib/io/manifest.py:metadata(version="0.1")
./python-stdlib/pkg_resources/manifest.py:metadata(version="0.2.1")
./python-stdlib/datetime/manifest.py:metadata(version="4.0.0")
./python-stdlib/warnings/manifest.py:metadata(version="0.1.1")
./python-stdlib/collections-defaultdict/manifest.py:metadata(version="0.3")
./python-stdlib/threading/manifest.py:metadata(version="0.1")
./python-stdlib/contextlib/manifest.py:metadata(description="Port of contextlib for micropython", version="3.4.2-4")
./python-stdlib/collections-deque/manifest.py:metadata(version="0.1.3")
./python-stdlib/abc/manifest.py:metadata(version="0.0.1")
./python-stdlib/random/manifest.py:metadata(version="0.2")
./python-stdlib/fnmatch/manifest.py:metadata(version="0.6.0")
./python-stdlib/unittest-discover/manifest.py:metadata(version="0.1.2")
./python-stdlib/locale/manifest.py:metadata(version="0.0.2")
./python-stdlib/hmac/manifest.py:metadata(version="3.4.2-3")
./python-stdlib/json/json/__init__.py:__version__ = "2.0.9"
./python-stdlib/datetime/datetime.py:__version__ = "2.0.0"

Some libraries define the __version__ attribute in the package so it can be loaded in runtime.

./python-stdlib/json/json/__init__.py:__version__ = "2.0.9"
./python-stdlib/datetime/datetime.py:__version__ = "2.0.0"

I've found this page https://docs.micropython.org/en/latest/reference/packages.html#writing-publishing-packages which describes how to install packages but does not give any advice how to write them.

Thank you.

[DavesCodeMusings]

Obligatory disclaimer: I have written exactly one MicroPython library. I do not consider myself an expert by any means.

Here's what I did:

• I used docstring because it can do double duty as code comments and also generate documentation for the class with pydoc. (I don't want to duplicate efforts if I don't have to.) It looks nice and it's easy to read. Sounds like you're already familiar with it.
• I use semantic versioning because MAJOR.MINOR.PATCH makes sense to me. I am using 0 as the MAJOR, because, again, this is my first library project. If I had it to do over, I might start with 1.0
• I include package.json in my repository so people can install using mip install github:DavesCodeMusings/thimble, but the repo is still in my GitHub namespace and under my control. (Best feature of mip if you ask me.)
• With all my projects, I tend to document with a README that lays out the what, where, why, and how of the project in a short and sweet manner. I use a wiki for longer examples and try to include sample code.

I think I tend to go heavy on documentation, but it comes from my experience (frustration?) over the years of trying to figure out other free/open source code that's either sparsely documented or has documentation that doesn't explain what someone new to the project needs to know to get started.

If you're keeping code in your own repo, I think you can do just about whatever you want, but I would definitely recommend using package.json so other folks can easily install your library.

[jimmo]

@mishal I'm the maintainer of micropython-lib, but I inherited the original structure and versioning from a previous contributor. I don't know what their system was (if any).

I've been gradually working away at trying to improve the quality and usefulness of micropython-lib, and in particular trying to adopt a more consistent approach to versioning. One thing I wish I had done was completely reset all version numbers to 1.0.0 when we introduced mip and introduced a versioning policy.

With regard to __version__. I'm not sure why json and datetime include __version__ (I should remove this). What actually happens is that when we publish packages to the mip repository (or if they are frozen into device firmware), they get a __version__ included automatically (i.e. we modify the .py file to append __version__, publish that, and it's corresponding .mpy file).

e.g. for the hmac library:
https://micropython.org/pi/v2/package/py/hmac/latest.json --> https://micropython.org/pi/v2/file/53/53627e1f

(I have just raised micropython/micropython-lib#690 to fix this for json/datetime).

To answer some other specific points:

1. Some libraries in micropython-lib do have docstrings (these are mostly the ones that have been copied from other sources before my time). In general we prefer to write the documentation separately because it unnecessary adds to the file size (although since we moved to mip, we're publishing .mpy files by default, and docstrings dont' get included in .mpy files). I think my preference is to move documentation into stub files (which we should then scoop up into the main docs.micropython.org rST built).

2. Having missed the opportunity to completely reset the versioning scheme, it might be worth considering doing kind of the next best thing, which is to round everything up to the next major release and go from there. e.g. a package that is currently 2.4.0-5 would become 3.0.0 and we go from there.
   For packages that are currently 0.x.x we should move to 1.0.0 except for packages we consider to be still "experimental" or similar.

3. We don't publish a changelog. We're pretty strict with commit descriptions and breaking up changes into small commits, so a git log in a given package's directory is quite useful. But it would be nice to provide a more convenient way to get a change log.

4. One of the things on my TODO list is to add a README.md to each package with a standard template. It would be reasonable to add a high-level changelog there.

[mishal]

Thank you for your answers.

• Is Pydoc another docstring format? It looks like it can be also parsed by sphinx, so using rst and pydoc as docstring formats can produce good documentation... I will check it. (https://sphinxcontrib-napoleon.readthedocs.io/en/latest/)

• +1 for giving library creators an advice/recommendation to use semantic versioning in their packages, maybe mention it here:? https://docs.micropython.org/en/latest/reference/packages.html#writing-publishing-packages

• I like the idea documenting using stub files, this will keep the size of the source smaller and my IDE can still give me hints... Is sphinx able to generate docs from stub files? This looks related:

  • Added support for reading .pyi files. sphinx-doc/sphinx#4824
  • Use PEP 484 stub files when documenting native extensions with autodoc sphinx-doc/sphinx#7630

• +1 for updating the versions!

• My project is not a library, so it's not installable using mip. I keep my vendor libraries in a separate directory vendor and simply create a symlink for each library I need to a lib folder. I freeze the lib into custom uf2.
  I'm wondering how to include the version information in the freezed firmware automatically (as you do when installing using mip) in __version__ attribute.

• A changelog in each package would be great, but git log -- folder/* works for me too.

[DavesCodeMusings]

In regard to your question about pydoc: it's the tool I used to generate the HTML from the embedded triple-quote comments.

pydoc3 --html thimble.py

I've not used Sphinx to generate docs, but at first glance it looks like the two tools have the same result of producing documentation from docstring comments in your code. I only chose pydoc because the module was included with the Python installation on my host system and therefor provided the path of least resistance.