-
Notifications
You must be signed in to change notification settings - Fork 0
PythonPackaging
I'm using this page to collect some resources related to Python packaging and take some notes to help understand it. It's very ugly, unorganized, and incomplete right now.
Table of Contents:
The Hitchhiker’s Guide to Packaging http://guide.python-distribute.org/ - DEAD
#distutils on freenode
http://eecs.oregonstate.edu/capstone/viewproposal2011.php?id=87
eggs? like java .jar files... zipped up package files.
Module/package terminology: http://docs.python.org/distutils/introduction.html#general-python-terminology
"5 tips for packaging your Python projects" mostly gives some tips on packaging your project so it's easier to support distutils2 when the time comes: http://tarekziade.wordpress.com/2011/08/19/5-tips-for-packaging-your-python-projects/
Packaging libraries:
- Allow developers to package up their module (source or binary distributions) and upload to PyPI
- Allow users to install the module
- and... ?
"Distutils2 is a new distutils library, started as a fork of the distutils codebase, with good ideas taken from setuptools and thoroughly discussed in PEPs for some of them, and a basic installer inspired by pip. The actual name that can be imported is “packaging” in the Python 3.3+ standard library or “distutils2” in 2.4+ and 3.1-3.2"
http://hg.python.org/distutils2
Instructions for trying out distutils2, pysetup, etc. https://tarekziade.wordpress.com/2011/06/02/help-us-ironing-packaging/
Latest dev documentation for Python 3.3's 'packaging' module http://docs.python.org/dev/packaging/
packaging library reference:
http://docs.python.org/dev/library/packaging.html
"A Quick Diff between Distutils and Distutils2" gives a quick overview of the diffs to the UI, and then an overview of the changes to the code: https://wokslog.wordpress.com/2011/06/04/distutils-diff/
"Distutils2 vs Pip" gives some high-level goals of distutils, a brief history of Python packaging, and talks about whether an installer like pip should be included in the standard library: https://tarekziade.wordpress.com/2010/05/31/distutils2-vs-pip/
This info mostly came from Tarek's PyCon talk, http://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2011-packaging-from-distutils-to-distutils2-4898961
- Defining dependencies:
- distutils has a
requiresargument to specify module-level dependencies. The problem is that there's no central place where people share modules. We share projects. Multiple projects may have modules with the same name. People never use this argument. - setuptools defines project-level dependencies with
install_requireswhich is what you want. This is an unofficial standard which isn't really what we want. - Solved with PEP 345, an official standard. Supports project-level dependencies. Supports environment markers -- OS-specific requirements (e.g., require pywin32 on Windows only). People used to have to do this with if statements in their setup.py checking for certain platforms.
- With older packaging standards, pip has to download a package from PyPI and run its setup.py to determine its dependencies. That sucks. With dependencies defined in the static setup.cfg file, pip will just need to download this one file to determine dependencies.
- distutils has a
- setup.py:
- "Gigantic swiss army knife with all the features you want" -- sometimes you just want to run a quick setup.py command to retrieve some metadata (like the project's name), but the script does all sorts of stuff like check your system, install other stuff, ... The people that write these setup.py files assume they're only used for installation.
- Three people use setup.py: Python packager users it to register/upload to PyPi and to create binary/source releases. End user to install it. OS packager that wants to read the metadata.
- setup.py is now gone. Use setup.cfg. Two sections. 1) metadata. 2) files. setup.py is no longer needed unless you want to provide backwards compatibility.
- New script, pysetup, is included with distutils2. It replaces 'python setup.py' for building projects, querying metadata, etc.
- distutils2 provides hooks so you can, for example, run some code before and after the install command is run if you need to. This allows people to do some funky stuff like they do with setup.py but only when it's necessary (on install). The metadata should still be static.
- Installation record to support uninstallation etc.
- distutils creates an egg-info file w/ the project's metadata, but doesn't record what was installed.
- easy_install creates an egg-info directory (a little different than distutils) and doesn't record what was installed either.
- pip creates an egg-info directory and records to install-record.txt (list of files that each package installed). Good, but it's another standard.
- PEP 376 defines a standard database for installed projects. distutils2 provides an implementation. The database is a Project.dist-info folder for each project in site-packages. It contains a RECORD (which files were installed + hash of file to detect changes), METADATA (project metadata), INSTALLER (e.g., pip), and REQUESTED (indicates if it was installed as a dependency or if it was specifically requested).
- New version of the pkgutil module in distutils2 for accessing this installed database. Can you pysetup to view install packages etc.
- Versioning
- Some projects have funky version numbers. Not cool. Hard to installers to know which one is newer.
- PEP 386 defines version numbers.
distutils2.versionimplements it.
- Defining data files
- distutils lets you define 3 things that get packaged up: modules, scripts, and data. Data could be a large variety of things (data files, PDFs, man pages, videos). There's no way to define what it is; what if an OS packager wants to put man pages in a certain place? When a package accesses one of its data files, it needs to use the ugly __file__ hack.
- Fix: 'resources' item in setup.cfg to specify files and type
(
appdata,icon, etc). Now you can use these appdata etc variables from your Python code (via pkgutil.resource_open) to locate the resources. Distutils2, when installing a project, will use platform-specific default locations (stored in sysconfig.cfg) for these things. This allows distros (like Redhat/Debian) to define their own defaults for where they want man pages installed, etc.
Distutils2 will allow hooks to support other build systems other than the built-in ones.
Entry points? Tarek was asked about them at 31:50 into the talk. I don't understand what they are. His answer was something about metadata extensions that would let you put additional files into the database of installed package info. He said a lot of people want entry points but they (the distutils people) don't think it should be there.
It should be pretty easy to support distutils2 and older libraries at the
same time, especially for projects with a simple setup.py (that doesn't do
anything except call setup()). Projects with fancy setup.py scripts
will have more work to do with hooks to run extra code during
installation.
A common thing people do in setup.py is read in a README to provide as the
long description for the project. distutils2 supports this by providing a
description_file option or something like that.
Another common thing people do in setup.py is import the project version from somewhere else in their code. I'm not sure how distutils2 will support this. Tarek didn't seem to know when he was asked during his 2011 (March) PyCon talk. distutils2 provides pre- and post-hooks for every command so you can sneak in some code, but you really do need to push a static setup.cfg to PyPI.
"Distribute is a fork of Setuptools that was started by developers feeling that its development pace was too slow and that it was not possible to evolve it. Its development was considerably slowed when distutils2 was started by the same group."
http://pypi.python.org/pypi/distribute
Fork of setuptools. Supports Python 3
"Setuptools is a project born from the desire to fill missing distutils functionality and explore new directions. In some subcommunities, it’s a de facto standard. It uses monkey-patching and magic that is frowned upon by Python core developers."
http://peak.telecommunity.com/DevCenter/setuptools
http://pypi.python.org/pypi/setuptools
easy_install is bundled with setuptools.
How did people install things before easy_install? Manually download and 'python setup.py install'?
Does easy_install depend on setuptools, or does it just happen to be bundled with it?
Setuptools points people to the distutils mailing list.
http://docs.python.org/distutils/
http://www.python.org/community/sigs/current/distutils-sig/
distutils supports a setup.cfg file. I thought this was a distutils2 thing. Is the purpose of this file the same in distutils2? I think it's the same file; distutils2 just expands its usage (and requires it).
Also supports a system-wide distutils.cfg and a $HOME/.pydistutils.cfg
distutils cookbook: http://wiki.python.org/moin/Distutils/Cookbook
Distutils2 provides its own installer, pysetup, but it doesn't obsolete pip. pysetup is intended to just provide a minimum set of features to get people started. Pip will continue to provide more advanced stuff.
"Do not upload dev releases at PyPI" explains how installers don't currently know how to tell the difference between a stable and dev version of a package on PyPI, so people shouldn't upload dev versions. Distutils2 and PEP 386 start to fix this by defining a version scheme so installers will be able to recognize dev releases. Pip will need to support this. It will need to have some way for the user to pick which version should be installed (if it doesn't already.. I'm not sure). https://tarekziade.wordpress.com/2011/02/15/do-not-upload-dev-releases-at-pypi/
They're considering putting virtualenv in the stdlib under the pythonv name.
Draft PEP: https://bitbucket.org/carljm/pythonv-pep/src/
Background: http://mail.python.org/pipermail/distutils-sig/2011-March/017574.html
Source: https://bitbucket.org/vinay.sajip/pythonv
http://peak.telecommunity.com/DevCenter/EasyInstall
Bundled with setuptools
Implemented by http://docs.python.org/dev/library/packaging.metadata.html
http://www.python.org/dev/peps/pep-0345/
http://www.python.org/dev/peps/pep-0314/
http://www.python.org/dev/peps/pep-0241/
Summary of changes: https://tarekziade.wordpress.com/2010/02/10/pep-345-and-386-accepted-summary-of-changes/
http://www.python.org/dev/peps/pep-0376/ Implemented by http://docs.python.org/dev/library/packaging.database.html
http://www.python.org/dev/peps/pep-0262/ - Superseded by PEP 376
http://www.python.org/dev/peps/pep-0386/ Implemented by http://docs.python.org/dev/library/packaging.version.html
Summary of changes: https://tarekziade.wordpress.com/2010/02/10/pep-345-and-386-accepted-summary-of-changes/
- Distutils SIG (special interest group)
- 2012 August - Python 3.3 scheduled for release. Distutils2 to be
included as
packaging. Distutils (original) removed? - 2011 May -
packaginglands in the standard library development tree - 2009 July - setuptools forked, renamed to Distribute
- 2000 - Distutils added to the standard library in Python 1.6
...
- A history of Python packaging - Nov 2009