src/doc/en/developer/packaging.rst: Consolidate section on 'sage --package', add subsection on '... metrics'

Author: Matthias Koeppe

src/doc/en/developer/packaging.rst

@@ -1035,13 +1035,10 @@ upstream is ``$SAGE_ROOT/upstream/FoO-1.3.tar.gz``, create a new file
     tarball=FoO-VERSION.tar.gz
 
 Sage internally replaces the ``VERSION`` substring with the content of
-``package-version.txt``. To recompute the checksums, run::
+``package-version.txt``.
 
-    [alice@localhost sage]$ sage --package fix-checksum foo
-
-which will modify the ``checksums.ini`` file with the correct
-checksums.
 
+.. _section-spkg-upstream-urls:
 
 Upstream URLs
 -------------
@@ -1068,39 +1065,33 @@ For Python packages available from PyPI, you should use an
 
     upstream_url=https://pypi.io/packages/source/m/matplotlib/matplotlib-VERSION.tar.gz
 
-A package that has the ``upstream_url`` information can be updated by
-simply typing::
-
-    [alice@localhost sage]$ sage --package update numpy 3.14.59
+Developers who wish to test a package update from a PR branch before
+the archive is available on a Sage mirror. Sage falls back to
+downloading package tarballs from the ``upstream_url`` after trying all
+Sage mirrors. (This can be disabled by using ``./configure
+--disable-download-from-upstream-url``.)  To speed up this process,
+you can trim ``upstream/mirror_list`` to fewer mirrors.
 
-which will automatically download the archive and update the
-information in ``build/pkgs/``.
 
-For Python packages available from PyPI, there is another shortcut::
+.. _section-sage-package-command:
 
-    [alice@localhost sage]$ sage --package update-latest matplotlib
-    Updating matplotlib: 3.3.0 -> 3.3.1
-    Downloading tarball to ...matplotlib-3.3.1.tar.bz2
-    [...............................................................]
+Utility script to create and maintain packages
+==============================================
 
-The ``upstream_url`` information serves yet another purpose.
-Developers who wish to test a package update from a PR branch before
-the archive is available on a Sage mirror can do so by configuring
-their Sage tree using ``./configure
---enable-download-from-upstream-url``.  Then Sage will fall back to
-downloading package tarballs from the ``upstream_url`` after trying all
-Sage mirrors.  (To speed up this process,  trim ``upstream/mirror_list``
-to fewer mirrors.)
-It is then no longer necessary to manually download upstream tarballs.
+The command ``sage --package`` offers a range of functionality for
+creating and maintaining packages of the Sage distribution.
 
 
-Utility script to create packages
-=================================
+Creating packages
+-----------------
 
 Assuming that you have downloaded
 ``$SAGE_ROOT/upstream/FoO-1.3.tar.gz``, you can use::
 
-    [alice@localhost sage]$ sage --package create foo --version 1.3 --tarball FoO-VERSION.tar.gz --type experimental
+    [alice@localhost sage]$ sage --package create foo                     \
+                                             --version 1.3                \
+                                             --tarball FoO-VERSION.tar.gz \
+                                             --type experimental
 
 to create ``$SAGE_ROOT/build/pkgs/foo/package-version.txt``,
 ``checksums.ini``, and ``type`` in one step.
@@ -1111,22 +1102,132 @@ set the ``upstream_url`` field in ``checksums.ini`` described above.
 
 For Python packages available from PyPI, you can use::
 
-    [alice@localhost sage]$ sage --package create scikit_spatial --pypi --type optional
+    [alice@localhost sage]$ sage --package create scikit_spatial --pypi   \
+                                             --type optional
 
 This automatically downloads the most recent version from PyPI and also
 obtains most of the necessary information by querying PyPI.
+
 The ``dependencies`` file may need editing (watch out for warnings regarding
 ``--no-deps`` that Sage issues during installation of the package!).
 Also you may want to set lower and upper bounds for acceptable package versions
 in the file ``install-requires.txt``.
 
-To create a pip package rather than a normal package, you can use::
+By default, when the package is available as a platform-independent
+wheel, the ``sage --package`` creates a wheel package. To create a normal package
+instead (for example, when the package requires patching), you can use::
+
+    [alice@localhost sage]$ sage --package create scikit_spatial --pypi   \
+                                             --source normal              \
+                                             --type optional
+
+To create a pip package rather than a normal or wheel package, you can use::
+
+    [alice@localhost sage]$ sage --package create scikit_spatial --pypi   \
+                                             --source pip                 \
+                                             --type optional
+
+When the package already exists, ``sage --package create`` overwrites it.
 
-    [alice@localhost sage]$ sage --package create scikit_spatial --pypi --source pip --type optional
 
-To create a wheel package rather than a normal package, you can use::
+Updating packages to a new version
+----------------------------------
+
+A package that has the ``upstream_url`` information can be updated by
+simply typing::
+
+    [alice@localhost sage]$ sage --package update numpy 3.14.59
+
+which will automatically download the archive and update the
+information in ``build/pkgs/numpy/``.
+
+For Python packages available from PyPI, there is another shortcut::
+
+    [alice@localhost sage]$ sage --package update-latest matplotlib
+    Updating matplotlib: 3.3.0 -> 3.3.1
+    Downloading tarball to ...matplotlib-3.3.1.tar.bz2
+    [...............................................................]
+
+If you pass the switch ``--commit``, the script will run ``git commit``
+for you.
+
+If you prefer to make update a package ``foo`` by making manual
+changes to the files in ``build/pkgs/foo``, you will need to run::
+
+    [alice@localhost sage]$ sage --package fix-checksum foo
+
+which will modify the ``checksums.ini`` file with the correct
+checksums.
+
 
-    [alice@localhost sage]$ sage --package create scikit_spatial --pypi --source wheel --type optional
+Obtaining package metrics
+-------------------------
+
+The command ``sage --package metrics`` computes machine-readable
+aggregated metrics for all packages in the Sage distribution or a
+given list of packages::
+
+    [alice@localhost sage]$ sage --package metrics
+    has_file_distros_arch_txt=181
+    has_file_distros_conda_txt=289
+    has_file_distros_debian_txt=172
+    has_file_distros_fedora_txt=183
+    has_file_distros_gentoo_txt=211
+    has_file_distros_homebrew_txt=95
+    has_file_distros_macports_txt=173
+    has_file_distros_nix_txt=72
+    has_file_distros_opensuse_txt=206
+    has_file_distros_slackware_txt=32
+    has_file_distros_void_txt=221
+    has_file_patches=63
+    has_file_spkg_check=106
+    has_file_spkg_configure_m4=262
+    has_file_spkg_install=322
+    has_tarball_upstream_url=291
+    line_count_file_patches=31904
+    line_count_file_spkg_check=585
+    line_count_file_spkg_configure_m4=3337
+    line_count_file_spkg_install=4342
+    packages=442
+    type_base=1
+    type_experimental=18
+    type_optional=151
+    type_standard=272
+
+Developers can use these metrics to monitor the complexity and quality
+of the Sage distribution. Here are some examples:
+
+- ``has_file_patches`` indicates how many packages have non-empty
+  ``patches/`` directories, and ``line_count_file_patches`` gives
+  the total number of lines in the patch files.
+
+  Ideally, we would not have to carry patches for a
+  package. For example, updating patches when a new upstream version
+  is released can be a maintenance burden.
+
+  Developers can help by working with the upstream maintainers of the
+  package to prepare a new version that requires fewer or smaller
+  patches, or none at all.
+
+- ``line_count_spkg_install`` gives the total number of lines in
+  ``spkg-install`` or ``spkg-install.in`` files; see
+  :ref:`section-spkg-install`.
+
+  When we carry complex ``spkg-install.in`` scripts for normal packages,
+  it may indicate that the upstream package's build and installation
+  scripts should be improved.
+
+  Developers can help by working with the upstream maintainers of the
+  package to prepare an improved version.
+
+- ``has_file_spkg_check`` indicates how many packages have an
+  ``spkg-check`` or ``spkg-check.in`` file; see :ref:`section-spkg-check`.
+
+- ``has_file_spkg_configure_m4`` indicates how many packages
+  are prepared to check for an equivalent system package, and
+  ``has_file_distros_arch_txt``, ``has_file_distros_conda_txt``
+  etc. count how many packages provide the corresponding system package
+  information.
 
 
 .. _section-manual-build:
@@ -1136,9 +1237,7 @@ Building the package
 
 At this stage you have a new tarball that is not yet distributed with
 Sage (``FoO-1.3.tar.gz`` in the example of section
-:ref:`section-directory-structure`). Now you need to manually place it
-in the ``SAGE_ROOT/upstream/`` directory and run
-``sage --fix-pkg-checksums`` if you have not done that yet.
+:ref:`section-directory-structure`).
 
 Now you can install the package using::
 
@@ -1157,8 +1256,7 @@ or::
 
     [alice@localhost sage]$ sage -f -c package_name
 
-If all went fine, open a PR, put a link to the original tarball in
-the PR and upload a branch with the code under
+If all went fine, open a PR with the code under
 ``SAGE_ROOT/build/pkgs``.