MakingANewRelease

How to Make a New Hwloc Release

There are several steps that are required to make a new hwloc release/prerelease and/or release series.

There are two cases:

1. You are making a release on a brand-new release series (it's probably a pre-release, like an alpha, beta, or RC -- but hey, it could be a final release, too).
2. You are making a release on an existing release series -- meaning that you've already followed the previous steps for initially creating the release series; now you're just putting a new release out on that existing series.

Announces

We usually announce the release to the following mailing lists:

• [email protected]
• [email protected]
• [email protected]

Make a New Hwloc Release Series

This is the case where you need to make a release for version A.B.C where C==0 -- the release is probably a pre-release of some kind (e.g., an alpha, beta, or rc tarball). None of the infrastructure yet exists to make this new series. There are several systems that are affected:

1. Build system
2. Git
3. Nightly / official tarball build server
4. Web site

Let's go over each of these in order.

Build System

Before making a branch for the new release series, it's usually a good idea to evaluate if upgrading to newer versions of the GNU Autotools is a good idea. Specifically: we make an effort to pick a tuple of Autotools for use with a release series and try very hard to use those same exact versions for every official release in that series.

If you end up upgrading the version of Autotools that you're using, it is best to edit source:/contrib/dist/make_dist_tarball (on the master -- before you make the Git branch for the new series) to set these versions. Run the script to ensure that these versions work properly with the master branch. Tune up anything that you need on the master branch to make these versions work. Once it's all working, commit the new version of make_dist_tarball with those version numbers back to the master branch.

API Version

If the API has changed, increasing HWLOC_API_VERSION in hwloc.h is usually a good idea. This should only be needed for major releases. Do it in master before branching.

Git

The main thing to do is to create a release branch in Git. Once you get the master branch in a good state, branch it. Note that there is a naming convention for branches:

$ git branch vA.B

where "A.B" is obviously the numbers that are relevant for your new release series (e.g., "1.1").

Once you make this branch, check it out and edit the VERSION file. Ensure that its VERSION numbers are set to A.B and the greek is set to something relevant (e.g., "a1" or "rc1").

Set the .so library version to the appropriate value (see the Libtool documentation for instructions: page 1 and page 2. And commit these changes.

Back on the master branch, edit the VERSION file and ensure that it is now set to version A.(B+1) and its greek is set to something like "a1".

Once everything is ready, push everything to github, as well as a tag on the new branch so that the nightly script (git describe) uses vA.B instead of dev as the basename.

$ git tag hwloc-A.B-branch vA.B
$ git push origin vA.B
$ git push origin hwloc-A.B-branch
$ git push origin master
``

### Build Server

Hwloc's build server is: mtt.open-mpi.org.  The user that is used to
create nightly tarballs is "mpiteam".  Contact Jeff Squyres or the IU
representative if you need to get the password (this password is
pretty tightly controlled; you'll need a very good reason to have it).

> **NOTE:** Be warned that this account is also used to build all Open
> MPI projects and subprojects.  Much of the infrastructure used for
> hwloc is also used by Open MPI.  So please share nicely.

Here's what to setup:

1. Login as mpiteam on mtt.open-mpi.org via ssh.
1. Edit the file `$HOME/scripts/hwloc-nightly-tarball.sh`.  Around
   line 23 is a list of the Git branches to build nightly tarballs for
   (it also specifies the order in which these tarballs are built).
   Add the basedir of your new release series (e.g.,
   `/branches/vA.B`).
1. Check to see if an environment modulefile exists of the specific
   tuple of Autotools that this hwloc series requires.
   1. Look in $HOME/modules/autotools for a modulefile named
      `<ac_version>-<am_version>-<lt_version>-<m4_version>`
   1. If it does not exist:
      1. Go to `$HOME/local`.
      1. Download the relevant Autotools tarballs from
         ftp://ftp.gnu.org/gnu/ that are not already there.
      1. Make the directory
         `$HOME/local/autotools-<ac_version>-<am_version>-<lt_version>-<m4_version>`.
      1. Copy the script `build.sh` from one of the other
         `autotools-<ac_version>-<am_version>-<lt_version>-<m4_version>`
         directories into your new
         `autotools-<ac_version>-<am_version>-<lt_version>-<m4_version>`
         directory.
      1. Edit your new `build.sh` to have the relevant version numbers
         at the very top.
      1. Run your new
         `autotools-<ac_version>-<am_version>-<lt_version>-<m4_version>/build.sh`
         script (from within that directory).
      1. This script will expand the tarballs, build them, and install
         them, and create a modulefile.
   1. At this point, the modulefile for your autotools tuple must
      exist.
   1. Sym link that modulefile to `hwloc-vA.B` in the
      `$HOME/modules/autotools` directory (such that you can "module use
      $HOME/modules; module load autotools/hwloc-vA.B" to get the Right
      autotools tuple.
1. Run the `$HOME/scripts/hwloc-nightly-tarball.sh` script manually
   (it may take several minutes; there will be no stdout because this
   script is normally run via cron).
1. If all goes well, you'll eventually get a mail sent to the
   hwloc-devel list saying that a snapshot tarball was created for your
   new release series.

> **NOTE:** Running the `hwloc-nightly-tarball.sh` will copy the
> nightly tarball to the live web site tree.  The nightly tarballs are
> not committed to the web site git repository (because they change
> too fast), but they are placed in the live tree itself at the
> location `/l/osl/www/www.open-mpi.org/software/hwloc/nightly/vA.B/`.
> If the directory for your new series does not exist, **the script
> will create it.** This will create a problem if you have not already
> created the `software/hwloc/nightly/vA.B/` directory in hwloc's web
> tree Git repo, because when you do, running `git pull` on the live
> web tree will run into a conflict: Git will not create a directory
> that already exists.  As such, if you run `hwloc-nightly-tarball.sh`
> before you update the live web tree with an Git-controlled
> `software/hwloc/nightly/vA.B/` directory, you should remove the
> script-created directory, create the directory in Git, run `git
> pull` on the live web tree (or let cron do it) and *then* run the
> `hwloc-nightly-tarball.sh` script.

Also remember that the `hwloc-nightly-tarball.sh` script is run
nightly via mpiteam's cron on mtt.open-mpi.org.  So you might want to
setup the script for your new release series on the same day that you
update the web site.  Otherwise, the script will run that night,
create the directory, and then you'll have the Git directory conflict
whenever you finally get to updating the web site.

The script should create a new tarball every night whenever necessary
(i.e., there have been Git commits in the relevant Git branch since
the last snapshot tarball was made).  As such, if you run the
`hwloc-nightly-tarball.sh` script twice in a row, it is likely that
nothing will happen the second time (because there will have been no
new Git commits and therefore no need to make a new tarball).

### Web Site

The web site requires several additions for a new release series.

You usually do not do this step until you have some kind of tarball to
release for the new release series (e.g., an "rc" tarball, or some
other prerelease tarball).

First, get a clone of the Open MPI web site (because the hwloc tree
is part of that web site) from
https://github.com/open-mpi/ompi-www/.

> **NOTE** it is a very large tree.  As of August 2014, a Git clone of
> this tree takes about 4.3GB of disk space.

#### Post the Tarballs

1. Go into the `software/hwloc` tree.
   1. Copy the prior release series tree into a new tree: `cp
      vA.(B-1) vA.B` (e.g., `cp v1.0 v1.1`).  Don't bother copying the
      directory recursively (i.e., with the `-r` flag).  If you do,
      you'll just need to remove a bunch of stuff in the new vA.B tree.
   1. Cd into the new vA.B tree.
   1. Edit the version.inc file and change all instances of `vA_(B-1)`
      to `vA_B` and all instances of `vA.(B-1)` to `vA.B`.
   1. Edit index.php.
      1. Do the same version substitution that you did for version.inc.
      1. Comment out the line defining the $t_stable and $t_older.
         Commenting out these lines will prevent showing the grid of
         the stable release and older releases.  It is *not* necessary
         to comment out any of the other PHP code.
   1. Make a `downloads` subdirectory and cd into it.
      1. If you recursively copied from a prior release version,
         remove all the release files (tarballs, SRPMs, etc.) from the
         downloads directory.
      1. Place your vA.B release (or prerelease)
         tarballs/SRPMs/whatever into this `downloads` directory.
      1. Make a file named `latest_snapshot.txt` and put in **just the
         version number** of the tarballs you just put there.  For
         example, if you have a vA.Brc1 tarball, put "A.Brc1" in the
         `latest_snapshot.txt` file.
      1. Generate MD5 and SHA1 sums on your tarballs:
         * csh syntax: `sha1sum hwloc-* >! sha1sums.txt; md5sum
           hwloc-* >! md5sums.txt`
         * sh syntax:  `sha1sum hwloc-* > sha1sums.txt; md5sum hwloc-*
           > md5sums.txt`
   1. If all that worked, you should be able to surf to your tree in
      `software/hwloc/vA.B/` and see your release tarballs listed.
      * Double check that the download links work
      * Double check that the link to the Git NEWS file works

#### Update Nightly Snapshots

1. Cd to the `software/hwloc/nightly` directory.
   1. Copy the prior release series tree into a new tree: `cp vA.(B-1)
      vA.B` (e.g., `cp v1.0 v1.1`).
   1. Edit the `vA.B/index.php` file and change all instances of
      `vA_(B-1)` to `vA_B` and all instances of `vA.(B-1)` to `vA.B`.
1. Edit the `software/hwloc/nightly/index.php` file and add some text
   about your new series, including a link to the `vA.B` directory.
   * **IMPORTANT:** Use a trailing slash in the HREF when you link to
     a directory.  E.g., `<a href="vA.B/">version A.B</a>`.

#### Post Doxygen Documentation

1. Cd to the `projects/hwloc/doc` directory.
   1. Put PDFs from your release tarball into that directory, obeying
      the naming conventions: `hwloc-vA.B[greek]-letter.pdf` and
      `<same>-a4.pdf`.
   1. Make a new directory `vA.B<greek>` for the new HTML
      documentation.
   1. Expand your release tarball and run configure on it.  Cd into
      the `doc` directory.
   1. **IMPORTANT:** Ensure to have a recent version of Doxygen before
      continuing.
      * I know that version 1.3.9.1 (i.e., what is on mtt.open-mpi.org)
        is too old.
      * Doxyen version 1.7.1 (on my Mac via ports) is new enough.
      * I don't know the minimum version that is necessary.
   1. **IMPORTANT:** You *must* do this from your expanded and
      configured tarball.  Do **NOT** generate the doxygen docs from a
      Git checkout.  The idea here is the GUARANTEE that the generated
      docs *exactly match* the release tarballs.
   1. Run `doxygen www.open-mpi.org.cfg`.  This will generate a
      directory named `www.open-mpi.org`.
   1. Run `make www-doc-sed` to remove the spurious qualifiers
      (inline, unused, etc.) from the generated html files.
   1. Copy the entire contents of the `www.open-mpi.org/html`
      directory to `projects/hwloc/doc/vA.B<greek>`.
   1. Also copy the `www.open-mpi.org.inc` files from the expanded
      tarball's `doc` directory to `projects/hwloc/doc/vA.B<greek>`.
   1. Edit the `projects/hwloc/doc/index.php` file.  Find the
      definition of the `$snapv_dir` and change it to be the directory of
      your hwloc software directory (e.g., "vA.B", or "v1.1").
      * Browse to `projects/hwloc/doc/` and ensure that your
        prerelease PDF and HTML links show up and work properly.
      * Browse to `projects/hwloc/doc/vA.B/` and ensure that the HTML
        documentation shows up properly.

#### Update Left-Hand Navigation

1. Edit the `projects/hwloc/nav.inc file`.
1. Add a line for the new series.  You might want to label it
   "(prerelease)", or somesuch.

#### Check Everything

 1. Browse all the pages that you just added and ensure that they are
 all working properly, including all links.

#### Git Commit

 1. Make sure to `git add` all new files.
 1. Then Git commit and push to Github when you're sure that
    everything is correct.
 1. **REMEMBER:**
    * Several sites around the world mirror the entire Open MPI site
      and synchronize via Git.  We have no control over when then run
      `git pull` to update their mirror.
    * Hence, as soon as you commit, some mirror somewhere may run `git
      pull` immediately and see your new changes.  So treat git pushing
      as "make it immediately live on the web site".
    * The main Open MPI web site runs `git pull` every hour or so.  So
      even if your changes aren't immediately visible on the main site,
      check back in an hour.

Make a New Hwloc (Pre)Release on an Existing Release Series
-----------------------------------------------------------

This is the case where you are making a release on a release series
that has already been created (see below).  Hence, all the
infrastructure for the release series is already in place and you just
need to make a new point release (potentially even the first formal
release of the series, version A.B.C, where C==0, because the release
series was created with a pre-release such as an alpha, beta, or rc
tarball).

### Make the Tarballs

1. Edit the VERSION file and change at least 2 things:
   * Set the greek value to something relevant (e.g., "b1" or "rc1"
     or whatever -- keep bumping the number as you make successive
     pre-releases towards the final release).
   * Set the .so library version to the appropriate value (see the
     Libtool documentation for instructions:
     [page 1](http://www.gnu.org/software/libtool/manual/html_node/Libtool-versioning.html#Libtool-versioning) and
     [page 2](http://www.gnu.org/software/libtool/manual/html_node/Updating-version-info.html#Updating-version-info)
   * You must commit the modified VERSION values back to Git
1. **NOTE:** Each hwloc release branch requires a specific tuple of
   Autoconf/Automake/Libtool.  **You are strongly discouraged from
   changing this tuple for the life of a given series.** The idea is
   to keep the build system fairly constant for the life of a series
   and only change the tuple at the next release series.
1. Double check your Git tree and ensure that "git status" returns no
   locally-modified files.
1. From the top-level source directory, run
   `contrib/dist/make_dist_tarball`.
   * The process take anywhere from 10 to 30 minutes, depending on
     your machine.
   * You will need to have a recent version of doxygen installed.

If the process succeeds, you'll have 4 tarballs:
 * tar.gz and tar.bz2 for the pre-release/greek version.
 * tar.gz and tar.bz2 for the final version (i.e., exactly the same,
   but with the greek value set to the empty string).

The rationale is that you can test the greek tarballs, and if they're
good, you have the final tarballs ready to ship.  They're guaranteed
to be exactly the same because they were all made at the same time.

### Web Site

There are several steps to updating the web site.

#### Post the Tarballs

1. Get a clone of the Open MPI web site.  See instructions about how
   to do this, above.
1. Place the tarballs in `software/hwloc/vVERSION/downloads/` and Git
   add them (don't bother committing yet; there's a bunch more Git
   changes coming).
1. Get a Windows build of the same tarball; place it in the same
   directory and Git add it.
   * Samuel has made all of the Windows builds; I'm not sure of the
     process he follows to make them.
1. Re-generate the `md5sums.txt` and `sha1sums.txt`:
   * `md5sum hwloc-* >! md5sums.txt`
   * `sha1sum hwloc-* >! sha1sums.txt`
1. Edit `latest-snapshot.txt` to reflect the latest release number
   (e.g., "1.0.3rc3", or "1.0.4", or whatever version number is
   relevant).
1. Edit `software/hwloc/vVERSION/index.php`:
   * If this is a pre-release (**this step is unnecessary in v1.3 and
     beyond**; the PHP has been made smarter to just see if
     last_snapshot.txt contains strings like "rc", and will
     automatically show the pre-release if so):
      * Ensure `$t_prerelease` is uncommented in
        `software/hwloc/downloads/vVERSION/index.php`.
      * That should be sufficient for the PHP to read the
        `latest-snapshot.txt` file, see that it's a pre-release, and
        add the appropriate pre-release box / download links on the
        web page.
   * If this is not a pre-release:
     * Edit `software/hwloc/vVERSION/version.inc`.  Set the
       `$ver_vX_Y` variable to the version.  E.g., "1.0" or "1.1.3",
       or whatever is relevant.
     * Git remove all pre-release tarballs and zip files.  **We do not
       continue hosting pre-release versions after a final release is
       made.**
     * Edit `software/hwloc/download/vVERSION/index.php`:
       * Comment out `$t_prerelease`. This is sufficient to remove the
         "pre-release" box / download links from the web page.
       * If this is the *first* production release of the series:
         * Ensure `$t_stable` is uncommented.
         * Ensure `$t_older` is commented out.
       * If this is *not* the first production release of the series:
         * Ensure `$t_older` is uncommented.
         * Expand the array of version numbers in `$t_older` to
           include all old version numbers.

#### Post Doxygen Documentation

Follow the same steps as above: [#PostDoxygenDocumentation for posting
the Doxygen documentation].  If you are releasing a production
version, `<greek>` will be empty.

Additionally, if this is the first production release, then Git remove
all the pre-release documentation directories.  **We do not continue
hosting pre-release documentation after a final release is made.**

#### Update Left-Hand Navigation

 * If this was the first production release of a series:
   * Set the `$dir` variable in `software/hwloc/current/version.inc`
     to point to the directory of the new release series.  E.g.,
     "v1.1".
   * Edit the `projects/hwloc/nav.inc` file.
   * Label the new series as "(stable)".
   * Update the labels on the prior series to indicate that they are
     "(older)", or "(ancient)", or whatever is relevant.
 * If this is not a pre-release
   * Edit `projects/hwloc/doc/index.php`
   * Add a new link for the new release series.

#### Check Everything

Follow the same steps as above for checking everything.

#### Git Commit

Once you have Git removed all relevant files and/or directories,
follow the same steps as above to commit all the web site changes back
to Github.

### Git

If this was a production release, then the release needs to be tagged
in Git and the VERSION file needs to be updated.

For example:

```sh
$ git tag -a hwloc-A.B.C

where "A.B.C" is obviously the numbers that are relevant for your new release series (e.g., "1.1.0"). Note that we include the C value in the tag, even if it's zero.

NOTE: Be sure to use the -a option so that an annotated Git tag is created. See git-tag(1) for more details.

Once you have made this tag, edit the VERSION file on the branch and increment the version number so that it is now "A.(B+1).0a1". This ensures that we don't make new nightly snapshot tarballs that have the "A.B" version number. Specifically, version A.B has been released -- so we shouldn't make any more snapshot tarballs with that version number.