diff --git a/.github/workflows/dev.yml b/.github/workflows/dev.yml index 1d0778798..d70961235 100644 --- a/.github/workflows/dev.yml +++ b/.github/workflows/dev.yml @@ -440,8 +440,8 @@ jobs: run: bazelisk test //... --enable_runfiles --incompatible_strict_action_env --test_output=all heron: - # Job to verify that the tasks performed by PrepareRelease have been done. It is - # the committer's responsibility (currently) to run PrepareRelease themselves when + # Job to verify that the tasks performed by UpdateAlways have been done. It is + # the committer's responsibility (currently) to run UpdateAlways themselves when # making a PR, so that everything is kept in-sync. name: Check autogenerated file freshness runs-on: ubuntu-24.04 # TODO: Update to ubuntu-latest when that switches to 24.04 @@ -452,15 +452,22 @@ jobs: uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 with: submodules: false - fetch-depth: 0 # Necessary for maint/PrepareRelease + fetch-depth: 0 # Necessary for maint/UpdateAlways fetch-tags: false # Check out the unmerged source branch for `pull_request`-triggered runs; # otherwise use the tip of the branch for `workflow_dispatch` and `pull` triggers. ref: ${{ github.event.pull_request.head.ref || github.ref }} repository: ${{ github.event.pull_request.head.repo.full_name || github.repository }} - - name: PrepareRelease - run: maint/PrepareRelease + - name: UpdateDates + if: | + github.event_name != 'pull_request' && + (startsWith(github.ref, 'refs/heads/release/') || + startsWith(github.ref, 'refs/tags/pcre2-')) + run: maint/UpdateDates.py + + - name: UpdateAlways + run: maint/UpdateAlways - name: 'Rebuild *.h.generic' run: | @@ -478,7 +485,7 @@ jobs: # checks locally to keep these files up to date (since the tool versions are very # fussy and brittle). # - # However, we still want to run the steps above, to check that the PrepareRelease + # However, we still want to run the steps above, to check that the UpdateAlways # process is able to run to completion, since it can pick up errors in the man pages. - name: Commit and push, if not in a forked repo diff --git a/README b/README index 89a862d53..46ceb8a3e 100644 --- a/README +++ b/README @@ -607,7 +607,7 @@ zip formats. The command "make distcheck" does the same, but then does a trial build of the new distribution to ensure that it works. If you have modified any of the man page sources in the doc directory, you -should first run the maint/PrepareRelease script before making a distribution. +should first run the maint/UpdateAlways script before making a distribution. This script creates the .txt and HTML forms of the documentation from the man pages. diff --git a/doc/html/README.txt b/doc/html/README.txt index 89a862d53..46ceb8a3e 100644 --- a/doc/html/README.txt +++ b/doc/html/README.txt @@ -607,7 +607,7 @@ zip formats. The command "make distcheck" does the same, but then does a trial build of the new distribution to ensure that it works. If you have modified any of the man page sources in the doc directory, you -should first run the maint/PrepareRelease script before making a distribution. +should first run the maint/UpdateAlways script before making a distribution. This script creates the .txt and HTML forms of the documentation from the man pages. diff --git a/doc/pcre2demo.3 b/doc/pcre2demo.3 index 1755a7531..709b237bc 100644 --- a/doc/pcre2demo.3 +++ b/doc/pcre2demo.3 @@ -1,5 +1,5 @@ .TH PCRE2DEMO 3 "31 August 2021" "PCRE2 10.46-DEV" -.\"AUTOMATICALLY GENERATED BY PrepareRelease - do not EDIT! +.\"AUTOMATICALLY GENERATED BY UpdateAlways - do not EDIT! .SH NAME PCRE2DEMO - A demonstration C program for PCRE2 .SH "SOURCE CODE" diff --git a/maint/README b/maint/README index 55074d75e..a9aba7d8b 100644 --- a/maint/README +++ b/maint/README @@ -9,7 +9,7 @@ and also contains some notes for maintainers. Its contents are: Files in the maint directory Updating to a new Unicode release Preparing for a PCRE2 release - Making a PCRE2 release + Updating version info for libtool Long-term ideas (wish list) For a description of the way PCRE2 works, see the file called HACKING in the @@ -21,19 +21,19 @@ Files in the maint directory 132html A Perl script to convert man pages to HTML (.1 and .3 files "two" HTML), - used by PrepareRelease. + used by UpdateAlways. CheckMan A Perl script to validate the syntax in PCRE2 man pages, used by - PrepareRelease. + UpdateAlways. -CheckMan +CleanTxt A Perl script to clean up the nroff output in PCRE2 man pages, used by - PrepareRelease. + UpdateAlways. Detrail A Perl script to remove trailing whitespace from PCRE2 files, used by - PrepareRelease. + UpdateAlways. GenerateCommon.py A Python module containing data and functions that are used by the other @@ -68,10 +68,21 @@ ManyConfigTests A shell script that runs "configure, make, test" a number of times with different configuration settings. -PrepareRelease +UpdateAlways A shell script to ensure that all auto-generated outputs are ready for release. + It should be run often (by CI on each commit) to ensure that the repository + is in a clean and consistent state. + +UpdateDates.py +UpdateRelease.py + Python scripts to be run less frequently than UpdateAlways. These should only + be needed immediately before a release, when finalising the repository. + UpdateDates.py checks in the last-updated times on documentation pages. + UpdateRelease.py is needed after any change to the version number in + configure.ac. + pcre2_chartables.c.non-standard This is a set of character tables that came from a Windows system. It has characters greater than 128 that are set as spaces, amongst other things. I @@ -181,71 +192,123 @@ done Preparing for a PCRE2 release ============================= -This section contains a checklist of things that I (PH) do before building a +This section contains a checklist of things that I (NCW) do before building a new release. -. Ensure that the version number and version date are correct in configure.ac. +* First of all, make sure that the master branch is in good condition. + + - Basically, test it. The CI jobs should all be passing. This ensures that + pcre2tests are passing, that the build is warning-free, and that all our + platforms are running correctly. The CI jobs should be running the Perl + tests (which assert that `testdata/testinput1` tests give the same results + using Perl's regex engine). The ManyConfigTests exercise a variety of + build options and combinations. The Autoconf and CMake builds must pass. + + - If new build options have been added, ensure that they are added to the + CMake files as well as to the Autoconf files. + + - Run perltest.sh on the test data for tests 1 and 4. The output should match + the PCRE2 test output, apart from the version identification at the start of + each test. Sometimes there are other differences in test 4 if PCRE2 and Perl + are using different Unicode releases. The other tests are not + Perl-compatible (they use various PCRE2-specific features or options). The + maint/RunPerlTest shell script can be used to do this testing in Unix-like + environment. + + - It is possible to test with the emulated memmove() function by undefining + HAVE_MEMMOVE and HAVE_BCOPY in config.h, though I do not do this often. + + - Check the external testing tools. CodeQL & Clang Static Analyzer report + their results to the GitHub "Security" dashboard. Coverity has its own + external dashboard, as does OSS-Fuzz. Since we have these tools, we should + at least confirm they haven't flagged anything. -. Update the library version numbers in configure.ac according to the rules + - Documentation: check the documentation is ready; and LICENCE, + NON-AUTOTOOLS-BUILD, and README. Many of these won't need changing, but over + the long term things do change. + +* Ensure the AUTHORS file is up-to-date with any new contributors since the + last release. + +* Ensure the ChangeLog and NEWS files are updated with everything that you want + to announce in the new release. + +* Update the library version numbers in configure.ac according to the rules given below. -. If new build options or new source files have been added, ensure that they - are added to the CMake files as well as to the autoconf files. The relevant - files are CMakeLists.txt and config-cmake.h.in. After making a release, test - it out with CMake if there have been changes here. - -. Run ./autogen.sh to ensure everything is up-to-date. - -. Run the script maint/ManyConfigTests. This compiles and runs the tests for - many different sets of configuration options, some with valgrind. It can take - quite a long time. - -. Run tests in both 32-bit and 64-bit environments if possible. I can no longer - run 32-bit tests. - -. Run tests with two or more different compilers (e.g. clang and gcc), and make - use of -fsanitize=address and friends where possible. For gcc, - -fsanitize=undefined -std=gnu99 picks up undefined behaviour at runtime. For - clang, -fsanitize=address,undefined,integer can be used but an exception is - needed to allow XCLASS with very large ranges in the 32-bit library so it - should be followed by -fno-sanitize=unsigned-shift-base, additionally - -fno-sanitize=unsigned-integer-overflow must be added when compiling with - JIT. Newer versions of clang also need -fno-sanitize=function, at least - until pcre2test stops using generic pointers on its callbacks. Another - useful clang option is -fsanitize=signed-integer-overflow but that should - be already included if using "integer". - -. Do a test build using CMake. Remove src/config.h first, lest it override the - version that CMake creates. Also ensure there is no leftover CMakeCache.txt - in the directory you are testing in. - -. Remove the CMake cache and then check that a CMake unity build works: - [c]cmake -DCMAKE_UNITY_BUILD=ON sets up a unity build. - -. Run perltest.sh on the test data for tests 1 and 4. The output should match - the PCRE2 test output, apart from the version identification at the start of - each test. Sometimes there are other differences in test 4 if PCRE2 and Perl - are using different Unicode releases. The other tests are not Perl-compatible - (they use various PCRE2-specific features or options). The maint/RunPerlTest - shell script can be used to do this testing in Unix-like environment. - -. It is possible to test with the emulated memmove() function by undefining - HAVE_MEMMOVE and HAVE_BCOPY in config.h, though I do not do this often. - -. Documentation: check AUTHORS, ChangeLog (check version and date), LICENCE, - NEWS (check version and date), NON-AUTOTOOLS-BUILD, and README. Many of these - won't need changing, but over the long term things do change. - -. I used to test new releases myself on a number of different operating - systems. For example, on Solaris it is helpful to test using Sun's cc - compiler as a change from gcc. Adding -m64 to the cc options does a 64-bit - build. Since I retired I can no longer do much of this. There are automated - tests under Ubuntu, Alpine, macOS and Windows that are now set up as GitHub - actions. Check that they are running clean. - -. The buildbots at http://buildfarm.opencsw.org/ do some automated testing - of PCRE2 and should also be checked before putting out a release. (June 2024: - I am not sure these are currently working properly.) +* Push all these changes to master. + +* Take a branch off master, named "release/pcre2-10.XX-RC1" or + "release/pcre2-10.XX". + +* In the new branch, remove the "-DEV" prefix from the version number and set + the release date in configure.ac. Update the release date in the ChangeLog and + NEWS files. + + ``` + vim configure.ac + vim NEWS + vim ChangeLog + git add -u configure.ac NEWS ChangeLog + git commit -m"Update version number and release date" + ``` + +* Perform updates of the automatically-generated files. + + ``` + ./autogen.sh && ./configure + rm src/config.h.generic src/pcre2.h.generic + make src/config.h.generic src/pcre2.h.generic + git clean -idx . + git add -u src/config.h.generic src/pcre2.h.generic + git commit -m"Automatic update of .generic files" + + maint/UpdateRelease.py + maint/UpdateDates.py + maint/UpdateDocs + git add -u + git commit -m"Automatic update of doc files #noupdate" + ``` + +* Commit the Autoconf files to the branch. This is required so that users can + check out the Git tag, and receive the same contents as the tarball users. + + ``` + ./autogen.sh + git add -f Makefile.in aclocal.m4 ar-lib compile config.guess config.sub \ + configure depcomp install-sh ltmain.sh m4/libtool.m4 m4/ltoptions.m4 \ + m4/ltsugar.m4 m4/ltversion.m4 m4/lt~obsolete.m4 missing src/config.h.in \ + test-driver + git commit -m"Commit autogen.sh output" + ``` + +* Now, wait for the CI job to build the tarball. We can't do this locally: we + want to be releasing a tarball which is signed by GitHub. The GitHub signature + says, "Yes, the developer did not tamper with this tarball, we certify that it + was derived solely from the contents of the Git repository at this commit + hash". + +* Create the tag locally. We can't do this via the GitHub UI: it has no way to + create signed tags (since my GPG key lives on a Yubikey). + + ``` + git config user.signingkey 'FB63B406!' + git tag -s pcre2-10.XX -m"Release 10.XX" + git tag -v pcre2-10.XX + ``` + +* Download the tarball from the CI artifacts. Sign these using the GPG key. + + ``` + KEYID=FB63B406 + for i in pcre2-10.XX.{zip,tar.gz,tar.bz2}; do + gpg --output $i.sig --detach-sig --default-key $KEYID $i + gpg --verify $i.sig + done + ``` + +* In the GitHub UI, create a "release" from the tag (which must have been + already pushed). Add the tarballs and GPG signatures. Updating version info for libtool @@ -292,21 +355,6 @@ changes in a shared library: new version. Increment current, set revision and age to 0. -Making a PCRE2 release -====================== - -Run PrepareRelease and commit the files that it changes. The first thing this -script does is to run CheckMan on the man pages; if it finds any markup errors, -it reports them and then aborts. Otherwise it removes trailing spaces from -sources and refreshes the HTML documentation. Update the GitHub repository. - -Once PrepareRelease has run clean, run "make distcheck" to create the tarballs -and the zipball. I then sign these files. Double-check with "git status" that -the repository is fully up-to-date, then create a new tag and a release on -GitHub. Upload the tarballs, zipball, and the signatures as "assets" of the -GitHub release. - - Future ideas (wish list) ======================== diff --git a/maint/PrepareRelease b/maint/UpdateAlways similarity index 98% rename from maint/PrepareRelease rename to maint/UpdateAlways index 5bc52fe7c..0a3c689dc 100755 --- a/maint/PrepareRelease +++ b/maint/UpdateAlways @@ -122,7 +122,7 @@ perl <<"END" >pcre2demo.3 my $t = `git log -n1 --date=format:"%d %B %Y" --format=%cd ../src/pcre2demo.c`; chomp $t; print OUT ".TH PCRE2DEMO 3 \"", $t, '" "', $version, "\"\n" . - ".\\\"AUTOMATICALLY GENERATED BY PrepareRelease - do not EDIT!\n" . + ".\\\"AUTOMATICALLY GENERATED BY UpdateAlways - do not EDIT!\n" . ".SH NAME\n" . "PCRE2DEMO - A demonstration C program for PCRE2\n" . ".SH \"SOURCE CODE\"\n" . @@ -316,7 +316,7 @@ echo Detrailing perl maint/Detrail "${txt_files[@]}" "${c_files[@]}" echo Validating all text -perl maint/CheckTxt "${files[@]}" +perl maint/CheckTxt "${txt_files[@]}" perl maint/CheckTxt -ascii "${c_files[@]}" perl maint/CheckTxt -crlf "${crlf_files[@]}" diff --git a/maint/UpdateDates.py b/maint/UpdateDates.py index aeb9d4f95..26598acce 100755 --- a/maint/UpdateDates.py +++ b/maint/UpdateDates.py @@ -47,6 +47,8 @@ def update_man_date(filename): # doc/*.3 for filename in glob.glob('doc/*.3'): + if filename == 'doc/pcre2demo.3': + continue update_man_date(filename) # README, NON-AUTOTOOLS-BUILD