script to pub final files; templatize readmes

Author: markusicu

docs/data-workflow.md

@@ -30,17 +30,15 @@ Process:
 *   Iterate between KenW and Michel.
 *   Generated from UnicodeData.txt and an annotations file, using some C program.
 *   Used for generating code charts.
-*   KenW posts NamesList.txt into https://www.unicode.org/Public/draft/UCD/ucd/ .
+*   KenW posts NamesList.txt somewhere.
 *   A unicodetools GitHub contributor fetches this file
     and creates a pull request as for “regular” data files.
 
 ### Folder readmes
 
-The “source of truth” for these is outside of GitHub for now.
-KenW updates or vets these files and posts them to https://www.unicode.org/Public/draft/ .
-A unicodetools GitHub contributor fetches these files and creates a pull request as above.
-
-See https://github.com/unicode-org/properties/issues/8 “simplify versioning of readme files”
+The various ReadMe.txt files are checked into the unicodetools repo.
+They are templatized, and the publication scripts below replace variables with the
+Unicode and emoji versions, copyright year, and publication date (date when the script was run).
 
 ### “Regular” data files
 
@@ -122,27 +120,45 @@ from a unicodetools workspace to a target folder with the layout of https://www.
 Send the resulting zip file to Rick for posting to https://www.unicode.org/Public/draft/ .
 Ask Rick to add other files that are not tracked in the unicodetools repo:
 *   Unihan.zip to .../draft/UCD/ucd
+*   UCDXML files to .../draft/UCD/ucdxml
 *   beta charts to .../draft/UCD/charts
 
 ### Publish a release
 
-TODO: Write a script like /pub/copy-release-to-draft.sh that will be run on the unicode.org server
-and copy the set of the .../dev/ data files for a beta snapshot
-from a unicodetools workspace to the location behind https://www.unicode.org/Public/draft/ .
+After the last UTC meeting for the release, collect all of the data file updates
+(mostly from recently opened action items).
 
+When complete, publish the draft files once more via the beta script.
 Verify the final set of files in the draft folder.
 
-TODO: Write a script like /pub/copy-draft-to-release.sh that will be run on the unicode.org server
-and copy the files from the location behind https://www.unicode.org/Public/draft/
-to the locations behind the version-specific release folders.
-For example:
-*   https://www.unicode.org/Public/draft/UCD/ → https://www.unicode.org/Public/15.1.0/
-*   https://www.unicode.org/Public/draft/UCA/ → https://www.unicode.org/Public/UCA/15.1.0/
-*   https://www.unicode.org/Public/draft/emoji/ → https://www.unicode.org/Public/emoji/15.1/
-*   etc.
+Run the [pub/copy-final.sh](https://github.com/unicode-org/unicodetools/blob/main/pub/copy-final.sh)
+script from an up-to-date repo workspace.
+
+Send the resulting zip file to Rick for posting to https://www.unicode.org/Public/ (not .../Public/draft/).
+Ask Rick to add other files that are not tracked in the unicodetools repo:
+*   Unihan.zip to .../<version>/ucd
+*   UCDXML files to .../<version>/ucdxml
+*   final charts to .../<version>/charts
+
+This script works much like the beta script, except it:
+*   assembles all of the files for Public/ in their release folder structure,
+    rather than for Public/draft/
+*   creates a zipped/<version> folder with UCD.zip
 
-After a Unicode release, copy a snapshot of the unicodetools repo .../dev/ files
-(matching the released files, of course) to a versioned unicodetools folder;
+### After a release
+
+Verify once more that the unicodetools repo .../dev/ files match the released/published files.
+(They better...)
+
+Copy a snapshot of the unicodetools repo .../dev/ files to a versioned unicodetools folder;
 for example: .../unicodetools/data/ucd/15.1.0/ .
 (We no longer append a “-Update” suffix to the folder name.)
 
+Create a release tag in the repo.
+
+Edit the pub/*.sh scripts and advance the version numbers and copyright years.
+
+Change the Unicode Tools code as necessary for the start of work on the next version.
+Settings.java lastVersion & latestVersion and more.
+
+Declare “main” to be open for the next version.

pub/copy-alpha-to-draft.sh

@@ -10,6 +10,25 @@ DRAFT=$2
 
 UNITOOLS_DATA=$UNICODETOOLS/unicodetools/data
 
+# Adjust the following for each year and version as needed.
+COPY_YEAR=2023
+UNI_VER=15.1.0
+EMOJI_VER=15.1
+
+TODAY=`date --iso-8601`
+
+mkdir -p $DRAFT
+
+cat > $DRAFT/sed-readmes.txt << eof
+s/COPY_YEAR/$COPY_YEAR/
+s/PUB_DATE/$TODAY/
+s/PUB_STATUS/draft/
+s/UNI_VER/$UNI_VER/
+s/EMOJI_VER/$EMOJI_VER/
+s%PUBLIC_EMOJI%Public/draft/emoji/%
+s%PUBLIC_UCD_EMOJI%Public/draft/UCD/ucd/emoji/%
+eof
+
 mkdir -p $DRAFT/UCD/ucd
 cp -r $UNITOOLS_DATA/ucd/dev/* $DRAFT/UCD/ucd
 rm -r $DRAFT/UCD/ucd/Unihan
@@ -22,6 +41,12 @@ cp $UNITOOLS_DATA/emoji/dev/* $DRAFT/emoji
 # Fix permissions. Everyone can read, and search directories.
 chmod a+rX -R $DRAFT
 
+# Update the readmes in-place (-i) as set up above.
+find $DRAFT -name '*ReadMe.txt' | xargs sed -i -f $DRAFT/sed-readmes.txt
+
+# Cleanup
+rm $DRAFT/sed-readmes.txt
+
 rm $DRAFT/alpha.zip
 (cd $DRAFT; zip -r alpha.zip *)
 

pub/copy-beta-to-draft.sh

@@ -10,6 +10,25 @@ DRAFT=$2
 
 UNITOOLS_DATA=$UNICODETOOLS/unicodetools/data
 
+# Adjust the following for each year and version as needed.
+COPY_YEAR=2023
+UNI_VER=15.1.0
+EMOJI_VER=15.1
+
+TODAY=`date --iso-8601`
+
+mkdir -p $DRAFT
+
+cat > $DRAFT/sed-readmes.txt << eof
+s/COPY_YEAR/$COPY_YEAR/
+s/PUB_DATE/$TODAY/
+s/PUB_STATUS/draft/
+s/UNI_VER/$UNI_VER/
+s/EMOJI_VER/$EMOJI_VER/
+s%PUBLIC_EMOJI%Public/draft/emoji/%
+s%PUBLIC_UCD_EMOJI%Public/draft/UCD/ucd/emoji/%
+eof
+
 mkdir -p $DRAFT/UCD/ucd
 cp -r $UNITOOLS_DATA/ucd/dev/* $DRAFT/UCD/ucd
 rm -r $DRAFT/UCD/ucd/Unihan
@@ -27,7 +46,7 @@ cp $UNITOOLS_DATA/idna/dev/* $DRAFT/idna
 
 mkdir -p $DRAFT/idna2008derived
 rm $DRAFT/idna2008derived/*
-cp $UNITOOLS_DATA/idna/idna2008derived/Idna2008-15.1.0.txt $DRAFT/idna2008derived
+cp $UNITOOLS_DATA/idna/idna2008derived/Idna2008-$UNI_VER.txt $DRAFT/idna2008derived
 cp $UNITOOLS_DATA/idna/idna2008derived/ReadMe.txt $DRAFT/idna2008derived
 
 mkdir -p $DRAFT/security
@@ -36,22 +55,29 @@ cp $UNITOOLS_DATA/security/dev/* $DRAFT/security
 # Fix permissions. Everyone can read, and search directories.
 chmod a+rX -R $DRAFT
 
+# Update the readmes in-place (-i) as set up above.
+find $DRAFT -name '*ReadMe.txt' | xargs sed -i -f $DRAFT/sed-readmes.txt
+
 # Zip files for some types of data, after fixing permissions
 rm $DRAFT/UCA/CollationTest.zip
 (cd $DRAFT/UCA; zip -r CollationTest.zip CollationTest && rm -r CollationTest)
 
 rm $DRAFT/security/*.zip
-(cd $DRAFT/security; zip -r uts39-data-15.1.0.zip *)
+(cd $DRAFT/security; zip -r uts39-data-$UNI_VER.zip *)
 
 # Fix permissions again to catch the zip files
 chmod a+rX -R $DRAFT
 
-# Zip file to deliver the whole set of beta data files
+# Cleanup
+rm $DRAFT/sed-readmes.txt
+
+# Zip file to deliver the whole set of data files
 rm $DRAFT/beta.zip
 (cd $DRAFT; zip -r beta.zip *)
 
 echo "--------------------"
 echo "Copy files from elsewhere:"
 echo "- Unihan.zip to $DRAFT/UCD/ucd"
+echo "- UCDXML files to $DRAFT/UCD/ucdxml"
 echo "- beta charts to $DRAFT/UCD/charts"
 

pub/copy-final.sh

@@ -0,0 +1,87 @@
+# Script for
+# https://github.com/unicode-org/unicodetools/blob/main/docs/data-workflow.md#publish-a-release
+#
+# Invoke like this:
+#
+# pub/copy-final.sh  ~/unitools/mine/src  /tmp/unicode/Public/final
+
+UNICODETOOLS=$1
+DEST=$2
+
+UNITOOLS_DATA=$UNICODETOOLS/unicodetools/data
+
+# Adjust the following for each year and version as needed.
+COPY_YEAR=2023
+UNI_VER=15.1.0
+EMOJI_VER=15.1
+
+TODAY=`date --iso-8601`
+
+mkdir -p $DEST
+
+cat > $DEST/sed-readmes.txt << eof
+s/COPY_YEAR/$COPY_YEAR/
+s/PUB_DATE/$TODAY/
+s/PUB_STATUS/final/
+s/UNI_VER/$UNI_VER/
+s/EMOJI_VER/$EMOJI_VER/
+s%PUBLIC_EMOJI%Public/emoji/$EMOJI_VER/%
+s%PUBLIC_UCD_EMOJI%Public/$UNI_VER/ucd/emoji/%
+eof
+
+mkdir -p $DEST/$UNI_VER/ucd
+mkdir -p $DEST/zipped/$UNI_VER
+cp -r $UNITOOLS_DATA/ucd/dev/* $DEST/$UNI_VER/ucd
+rm -r $DEST/$UNI_VER/ucd/Unihan
+mv $DEST/$UNI_VER/ucd/version-ReadMe.txt $DEST/$UNI_VER/ReadMe.txt
+mv $DEST/$UNI_VER/ucd/zipped-ReadMe.txt $DEST/zipped/$UNI_VER/ReadMe.txt
+
+mkdir -p $DEST/UCA/$UNI_VER
+cp -r $UNITOOLS_DATA/uca/dev/* $DEST/UCA/$UNI_VER
+
+mkdir -p $DEST/emoji/$EMOJI_VER
+cp $UNITOOLS_DATA/emoji/dev/* $DEST/emoji/$EMOJI_VER
+
+mkdir -p $DEST/idna/$UNI_VER
+cp $UNITOOLS_DATA/idna/dev/* $DEST/idna/$UNI_VER
+
+mkdir -p $DEST/idna/idna2008derived
+rm $DEST/idna/idna2008derived/*
+cp $UNITOOLS_DATA/idna/idna2008derived/Idna2008-$UNI_VER.txt $DEST/idna/idna2008derived
+cp $UNITOOLS_DATA/idna/idna2008derived/ReadMe.txt $DEST/idna/idna2008derived
+
+mkdir -p $DEST/security/$UNI_VER
+cp $UNITOOLS_DATA/security/dev/* $DEST/security/$UNI_VER
+
+# Fix permissions. Everyone can read, and search directories.
+chmod a+rX -R $DEST
+
+# Update the readmes in-place (-i) as set up above.
+find $DEST -name '*ReadMe.txt' | xargs sed -i -f $DEST/sed-readmes.txt
+
+# Zip files for some types of data, after fixing permissions
+rm $DEST/$UNI_VER/ucd/UCD.zip
+(cd $DEST/$UNI_VER/ucd; zip -r UCD.zip * && mv UCD.zip $DEST/zipped/$UNI_VER)
+
+rm $DEST/UCA/$UNI_VER/CollationTest.zip
+(cd $DEST/UCA/$UNI_VER; zip -r CollationTest.zip CollationTest && rm -r CollationTest)
+
+rm $DEST/security/$UNI_VER/*.zip
+(cd $DEST/security/$UNI_VER; zip -r uts39-data-$UNI_VER.zip *)
+
+# Fix permissions again to catch the zip files
+chmod a+rX -R $DEST
+
+# Cleanup
+rm $DEST/sed-readmes.txt
+
+# Zip file to deliver the whole set of data files
+rm $DEST/final.zip
+(cd $DEST; zip -r final.zip *)
+
+echo "--------------------"
+echo "Copy files from elsewhere:"
+echo "- Unihan.zip to $DEST/$UNI_VER/ucd"
+echo "- UCDXML files to $DEST/$UNI_VER/ucdxml"
+echo "- final charts to $DEST/$UNI_VER/charts"
+

unicodetools/data/emoji/dev/ReadMe.txt

@@ -1,21 +1,21 @@
 # Unicode Emoji
-# © 2023 Unicode®, Inc.
+# © COPY_YEAR Unicode®, Inc.
 # Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries.
 # For terms of use, see https://www.unicode.org/terms_of_use.html
 
-This directory contains draft data files for Unicode Emoji, Version 15.1
+This directory contains PUB_STATUS data files for Unicode Emoji, Version EMOJI_VER
 
-Public/draft/emoji/
+PUBLIC_EMOJI
 
   emoji-sequences.txt
   emoji-zwj-sequences.txt
   emoji-test.txt
 
-The following related files are found in the UCD for Version 15.1
+The following related files are found in the UCD for Version EMOJI_VER
 
-Public/draft/UCD/ucd/emoji/
+PUBLIC_UCD_EMOJI
 
   emoji-data.txt
   emoji-variation-sequences.txt
 
-For documentation, see UTS #51 Unicode Emoji, Version 15.1
+For documentation, see UTS #51 Unicode Emoji, Version EMOJI_VER

unicodetools/data/idna/dev/ReadMe.txt

@@ -1,10 +1,10 @@
 # Unicode IDNA Mapping and Test Data
-# Date: 2023-01-30
-# © 2023 Unicode®, Inc.
+# Date: PUB_DATE
+# © COPY_YEAR Unicode®, Inc.
 # Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries.
 # For terms of use, see https://www.unicode.org/terms_of_use.html
 
-This directory contains draft data files for version 15.1.0 of
+This directory contains PUB_STATUS data files for version UNI_VER of
 UTS #46, Unicode IDNA Compatibility Processing.
 
 https://www.unicode.org/reports/tr46/

unicodetools/data/idna/idna2008derived/ReadMe.txt

@@ -1,6 +1,6 @@
 # IDNA2008_Category Property
-# Date: 2023-05-16
-# © 2023 Unicode®, Inc.
+# Date: PUB_DATE
+# © COPY_YEAR Unicode®, Inc.
 # Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries.
 # For terms of use, see https://www.unicode.org/terms_of_use.html
 

unicodetools/data/security/dev/ReadMe.txt

@@ -1,7 +1,7 @@
 # Unicode Security Data (Confusables and Identifiers)
-# © 2023 Unicode®, Inc.
+# © COPY_YEAR Unicode®, Inc.
 # Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries.
 # For terms of use, see https://www.unicode.org/terms_of_use.html
 
-This directory contains the data files for Version 15.1.0 of
+This directory contains the data files for Version UNI_VER of
 UTS #39: Unicode Security Mechanisms (https://www.unicode.org/reports/tr39/)

unicodetools/data/uca/dev/ReadMe.txt

@@ -1,6 +1,6 @@
 # Unicode Collation Algorithm
-# Date: 2023-01-30
-# © 2023 Unicode®, Inc.
+# Date: PUB_DATE
+# © COPY_YEAR Unicode®, Inc.
 # Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries.
 # For terms of use, see https://www.unicode.org/terms_of_use.html
 #

unicodetools/data/ucd/dev/ReadMe.txt

@@ -1,6 +1,6 @@
 # Unicode Character Database
-# Date: 2023-01-30
-# © 2023 Unicode®, Inc.
+# Date: PUB_DATE
+# © COPY_YEAR Unicode®, Inc.
 # Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries.
 # For terms of use, see https://www.unicode.org/terms_of_use.html
 #
@@ -10,7 +10,7 @@
 # UAX #44, "Unicode Character Database"
 # UTS #51, "Unicode Emoji"
 #
-# The UAXes and UTS #51 can be accessed at https://www.unicode.org/versions/Unicode15.1.0/
+# The UAXes and UTS #51 can be accessed at https://www.unicode.org/versions/UnicodeUNI_VER/
 
-This directory contains draft data files
-for the Unicode Character Database, for Version 15.1.0 of the Unicode Standard.
+This directory contains PUB_STATUS data files
+for the Unicode Character Database, for Version UNI_VER of the Unicode Standard.