release 0.9.2

fix #5 and other improvements:
- new option -z to index archives and compressed files
- new option --zmax=NUM to specify nesting depth of archives to index
- new ugrep-indexer.exe Windows version

Author: genivia-inc

bin/win32/ugrep-indexer.exe

@@ -58,34 +58,41 @@ Indexes are up to date.
 .IP 1
 Indexing check with option \fB-c\fR detected missing and outdated index files.
 .SH EXAMPLES
-Recursively and incrementally index all non-binary files showing progress
+Recursively and incrementally index all non-binary files showing progress:
 .IP
 $ ugrep-indexer -I -v
 .PP
-Index all non-binary files, show progress, follow symbolic links to files (but
-not to directories), and do not index files and directories matching the globs
-in .gitignore:
+Recursively and incrementally index all non-binary files, including non-binary
+files stored in archives and in compressed files, showing progress:
 .IP
-$ ugrep-indexer -I -v -S -X
+$ ugrep-indexer -z -I -v
 .PP
-Recursively force re-indexing of all non-binary files:
+Incrementally index all non-binary files, including archives and compressed
+files, show progress, follow symbolic links to files (but not to directories),
+but do not index files and directories matching the globs in .gitignore:
 .IP
-$ ugrep-indexer -f -I
+$ ugrep-indexer -z -I -v -S -X
 .PP
-Recursively delete all hidden ._UG#_Store index files to restore the directory
-tree to non-indexed:
+Force re-indexing of all non-binary files, including archives and compressed
+files, follow symbolic links to files (but not to directories), but do not
+index files and directories matching the globs in .gitignore:
 .IP
-$ ugrep-indexer -d
+$ ugrep-indexer -f -z -I -v -S -X
 .PP
-Decrease index file storage to a minimum by decreasing indexing accuracy from 5
-(default) to 0:
+Same, but decrease index file storage to a minimum by decreasing indexing
+accuracy from 5 (default) to 0:
 .IP
-$ ugrep-indexer -If0
+$ ugrep-indexer -f -0 -z -I -v -X
 .PP
 Increase search performance by increasing the indexing accuracy from 5
 (default) to 7 at a cost of larger index files:
 .IP
-$ ugrep-indexer -If7
+$ ugrep-indexer -f7zIvX
+.PP
+Recursively delete all hidden ._UG#_Store index files to restore the directory
+tree to non-indexed:
+.IP
+$ ugrep-indexer -d
 .SH BUGS
 Report bugs at:
 .IP

bin/win64/ugrep-indexer.exe

@@ -1,4 +1,4 @@
-.TH UGREP-INDEXER "1" "August 12, 2023" "ugrep-indexer 0.9.1" "User Commands"
+.TH UGREP-INDEXER "1" "December 05, 2023" "ugrep-indexer 0.9.2" "User Commands"
 .SH NAME
 \fBugrep-indexer\fR -- file indexer for accelerated ugrep search
 .SH SYNOPSIS
@@ -11,7 +11,8 @@ The following options are available:
 Usage:
 ugrep\-indexer [\fB\-0\fR|...|\fB\-9\fR] [\fB\-.\fR] [\fB\-c\fR|\fB\-d\fR|\fB\-f\fR] [\fB\-I\fR] [\fB\-q\fR] [\fB\-S\fR] [\fB\-s\fR] [\fB\-X\fR] [\fB\-z\fR] [\fIPATH\fR]
 .TP
-PATH    Optional pathname to the root of the directory tree to index.
+PATH    Optional pathname to the root of the directory tree to index.  The
+default is to recursively index the working directory tree.
 .TP
 \fB\-0\fR, \fB\-1\fR, \fB\-2\fR, \fB\-3\fR, ..., \fB\-9\fR, \fB\-\-accuracy\fR=\fIDIGIT\fR
 Specifies indexing accuracy.  A low accuracy reduces the indexing
@@ -54,53 +55,105 @@ their error messages and warnings are suppressed.
 Display version and exit.
 .TP
 \fB\-v\fR, \fB\-\-verbose\fR
-Produce verbose output.
+Produce verbose output.  Indexed files are indicated with an A for
+archive, C for compressed, B for binary or I for ignored binary.
 .TP
 \fB\-X\fR, \fB\-\-ignore\-files\fR[=\fIFILE\fR]
 Do not index files and directories matching the globs in a FILE
 encountered during indexing.  The default FILE is `.gitignore'.
+This option may be repeated to specify additional files.
 .TP
 \fB\-z\fR, \fB\-\-decompress\fR
-Index the contents of compressed files and archives.
-This option is not yet available in this version.
-ugrep\-indexer 0.9.1 beta
-License BSD\-3\-Clause: <https://opensource.org/licenses/BSD\-3\-Clause>
-Written by Robert van Engelen and others: <https://github.com/Genivia/ugrep>
+Index the contents of compressed files and archives.  When used
+with option \fB\-\-zmax\fR=\fINUM\fR, indexes the contents of compressed files
+and archives stored within archives up to NUM levels deep.
+Supported compression formats: gzip (.gz), compress (.Z), zip,
+bzip2 (requires suffix .bz, .bz2, .bzip2, .tbz, .tbz2, .tb2, .tz2),
+lzma and xz (requires suffix .lzma, .tlz, .xz, .txz),
+lz4 (requires suffix .lz4),
+zstd (requires suffix .zst, .zstd, .tzst),
+brotli (requires suffix .br).
+.TP
+\fB\-\-zmax\fR=\fINUM\fR
+When used with option \fB\-z\fR (\fB\-\-decompress\fR), indexes the contents of
+compressed files and archives stored within archives by up to NUM
+expansion levels deep.  The default \fB\-\-zmax\fR=1 only permits indexing
+uncompressed files stored in cpio, pax, tar and zip archives;
+compressed files and archives are detected as binary files and are
+effectively ignored.  Specify \fB\-\-zmax\fR=2 to index compressed files
+and archives stored in cpio, pax, tar and zip archives.  NUM may
+range from 1 to 99 for up to 99 decompression and de\-archiving
+steps.  Increasing NUM values gradually degrades performance.
+.TP
+Indexes are incrementally updated unless option \fB\-f\fR or \fB\-\-force\fR is specified.
+.TP
+
+.TP
+When option \fB\-I\fR or \fB\-\-ignore\-binary\fR is specified, binary files are ignored
+.TP
+and not indexed.  Searching with ugrep \fB\-\-index\fR still searches binary files
+.TP
+unless ugrep option \fB\-I\fR or \fB\-\-ignore\-binary\fR is specified also.
+.TP
+
+.TP
+Archives and compressed files are incrementally indexed only when option \fB\-z\fR
+.TP
+or \fB\-\-decompress\fR is specified.  Otherwise, archives and compressed files are
+.TP
+indexed as binary files, or are ignored with option \fB\-I\fR or \fB\-\-ignore\-binary\fR.
+.TP
+
+.TP
+To create an indexing log file, specify option \fB\-v\fR or \fB\-\-verbose\fR and redirect
+.TP
+standard output to a log file.  All messages are sent to standard output.
+.TP
+
+.TP
+
 .SH "EXIT STATUS"
 The \fBugrep-indexer\fR utility exits with one of the following values:
 .IP 0
 Indexes are up to date.
 .IP 1
 Indexing check with option \fB-c\fR detected missing and outdated index files.
 .SH EXAMPLES
-Recursively and incrementally index all non-binary files showing progress
+Recursively and incrementally index all non-binary files showing progress:
 .IP
 $ ugrep-indexer -I -v
 .PP
-Index all non-binary files, show progress, follow symbolic links to files (but
-not to directories), and do not index files and directories matching the globs
-in .gitignore:
+Recursively and incrementally index all non-binary files, including non-binary
+files stored in archives and in compressed files, showing progress:
 .IP
-$ ugrep-indexer -I -v -S -X
+$ ugrep-indexer -z -I -v
 .PP
-Recursively force re-indexing of all non-binary files:
+Incrementally index all non-binary files, including archives and compressed
+files, show progress, follow symbolic links to files (but not to directories),
+but do not index files and directories matching the globs in .gitignore:
 .IP
-$ ugrep-indexer -f -I
+$ ugrep-indexer -z -I -v -S -X
 .PP
-Recursively delete all hidden ._UG#_Store index files to restore the directory
-tree to non-indexed:
+Force re-indexing of all non-binary files, including archives and compressed
+files, follow symbolic links to files (but not to directories), but do not
+index files and directories matching the globs in .gitignore:
 .IP
-$ ugrep-indexer -d
+$ ugrep-indexer -f -z -I -v -S -X
 .PP
-Decrease index file storage to a minimum by decreasing indexing accuracy from 5
-(default) to 0:
+Same, but decrease index file storage to a minimum by decreasing indexing
+accuracy from 5 (default) to 0:
 .IP
-$ ugrep-indexer -If0
+$ ugrep-indexer -f -0 -z -I -v -X
 .PP
 Increase search performance by increasing the indexing accuracy from 5
 (default) to 7 at a cost of larger index files:
 .IP
-$ ugrep-indexer -If7
+$ ugrep-indexer -f7zIvX
+.PP
+Recursively delete all hidden ._UG#_Store index files to restore the directory
+tree to non-indexed:
+.IP
+$ ugrep-indexer -d
 .SH BUGS
 Report bugs at:
 .IP

man.sh

@@ -242,7 +242,7 @@ const char ugrep_index_file_magic[5] = "UG#\x03";
 const char *arg_pathname = NULL;
 
 // command-line options
-int    flag_accuracy          = 6;     // -0 ... -9 (--accuracy) default is -6
+int    flag_accuracy          = 5;     // -0 ... -9 (--accuracy) default is -5
 bool   flag_check             = false; // -c (--check)
 bool   flag_decompress        = false; // -z (--decompress)
 bool   flag_delete            = false; // -d (--delete)
@@ -524,7 +524,7 @@ void help()
 {
   std::cout << "\nUsage:\n\nugrep-indexer [-0|...|-9] [-.] [-c|-d|-f] [-I] [-q] [-S] [-s] [-X] [-z] [PATH]\n\n\
     PATH    Optional pathname to the root of the directory tree to index.  The\n\
-            default is to index the working directory tree.\n\n\
+            default is to recursively index the working directory tree.\n\n\
     -0, -1, -2, -3, ..., -9, --accuracy=DIGIT\n\
             Specifies indexing accuracy.  A low accuracy reduces the indexing\n\
             storage overhead at the cost of a higher rate of false positive\n\
@@ -612,8 +612,26 @@ void help()
             "\
             This option is not available in this build configuration of ugrep.\n"
 #endif
-            "\n";
-  version();
+            "\n\
+    Indexes are incrementally updated unless option -f or --force is specified.\n\
+    \n\
+    When option -I or --ignore-binary is specified, binary files are ignored\n\
+    and not indexed.  Searching with ugrep --index still searches binary files\n\
+    unless ugrep option -I or --ignore-binary is specified also.\n\
+    \n\
+    Archives and compressed files are incrementally indexed only when option -z\n\
+    or --decompress is specified.  Otherwise, archives and compressed files are\n\
+    indexed as binary files, or are ignored with option -I or --ignore-binary.\n\
+    \n\
+    To create an indexing log file, specify option -v or --verbose and redirect\n\
+    standard output to a log file.  All messages are sent to standard output.\n\
+    \n\
+    The ugrep-indexer utility exits with one of the following values:\n\
+    0       Indexes are up to date.\n\
+    1       Some indexes appear to be stale and are outdated or missing.\n\
+    \n";
+
+  exit(EXIT_SUCCESS);
 }
 
 // display usage information and exit
@@ -629,7 +647,7 @@ void warning(const char *message, const char *arg = NULL)
   if (flag_no_messages)
     return;
   fflush(stdout);
-  fprintf(stderr, "ugrep-indexer: warning: %s%s%s\n", message, arg != NULL ? " " : "", arg != NULL ? arg : "");
+  printf("ugrep-indexer: warning: %s%s%s\n", message, arg != NULL ? " " : "", arg != NULL ? arg : "");
 }
 
 // display an error message unless option -s (--no-messages)
@@ -645,7 +663,7 @@ void error(const char *message, const char *arg)
   const char *errmsg = strerror(errno);
 #endif
   fflush(stdout);
-  fprintf(stderr, "ugrep-indexer: error: %s%s%s: %s\n", message, arg != NULL ? " " : "", arg != NULL ? arg : "", errmsg);
+  printf("ugrep-indexer: error: %s%s%s: %s\n", message, arg != NULL ? " " : "", arg != NULL ? arg : "", errmsg);
 }
 
 #ifdef HAVE_LIBZ
@@ -655,7 +673,7 @@ void cannot_decompress(const char *pathname, const char *message)
   if (!flag_verbose || flag_no_messages)
     return;
   fflush(stdout);
-  fprintf(stderr, "ugrep-indexer: warning: cannot decompress %s: %s\n", pathname, message != NULL ? message : "");
+  printf("ugrep-indexer: warning: cannot decompress %s: %s\n", pathname, message != NULL ? message : "");
 }
 #endif
 
@@ -817,6 +835,10 @@ void options(int argc, const char **argv)
     }
   }
 
+  // -q overrides -v
+  if (flag_quiet)
+    flag_verbose = false;
+
   // -c silently overrides -d and -f
   if (flag_check)
     flag_delete = flag_force = false;
@@ -1430,8 +1452,6 @@ void cat(const std::string& pathname, std::stack<Entry>& dir_entries, std::vecto
 // recursively delete index files
 void deleter(const char *pathname)
 {
-  flag_no_messages = true;
-
   std::stack<Entry> dir_entries;
   std::vector<Entry> file_entries;
   std::string index_filename;
@@ -1444,6 +1464,7 @@ void deleter(const char *pathname)
   int64_t ign_files = 0;
   uint64_t index_time;
   uint64_t last_time;
+  uint64_t num_removed = 0;
 
   // pathname to the directory tree to index or .
   if (pathname == NULL)
@@ -1459,13 +1480,25 @@ void deleter(const char *pathname)
 
     cat(visit.pathname, dir_entries, file_entries, num_dirs, num_links, num_other, ign_dirs, ign_files, index_time, last_time, true);
 
-    // if index time is nonzero, there is a valid index file in this directory we should remove
+    // if index time is nonzero, there is a valid index file in this directory that we should remove
     if (index_time > 0)
     {
       index_filename.assign(visit.pathname).append(PATHSEPSTR).append(ugrep_index_filename);
-      remove(index_filename.c_str());
+      if (remove(index_filename.c_str()) != 0)
+      {
+        error("cannot remove", index_filename.c_str());
+      }
+      else
+      {
+        ++num_removed;
+        if (flag_verbose)
+          printf("%13" PRIu64 " %s\n", num_removed, index_filename.c_str());
+      }
     }
   }
+
+  if (!flag_quiet)
+    printf("\n%13" PRIu64 " indexes removed from %" PRIu64 " directories\n\n", num_removed, num_dirs);
 }
 
 // recursively index files