Merge remote-tracking branch 'origin/topic/timw/remove-bro-references'

* origin/topic/timw/remove-bro-references:
  Update man page with changes to other documentation
  Remove references to bro-pkg.meta and bro-pkg.index
  Remove references to legacy bro naming

Author: timwoj

CHANGES

@@ -1,3 +1,86 @@
+2.13.0-46 | 2023-06-30 12:47:34 -0700
+
+  * Update man page with changes to other documentation (Tim Wojtulewicz, Corelight)
+
+  * Remove references to bro-pkg.meta and bro-pkg.index (Tim Wojtulewicz, Corelight)
+
+  * Remove references to legacy bro naming (Tim Wojtulewicz, Corelight)
+
+    This intentionally leaves behind references to bro-pkg.meta and __bro_plugin__
+    but removes all of the other references to scripts and files that have been
+    deprecated and/or removed since Zeek 3.0.
+
+  * User-visible warning when bro-pkg.meta is in use. (Arne Welzel, Corelight)
+
+    Relates to #119
+
+  * bundle/unbundle: Rework built-in dependency check (Arne Welzel, Corelight)
+
+    Extend unbundle to validate dependencies of contained packages and log a
+    warning if any are not fulfilled or fail to validate. This can happen when
+    a bundle is created with --nodeps or the target system has different
+    built-in packages as compared to the source system.
+
+    Built-in packages are included in a bundle's manifest.txt file in a new
+    [meta] section.
+
+  * manager: Do not include built-in packages in manifest (Arne Welzel, Corelight)
+
+    Bugfix and test for mistakenly recording built-in packages within
+    the manifest.json file. The built-in information is only ever
+    requested from Zeek.
+
+  * package: Add PackageVersion (Arne Welzel, Corelight)
+
+    Make checking if a package version fullfills a version_spec re-usable.
+
+    This certainly changes the messages produced, but doesn't seem there
+    are many tests upset about that.
+
+  * Add built-in package awareness (Arne Welzel, Corelight)
+
+    This change uses Zeek 6.0's --build-info flags to discover the "zkg.provides"
+    entry and makes ``zkg`` aware of these as installed packages.
+
+    It introduces the following changes
+
+    * A pseudo zeek-builtin:// scheme for git_urls to recognize when a
+      Package instance relates to a built-in package
+    * The package source name "zeek-builtin" is now reserved
+    * A zkg bundle's manifest now contains a new section called `[bundle_builtin]`
+      listing any packages that are expected to exists as built-in packages on
+      the target system. The content is derived from the built-in packages on
+      the system building the bundle. This is somewhat narrow and opinionated,
+      but at the same time avoids the complexity of any constructed scenarios
+      one can come up with here.
+    * A new tracking method "builtin" exists. It should act as version but
+      makes it explicit that a package is built-in.
+
+    Most commands have been extended to either ignore (purge) or fail
+    (install, remove, test, load, unload, pin, unpin) when they detect
+    a built-in package being operated on. The commands list and info
+    received a ``--include-builtin`` option for explicitly including
+    built-in packages in the output.
+
+  * zkg: Log warnings by default (Arne Welzel, Corelight)
+
+    Currently, to see warnings generated by zkg a user has to pass a single -v.
+    Change this behavior by always installing the StreamHandler() with a
+    warning level so that warnings are displayed by default.
+
+  * package: Add a few __repr__() implementations (Arne Welzel, Corelight)
+
+    Not having __repr__() implemented makes it more difficult than needed
+    when using print / IPython shell to look around and discover state.
+    Implement some of them to aid that development style.
+
+    They aren't perfect, but better than not having them at all.
+
+  * _util: Add ZeekInfo helper class (Arne Welzel, Corelight)
+
+    Mostly for collecting executable paths for now, but could also see
+    the ZEEKPATH collections being done there down the road.
+
 2.13.0-32 | 2023-06-20 15:55:21 -0700
 
   * Add --ignore-dirty-git option to test and install commands (Tim Wojtulewicz, Corelight)

VERSION

@@ -1 +1 @@
-2.13.0-32
+2.13.0-46

doc/man/zkg.1

@@ -22,12 +22,6 @@ This is the package's metadata file in INI file format and may contain
 :ref:`additional fields <metadata-fields>` that describe the package as well
 as how it inter-operates with Zeek, the package manager, or other packages.
 
-.. note::
-
-   :file:`zkg.meta` is the canonical metadata file name used :program:`since
-   zkg v2.0`.  The previous metadata file name of :file:`bro-pkg.meta` is also
-   accepted when no :file:`zkg.meta` exists.
-
 .. _package-shorthand-name:
 
 Note that the shorthand name for your package that may be used by :ref:`zkg
@@ -36,7 +30,7 @@ last component of its git URL. E.g. a package at ``https://github.com/zeek/foo``
 may be referred to as **foo** when using :program:`zkg` and a Zeek
 script that wants to load all the scripts within that package can use:
 
-.. code-block:: bro
+.. code-block:: zeek
 
   @load foo
 
@@ -229,11 +223,6 @@ though the following step are the essentials needed to create a package.
      script_dir = scripts/Demo/Rot13
      build_command = ./configure && make
 
-   .. note::
-
-      See :ref:`legacy-bro-support` for notes on configuring packages to
-      support Bro 2.5 or earlier.
-
 #. Add example script code:
 
    .. code-block:: console
@@ -257,9 +246,7 @@ though the following step are the essentials needed to create a package.
               *p  = (*p - b + 13) % 26 + b;
               }
 
-          BroString* bs = new BroString(1, reinterpret_cast<byte_vec>(rot13),
-                                        strlen(rot13));
-          return new StringVal(bs);
+          return make_intrusive<StringVal>(strlen(rot13), rot13);
           %}
 
 #. Commit everything to git:
@@ -576,11 +563,6 @@ An example :file:`zkg.meta`::
   script_dir = scripts/Demo/Rot13
   build_command = ./configure && make
 
-.. note::
-
-   See :ref:`legacy-bro-support` for notes on configuring packages to
-   support Bro 2.5 or earlier.
-
 The default CMake skeleton for Zeek plugins will use :file:`build/` as the
 directory for the final/built version of the plugin, which matches the defaulted
 value of the omitted `plugin_dir` metadata field.
@@ -591,31 +573,6 @@ script components, the "plugin" part is always unconditionally loaded by Zeek,
 but the "script" components must either be explicitly loaded (e.g. :samp:`@load
 {<package_name>}`) or the package marked as :ref:`loaded <load-command>`.
 
-.. _legacy-bro-support:
-
-Supporting Older Bro Versions
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Plugin skeletons generated before Bro v2.6 and also any packages
-that generally want to support such Bro versions need to pass
-an additional configuration option such as::
-
-    build_command = ./configure --bro-dist=%(bro_dist)s && make
-
-See the :ref:`Value Interpolation <metadata-interpolation>`
-section for more information on what the ``%(bro_dist)s``
-string does, but a brief explanation is that it will expand to
-a path containing the Bro source-code on the user's system.
-For newer versions of Bro, packages are able to work entirely
-with the installation path and don't require original source code.
-
-Also note that other various Zeek scripting and CMake infrastructure may
-have changed between Bro v2.6 and Zeek v3.0.  So if you plan to support
-older version of Bro (before the Zeek rename), then you should keep an eye
-out for various things that got renamed.  For example, the `zeek_init` event
-won't exist in any version before Zeek v3.0, nor will any CMake macros
-that start with `zeek_plugin`.
-
 .. _metadata-interpolation:
 
 Value Interpolation
@@ -624,18 +581,14 @@ Value Interpolation
 The `build_command field`_ may reference the settings any given user has in
 their customized :ref:`package manager config file <zkg-config-file>`.
 
-For example, if a metadata field's value contains the ``%(bro_dist)s`` string,
+For example, if a metadata field's value contains the ``%(zeek_dist)s`` string,
 then :program:`zkg` operations that use that field will automatically
-substitute the actual value of `bro_dist` that the user has in their local
+substitute the actual value of `zeek_dist` that the user has in their local
 config file.  Note the trailing 's' character at the end of the interpolation
-string, ``%(bro_dist)s``, is intended/necessary for all such interpolation
-usages.  Note that :program:`since zkg v2.0`, `zeek_dist` is the canonical name
-for `bro_dist` within the :ref:`zkg config file <zkg-config-file>`,
-but either one means the same thing and should work.  To support older
-versions of :program:`bro-pkg`, you'd want to use `bro_dist` in package
-metadata files.
-
-Besides the `bro_dist`/`zeek_dist` config keys, any key inside the
+string, ``%(zeek_dist)s``, is intended/necessary for all such interpolation
+usages.
+
+Besides the `zeek_dist` config key, any key inside the
 `user_vars` sections of their :ref:`package manager config file
 <zkg-config-file>` that matches the key of an entry in the package's
 `user_vars field`_ will be interpolated.
@@ -777,38 +730,24 @@ An example :file:`zkg.meta`::
 The field is a list of dependency names and their version requirement
 specifications.
 
-A dependency name may be either `zeek`, `zkg`, `bro`, `bro-pkg`,
+A dependency name may be either `zeek`, `zkg`,
 a full git URL of the package, or a :ref:`package shorthand name
 <package-shorthand-name>`.
 
-- The special `zeek` and `bro` dependencies refers not to a package,
-  but the version of Zeek that the package requires in order to function.  If
-  the user has :program:`zeek-config` or :program:`bro-config` in their
-  :envvar:`PATH` when installing/upgrading a package that specifies a `zeek` or
-  `bro` dependency, then :program:`zkg` will enforce that the requirement is
-  satisfied.
+- The special `zeek` dependency refers not to a package, but the version of Zeek
+  that the package requires in order to function.  If the user has
+  :program:`zeek-config` in their :envvar:`PATH` when installing/upgrading a
+  package that specifies a `zeek` dependency, then :program:`zkg` will enforce
+  that the requirement is satisfied.
 
-  .. note::
-
-     In this context, `zeek` and `bro` mean the same thing -- the
-     later is maintained for backwards compatibility while the former
-     became available :program:`since zkg v2.0`.
-
-- The special `zkg` and `bro-pkg` dependencies refers to the version of the
+- The special `zkg` dependency refers to the version of the
   package manager that is required by the package.  E.g. if a package takes
   advantage of new features that are not present in older versions of the
   package manager, then it should indicate that so users of those old version
   will see an error message an know to upgrade instead of seeing a cryptic
   error/exception, or worse, seeing no errors, but without the desired
   functionality being performed.
 
-  .. note::
-
-     This feature itself, via use of a `bro-pkg` dependency, is only
-     available :program:`since bro-pkg v1.2` while a `zkg` dependency is only
-     recognized :program:`since zkg v2.0`.  Otherwise, `zkg` and `bro-pkg` mean
-     the same thing in this context.
-
 - The full git URL may be directly specified in the `depends` metadata if you
   want to force the dependency to always resolve to a single, canonical git
   repository.  Typically this is the safe approach to take when listing

doc/package.rst

@@ -141,7 +141,7 @@ have Zeek automatically load the scripts from all
 :ref:`installed <install-command>` packages that are also marked as
 ":ref:`loaded <load-command>`" add:
 
-.. code-block:: bro
+.. code-block:: zeek
 
   @load packages
 

doc/quickstart.rst

@@ -22,12 +22,6 @@ usernames or some unique way of identifying the organization/person that
 maintains Zeek packages.  However, a source is free to use a flat organization
 with a single, top-level :file:`zkg.index`.
 
-.. note::
-
-   The magic index file name of :file:`zkg.index` is available :program:`since
-   zkg v2.0`.  For compatibility purposes, the old index file name of
-   :file:`bro-pkg.index` is also still supported.
-
 After creating a git repo for the package source and adding package index files
 to it, it's ready to be used by :ref:`zkg <zkg>`.
 
@@ -36,10 +30,9 @@ to it, it's ready to be used by :ref:`zkg <zkg>`.
 Package Index Files
 -------------------
 
-Files named :file:`zkg.index` (or the legacy :file:`bro-pkg.index`) are used to
-describe the :doc:`Zeek Packages <package>` found within the package source.
-They are simply a list of git URLs pointing to the git repositories of
-packages.  For example::
+Files named :file:`zkg.index` are used to describe the :doc:`Zeek Packages
+<package>` found within the package source.  They are simply a list of git URLs
+pointing to the git repositories of packages.  For example::
 
   https://github.com/zeek/foo
   https://github.com/zeek/bar
@@ -75,9 +68,8 @@ Aggregating Metadata
 --------------------
 
 The maintainer/operator of a package source may choose to periodically aggregate
-the metadata contained in its packages' :file:`zkg.meta` (and legacy
-:file:`bro-pkg.meta`) files.  The :ref:`zkg refresh <refresh-command>`
-is used to perform the task.  For example:
+the metadata contained in its packages' :file:`zkg.meta` files.  The :ref:`zkg
+refresh <refresh-command>` is used to perform the task. For example:
 
 .. code-block:: console
 

doc/source.rst

@@ -1,5 +1,3 @@
-.. _bro-pkg:
-
 .. _zkg:
 
 zkg Command-Line Tool
@@ -217,8 +215,6 @@ template info
     :prog: zkg
     :path: template info
 
-.. _bro-pkg-config-file:
-
 .. _zkg-config-file:
 
 Config File

doc/zkg.rst

@@ -1,8 +1,8 @@
 ### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
 XXXX-XX-XX XX:XX:XX DEBUG    found source clone of "one" at <...>/one
 XXXX-XX-XX XX:XX:XX DEBUG    refresh "one": pulling <...>/one
-XXXX-XX-XX XX:XX:XX WARNING  <...>/bro-pkg.meta: missing metadata file
-XXXX-XX-XX XX:XX:XX WARNING  skipping aggregation of <...>/bad_pkg: bad metadata: missing zkg.meta (or bro-pkg.meta) metadata file
+XXXX-XX-XX XX:XX:XX WARNING  <...>/zkg.meta: missing metadata file
+XXXX-XX-XX XX:XX:XX WARNING  skipping aggregation of <...>/bad_pkg: bad metadata: missing zkg.meta metadata file
 XXXX-XX-XX XX:XX:XX DEBUG    metadata refresh: 8 additions (alice/bar, alice/baz, alice/foo, alice/i-have-no-scripts, alice/new_pkg, alice/qux, bob/corge, bob/grault), 0 changes, 0 removals
 XXXX-XX-XX XX:XX:XX INFO     committed package source "one" metadata update
 XXXX-XX-XX XX:XX:XX DEBUG    fetch package one<...>/foo

testing/baselines/tests.install-bro-pkg-warning/stderr

@@ -1,7 +1,7 @@
 ### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
 XXXX-XX-XX XX:XX:XX DEBUG    found source clone of "one" at <...>/one
 XXXX-XX-XX XX:XX:XX DEBUG    refresh "one": pulling <...>/one
-XXXX-XX-XX XX:XX:XX WARNING  <...>/bro-pkg.meta: missing metadata file
-XXXX-XX-XX XX:XX:XX WARNING  skipping aggregation of <...>/bad_pkg: bad metadata: missing zkg.meta (or bro-pkg.meta) metadata file
+XXXX-XX-XX XX:XX:XX WARNING  <...>/zkg.meta: missing metadata file
+XXXX-XX-XX XX:XX:XX WARNING  skipping aggregation of <...>/bad_pkg: bad metadata: missing zkg.meta metadata file
 XXXX-XX-XX XX:XX:XX DEBUG    metadata refresh: 8 additions (alice/bar, alice/baz, alice/foo, alice/i-have-no-scripts, alice/new_pkg, alice/qux, bob/corge, bob/grault), 0 changes, 0 removals
 XXXX-XX-XX XX:XX:XX DEBUG    fetch package one<...>/foo