feat(flowd-go): switch to firefly backend for measurement flow events, v2.5.0+ only

Switch from fireflyp plugin to the firefly backend:
- fireflyb backend now emits SciTags firefly datagrams at the start/end of
  measurement flows, sending to global.scitags.org by default
- This allows perfSONAR measurement flows to appear in ESnet Stardust dashboard
- Requires flowd-go >= 2.5.0 (no longer support 2.4.x)

Update documentation:
- Fix ESnet Stardust dashboard link to the 12-hour SciTags data view
- Clarify firefly backend vs plugin distinction
- Remove 2.4.2 references and --no-firefly-receiver obsolete flag
- Update version to 1.2.0 for perfSONAR-install-flowd-go.sh

Updated files:
- perfSONAR-install-flowd-go.sh (v1.2.0) - main installation script
- perfSONAR-toolkit-install.sh - remove obsolete flags
- install-perfsonar-toolkit.md - update ESnet Stardust integration docs
- scitags-fireflies.md - clarify backend/plugin distinction
- scripts.sha256, hash sidecars - update hashes

Author: root

docs/perfsonar/scitags-fireflies.md

@@ -57,7 +57,7 @@ single statically-linked binary (with embedded eBPF programs for packet marking
 | **Packet marking** | Uses eBPF to stamp the IPv6 Flow Label on egress packets matching tracked flows |
 | **Firefly emission** | Sends UDP fireflies to local or remote collectors when flows start, continue, or end |
 | **perfSONAR plugin** | Built-in plugin that marks *all* egress traffic with a configured experiment/activity ID — ideal for dedicated measurement hosts |
-| **fireflyp plugin** | Listens for incoming firefly UDP datagrams and relays them verbatim to one or more upstream collectors (global or regional) before processing them locally — requires flowd-go ≥ 2.5.0 |
+| **firefly backend** | Sends JSON firefly datagrams at the start, during, and end of flows to a collector (default: global.scitags.org) — requires flowd-go >= 2.5.0 |
 | **Low overhead** | Statically compiled Go binary; no Python, no containers, no runtime dependencies beyond `libz` and `libelf` (typically already present) |
 | **RPM packaged** | Available from the SciTags repository for EL9 (`x86_64` and `aarch64`) |
 
@@ -82,24 +82,26 @@ For perfSONAR deployments the full pipeline is:
   perfSONAR tests
        │
        ▼
-  ┌──────────────┐   all flows    ┌──────────────────────┐
-  │ perfsonar    │──────────────▶ │ marker backend       │
-  │ plugin       │                │ (eBPF IPv6 label)    │
-  └──────────────┘                └──────────────────────┘
-
-  Local applications
-  (pScheduler etc.)
-       │ UDP firefly
-       ▼
-  ┌──────────────┐   forward      ┌──────────────────────┐
-  │ fireflyp     │──────────────▶ │ global.scitags.org   │
-  │ plugin       │                │ (UDP collector)      │
-  └──────────────┘                └──────────────────────┘
+  ┌──────────────┐   all flows    ┌──────────────────────────┐
+  │ perfsonar    │──────────────▶ │ marker backend           │
+  │ plugin       │                │ (eBPF IPv6 Flow Label)   │
+  └──────────────┘                └──────────────────────────┘
+       │
+       ├─────────────────────┐
+       │ (start/end events)  │
+       ▼                     ▼
+  ┌──────────────────────────────────────────┐
+  │ firefly backend                          │
+  │ (generates flow event notifications)     │
+  │                                          │
+  │ → global.scitags.org (UDP), collected in │
+  │   ESnet Stardust dashboard               │
+  └──────────────────────────────────────────┘
 ```
 
 For perfSONAR deployments the recommended configuration uses the **perfsonar** plugin (marks all egress
-traffic) together with the **marker** backend (eBPF-based IPv6 Flow Label stamping) and the **fireflyp**
-plugin (forwarding firefly announcements to regional or global collectors).
+traffic), the **marker** backend (eBPF-based IPv6 Flow Label stamping), and the **firefly** backend
+(emitting flow start/end notifications to a global or regional collector).
 
 ---
 
@@ -111,13 +113,12 @@ view of active flows across the network. Collectors can be operated at site, reg
 ### ESnet Stardust — the global SciTags monitoring platform
 
 Firefly datagrams sent to `global.scitags.org` are ingested by the
-**[ESnet Stardust](https://stardust.es.net/)** platform, which provides a
-[Scientific Network Tags dashboard](https://dashboard.stardust.es.net/goto/dfejd22wk1k3kb?orgId=2)
+**[ESnet Stardust](https://dashboard.stardust.es.net/d/b8dddac0-5b24-4739-9c8d-e88a05c1344f/scientific-network-tags3a-rande-dashboard?orgId=2&from=now-12h&to=now&timezone=browser)** platform, which provides a dashboard
 showing real-time and historical flow counts broken down by experiment and activity across all
 participating sites.
 
-> **The `fireflyp` plugin is the only mechanism by which perfSONAR measurement flows appear in
-> Stardust.** eBPF packet marking alone (the `marker` backend) stamps packets in-flight but does
+> **The `firefly` backend is the primary mechanism by which perfSONAR measurement flows appear in
+> Stardust.** The `marker` backend stamps packets in-flight but does not generate flow event notifications.
 > *not* notify any collector. Only when flowd-go also runs the `fireflyp` plugin — forwarding
 > locally-emitted firefly datagrams to `global.scitags.org` — will the flows be visible in the
 > Stardust monitoring system.

docs/perfsonar/tools_scripts/perfSONAR-install-flowd-go.sh

@@ -4,7 +4,12 @@ set -euo pipefail
 # perfSONAR-install-flowd-go.sh
 # Install and configure flowd-go (SciTags flow-marking daemon) for perfSONAR hosts.
 #
-# Version: 1.1.0 - 2026-02-27
+# Version: 1.2.0 - 2026-03-04
+#   - Switch from fireflyp plugin to firefly backend for emitting flow start/end events
+#   - Update to flowd-go 2.5.0+ (requires flowd-go >= 2.5.0)
+#   - Remove support for flowd-go 2.4.x (use v1.1.0 of this script for 2.4.x support)
+#   - Simplify configuration: firefly backend replaces fireflyp plugin for main flow emission
+#   - Update ESnet Stardust documentation links
 # Author: Shawn McKee, University of Michigan
 # Acknowledgements: Supported by IRIS-HEP and OSG-LHC
 #
@@ -16,27 +21,23 @@ set -euo pipefail
 #   --activity-id N            Set the SciTags activity ID (default: 2 = network testing)
 #   --interfaces LIST          Comma-separated list of NIC names for packet marking
 #                              (auto-detected from /etc/perfSONAR-multi-nic-config.conf if omitted)
-#   --firefly-receiver HOST    Firefly collector address for the fireflyp plugin
+#   --firefly-receiver HOST    Collector address for the fireflyb backend
 #                              (default: global.scitags.org; set to empty to disable)
 #   --firefly-receiver-port N  UDP port on the firefly collector (default: 10514)
-#   --firefly-bind-port N      Local UDP port flowd-go listens on for fireflies (default: 10514)
-#   --no-firefly-receiver      Disable the fireflyp plugin entirely (eBPF marking only)
 #   --list-experiments         Show available experiment IDs and exit
 #   --yes                      Skip interactive prompts
 #   --dry-run                  Preview actions without making changes
 #   --uninstall                Remove flowd-go and its configuration
 #   --version                  Show script version
 #   --help, -h                 Show this help message
 #
-# Firefly Receiver:
-#   When --firefly-receiver is set (the default), flowd-go will also run the
-#   'fireflyp' plugin which listens for locally-emitted firefly UDP datagrams
-#   and forwards them verbatim to the configured collector. This requires
-#   flowd-go >= 2.5.0 (built after scitags/flowd-go#49 is merged). Older
-#   versions will ignore or reject the 'fireflyp' stanza; use --no-firefly-
-#   receiver if you are running an older RPM.
-
-VERSION="1.1.0"
+# Firefly Backend:
+#   When --firefly-receiver is set (the default), the 'firefly' backend in flowd-go
+#   will emit firefly JSON datagrams at the start and end of each flow, sending them
+#   to the configured collector. This allows perfSONAR measurement flows to be tracked
+#   by ESnet Stardust and other SciTags-aware systems. Requires flowd-go >= 2.5.0.
+
+VERSION="1.2.0"
 PROG_NAME="$(basename "$0")"
 LOG_FILE="/var/log/perfSONAR-install-flowd-go.log"
 
@@ -49,14 +50,14 @@ DRY_RUN=false
 UNINSTALL=false
 MULTI_NIC_CONF="/etc/perfSONAR-multi-nic-config.conf"
 FLOWD_GO_CONF="/etc/flowd-go/conf.yaml"
-FLOWD_GO_RPM_URL="https://linuxsoft.cern.ch/repos/scitags9al-testing/x86_64/os/Packages/f/flowd-go-2.4.2-1.x86_64.rpm"
+FLOWD_GO_RPM_URL="https://linuxsoft.cern.ch/repos/scitags9al-testing/x86_64/os/Packages/f/flowd-go-2.5.0-1.x86_64.rpm"
 
-# Firefly receiver defaults
+# Firefly backend defaults
 # global.scitags.org is the authoritative global SciTags firefly collector.
+# Fireflies are sent at the start and end of each flow.
 # Regional DNS aliases (e.g. us-east.scitags.org) may be used in future.
 FIREFLY_RECEIVER="global.scitags.org"
 FIREFLY_RECEIVER_PORT=10514
-FIREFLY_BIND_PORT=10514
 
 # Experiment name lookup
 declare -A EXPERIMENT_NAMES=(
@@ -191,26 +192,32 @@ build_config() {
     done
     iface_yaml+="]"
 
-    # Build optional firefly plugin block
-    # Requires flowd-go >= 2.5.0 (scitags/flowd-go#49).
-    # Note: In flowd-go 2.5.0+, the YAML key is 'firefly' (not 'fireflyp').
-    local fireflyp_block=""
+    # Build perfsonar plugin block (always included)
+    local plugins_block
+    plugins_block=$(printf 'plugins:\n  perfsonar:\n    activityId: %s\n    experimentId: %s' \
+        "$ACTIVITY_ID" "$EXPERIMENT_ID")
+
+    # Build firefly backend block
+    local firefly_backend=""
     if [ -n "$FIREFLY_RECEIVER" ]; then
-        fireflyp_block=$(printf '  firefly:\n    bindAddress: "127.0.0.1"\n    bindPort: %s\n    fireflyReceivers:\n      - address: "%s"\n        port: %s\n' \
-            "$FIREFLY_BIND_PORT" "$FIREFLY_RECEIVER" "$FIREFLY_RECEIVER_PORT")
+        firefly_backend=$(printf '  firefly:\n    sendToCollector: true\n    collectorAddress: "%s"\n    collectorPort: %s\n    prependSyslog: false\n    enrich: false' \
+            "$FIREFLY_RECEIVER" "$FIREFLY_RECEIVER_PORT")
     fi
 
-    local plugins_block
-    if [ -n "$fireflyp_block" ]; then
-        plugins_block=$(printf 'plugins:\n  perfsonar:\n    activityId: %s\n    experimentId: %s\n%s' \
-            "$ACTIVITY_ID" "$EXPERIMENT_ID" "$fireflyp_block")
+    # Build marker backend block
+    local marker_backend
+    marker_backend=$(printf '  marker:\n    targetInterfaces: %s\n    markingStrategy: label\n    forceHookRemoval: true' \
+        "$iface_yaml")
+
+    # Combine everything
+    local backends_block
+    if [ -n "$firefly_backend" ]; then
+        backends_block=$(printf 'backends:\n%s\n%s' "$firefly_backend" "$marker_backend")
     else
-        plugins_block=$(printf 'plugins:\n  perfsonar:\n    activityId: %s\n    experimentId: %s' \
-            "$ACTIVITY_ID" "$EXPERIMENT_ID")
+        backends_block=$(printf 'backends:\n%s' "$marker_backend")
     fi
 
-    printf '%s\n\nbackends:\n  marker:\n    targetInterfaces: %s\n    markingStrategy: label\n    forceHookRemoval: true\n' \
-        "$plugins_block" "$iface_yaml"
+    printf '%s\n\n%s\n' "$plugins_block" "$backends_block"
 }
 
 install_flowd_go() {
@@ -284,7 +291,7 @@ install_flowd_go() {
     echo "  Activity:   network testing (ID=$ACTIVITY_ID)"
     if [ -n "$FIREFLY_RECEIVER" ]; then
         echo "  Firefly collector: $FIREFLY_RECEIVER:$FIREFLY_RECEIVER_PORT"
-        echo "  (fireflyp plugin forwards locally-received fireflies to the collector)"
+        echo "  (fireflyb backend sends fireflies at start and end of measurement flows)"
     else
         echo "  Firefly collector: disabled (eBPF packet marking only)"
     fi
@@ -349,12 +356,7 @@ parse_cli() {
             --firefly-receiver-port)
                 FIREFLY_RECEIVER_PORT="${2:-10514}"
                 shift 2 ;;
-            --firefly-bind-port)
-                FIREFLY_BIND_PORT="${2:-10514}"
-                shift 2 ;;
-            --no-firefly-receiver)
-                FIREFLY_RECEIVER=""
-                shift ;;
+
             --list-experiments)
                 list_experiments
                 exit 0 ;;

docs/perfsonar/tools_scripts/perfSONAR-toolkit-install.sh

@@ -32,8 +32,7 @@ set -euo pipefail
 #   --dry-run              Print steps but do not execute destructive operations
 #   --no-flowd-go          Skip flowd-go (SciTags) installation (installed by default)
 #   --experiment-id N      SciTags experiment ID for flowd-go (1-14; interactive prompt if omitted)
-#   --no-firefly-receiver  Disable fireflyp plugin in flowd-go config (use with flowd-go 2.4.x RPM)
-#                          Requires flowd-go >= 2.5.0; omit with current 2.4.2 RPM to avoid errors
+
 #   --exporter-allowlist   Override default exporter endpoint allow-list (comma-separated CIDRs/IPs)
 #                          Default: AGLT2 + CERN monitoring subnets (matching container protection)
 #                          (/node_exporter/metrics and /perfsonar_host_exporter/)
@@ -48,7 +47,6 @@ AUTO_YES=false
 NON_INTERACTIVE=false
 INSTALL_FLOWD_GO=true
 FLOWD_GO_EXPERIMENT_ID=""
-NO_FIREFLY_RECEIVER=false
 NO_EXPORTER_ACLS=false
 BUNDLE="toolkit"
 LE_FQDN=""
@@ -108,7 +106,7 @@ parse_cli() {
       --email)          LE_EMAIL="${2:-}"; shift 2;;
       --no-flowd-go)         INSTALL_FLOWD_GO=false; shift;;
       --experiment-id)       FLOWD_GO_EXPERIMENT_ID="${2:-}"; shift 2;;
-      --no-firefly-receiver) NO_FIREFLY_RECEIVER=true; shift;;
+
       --exporter-allowlist)  EXPORTER_ALLOWLIST="${2:-}"; shift 2;;
       --no-exporter-acls)    NO_EXPORTER_ACLS=true; EXPORTER_ALLOWLIST=""; shift;;
       --help|-h)        sed -n '1,80p' "$0"; exit 0;;
@@ -335,7 +333,7 @@ step_flowd_go() {
   local flowd_cmd=("$HELPER_DIR/tools_scripts/perfSONAR-install-flowd-go.sh")
   [ "$AUTO_YES" = true ]             && flowd_cmd+=(--yes)
   [ -n "$FLOWD_GO_EXPERIMENT_ID" ]   && flowd_cmd+=(--experiment-id "$FLOWD_GO_EXPERIMENT_ID")
-  [ "$NO_FIREFLY_RECEIVER" = true ]  && flowd_cmd+=(--no-firefly-receiver)
+
 
   if [ -x "$HELPER_DIR/tools_scripts/perfSONAR-install-flowd-go.sh" ]; then
     run "${flowd_cmd[@]}" || true

docs/perfsonar/tools_scripts/scripts.sha256

@@ -18,8 +18,8 @@ e1944123e17c89e8f202cca960f147397d64ae1e675af132c84b02ced2564abb  node_exporter.
 50dfab90bc21d5c566b713f48b00b079a32a8b8756432a0d0f66ac6a64e6e581  perfSONAR-health-monitor.sh
 e184c633ca9ca5d3d79321213843a4eaa951e5596d3dc05f082e5f76e860e580  install_tools_scripts.sh
 7be726de5dfdbe8f7f5ac8e803b0b71e8f98f1ba274ca70b42f8eba4822cc67b  perfSONAR-orchestrator.sh
-26ebf225dd223ed783190e395fd21b6346d95abf036434c4b51f8cc2db5cad68  perfSONAR-toolkit-install.sh
+712a2b699df90fca02e7c04912beb0aac991bc2f56ea785636683d7f3fb13180  perfSONAR-toolkit-install.sh
 cd0e7afd1ca7a20a972e585018802be0cb6f67cd3ffee449dea45d785c23145a  perfSONAR-configure-exporter-acls.sh
 2615a29d65e285391adb547046584c4534ea548e69571b67e0cf35773b010c57  perfSONAR-diagnostic-report.sh
-6854dc66a5d11e4c2fe5a439c4f3649d93f4e376fe8567c96f6abec0d42c6a6b  perfSONAR-install-flowd-go.sh
+ea703d8e330276bfd09af6d1415aa472b62813a861c6f310d93cfd378cfa8ad6  perfSONAR-install-flowd-go.sh
 39d226a857eb1a0956003c75ca8b558fcb55c63176286ca9597f031d08cb38a7  docs/perfsonar/tools_scripts/update-perfsonar-deployment.sh

docs/personas/quick-deploy/install-perfsonar-toolkit.md

@@ -1587,40 +1587,32 @@ service:
     1. Downloads and installs the `flowd-go` RPM from the SciTags repository
     2. Prompts for the SciTags experiment ID (ATLAS, CMS, LHCb, etc.)
     3. Auto-detects target interfaces from `/etc/perfSONAR-multi-nic-config.conf` or the routing table
-    4. Writes `/etc/flowd-go/conf.yaml` with the `perfsonar` plugin, `marker` backend, and (if supported) the `fireflyp` plugin
+    4. Writes `/etc/flowd-go/conf.yaml` with the `perfsonar` plugin, `firefly` backend, and `marker` backend
     5. Enables and starts the `flowd-go` systemd service
 
-### ESnet Stardust Integration (fireflyp plugin)
+### ESnet Stardust Integration (firefly backend)
 
-The `fireflyp` plugin is an optional flowd-go component that **forwards IPv6 firefly datagrams** to a
-central SciTags collector. The default target is `global.scitags.org:10514`, the authoritative global
-aggregation service operated by the SciTags project. Flows forwarded there become visible in
-[ESnet Stardust](https://stardust.es.net/), ESnet's network-level flow visualization platform, allowing
+The `firefly` backend is a flowd-go component that **emits SciTags firefly datagrams at the start and end of each flow**, sending them to a central SciTags collector. The default target is `global.scitags.org:10514`, the authoritative global aggregation service operated by the SciTags project. Fireflies sent there become visible in
+[ESnet Stardust](https://dashboard.stardust.es.net/d/b8dddac0-5b24-4739-9c8d-e88a05c1344f/scientific-network-tags3a-rande-dashboard?orgId=2&from=now-12h&to=now&timezone=browser), ESnet's network-level flow visualization platform, allowing
 collaborators and network operators to attribute traffic to specific experiments.
 
-!!! warning "fireflyp requires flowd-go ≥ 2.5.0"
+!!! note "flowd-go 2.5.0 or later required"
 
-    The `fireflyp` plugin is only available in flowd-go builds that include
-    [scitags/flowd-go#49](https://github.com/scitags/flowd-go/issues/49).
-    The current RPM release (2.4.2) does **not** include this plugin.
+    The `firefly` backend requires flowd-go >= 2.5.0. This version is the current standard
+    and is installed by default by all perfSONAR installation methods.
 
-    - If you are running **2.4.2**, pass `--no-firefly-receiver` to the install script to disable
-      the fireflyp stanza (eBPF flow marking still works; datagrams just are not forwarded upstream).
-    - Once a ≥ 2.5.0 RPM is published, re-run the install script or manually add the `fireflyp`
-      stanza to `/etc/flowd-go/conf.yaml`.
-
-**Install with fireflyp enabled** (flowd-go ≥ 2.5.0):
+**Standard installation with firefly backend enabled**:
 
 ```bash
 /opt/perfsonar-toolkit/tools_scripts/perfSONAR-install-flowd-go.sh \
-    --experiment-id 1 --firefly-receiver global.scitags.org --yes
+    --experiment-id 1 --yes
 ```
 
-**Install without fireflyp** (current 2.4.2 RPM):
+**Install with custom firefly collector address**:
 
 ```bash
 /opt/perfsonar-toolkit/tools_scripts/perfSONAR-install-flowd-go.sh \
-    --experiment-id 1 --no-firefly-receiver --yes
+    --experiment-id 1 --firefly-receiver collector.example.org --yes
 ```
 
 ??? info "Flags reference"
@@ -1633,7 +1625,6 @@ collaborators and network operators to attribute traffic to specific experiments
     | `--list-experiments` | Show experiment IDs and exit |
     | `--firefly-receiver HOST` | Firefly collector address (default: `global.scitags.org`) |
     | `--firefly-receiver-port N` | UDP port on the collector (default: 10514) |
-    | `--no-firefly-receiver` | Disable fireflyp plugin (eBPF marking only; use with flowd-go 2.4.x) |
     | `--yes` | Skip interactive prompts |
     | `--dry-run` | Preview without changes |
     | `--uninstall` | Remove flowd-go and configuration |