adding HTTP transport updates

add pv and pvc for ZTP events

Jack's http transport for events comments

Adding HTTP migration procedure

Adding Jack's ZTP events storage

adding some extra details for PTP events transport

jack's comments

for jack: added PVC docs updates to ZTP

Aneesh: Clarify transportHost is not required to be set for HTTP transport

event service and pv updates for jack

jacks comments

PV updates + splitting into 2 topics for HTTP/AMQP transport

jack's latest comments on PTP via ZTP procedures

updates for jack and tao

Author: aireilly

images/319_OpenShift_redfish_bare-metal_OCP_nodes_0323.png

@@ -6,15 +6,15 @@
 [id="cnf-about-ptp-and-clock-synchronization_{context}"]
 = About PTP and clock synchronization error events
 
-Cloud native applications such as virtual RAN require access to notifications about hardware timing events that are critical to the functioning of the overall network. Fast event notifications are early warning signals about impending and real-time Precision Time Protocol (PTP) clock synchronization events. PTP clock synchronization errors can negatively affect the performance and reliability of your low latency application, for example, a vRAN application running in a distributed unit (DU).
-
 Loss of PTP synchronization is a critical error for a RAN network. If synchronization is lost on a node, the radio might be shut down and the network Over the Air (OTA) traffic might be shifted to another node in the wireless network. Fast event notifications mitigate against workload errors by allowing cluster nodes to communicate PTP clock sync status to the vRAN application running in the DU.
 
-Event notifications are available to RAN applications running on the same DU node. A publish/subscribe REST API passes events notifications to the messaging bus. Publish/subscribe messaging, or pub/sub messaging, is an asynchronous service to service communication architecture where any message published to a topic is immediately received by all the subscribers to the topic.
+Event notifications are available to vRAN applications running on the same DU node. A publish-subscribe REST API passes events notifications to the messaging bus. Publish-subscribe messaging, or pub-sub messaging, is an asynchronous service-to-service communication architecture where any message published to a topic is immediately received by all of the subscribers to the topic.
 
-Fast event notifications are generated by the PTP Operator in {product-title} for every PTP-capable network interface. The events are made available using a `cloud-event-proxy` sidecar container over an Advanced Message Queuing Protocol (AMQP) message bus. The AMQP message bus is provided by the AMQ Interconnect Operator.
+The PTP Operator generates fast event notifications for every PTP-capable network interface. You can access the events by using a `cloud-event-proxy` sidecar container over an HTTP or Advanced Message Queuing Protocol (AMQP) message bus.
 
 [NOTE]
 ====
 PTP fast event notifications are available for network interfaces configured to use PTP ordinary clocks or PTP boundary clocks.
 ====
+
+include::snippets/ptp-amq-interconnect-eol.adoc[]

modules/cnf-about-ptp-and-clock-synchronization.adoc

@@ -8,40 +8,45 @@
 
 Use the Precision Time Protocol (PTP) fast event notifications framework to subscribe cluster applications to PTP events that the bare-metal cluster node generates.
 
+[NOTE]
+====
+The fast events notifications framework uses a REST API for communication. The REST API is based on the _O-RAN O-Cloud Notification API Specification for Event Consumers 3.0_ that is available from link:https://orandownloadsweb.azurewebsites.net/specifications[O-RAN ALLIANCE Specifications].
+====
+
 The framework consists of a publisher, subscriber, and an AMQ or HTTP messaging protocol to handle communications between the publisher and subscriber applications.
 Applications run the `cloud-event-proxy` container in a sidecar pattern to subscribe to PTP events.
 The `cloud-event-proxy` sidecar container can access the same resources as the primary application container without using any of the resources of the primary application and with no significant latency.
 
-[NOTE]
-====
-The fast events notifications framework uses a REST API for communication. The REST API is based on the _O-Cloud Notification API Specification for Event Consumers_ available from link:https://orandownloadsweb.azurewebsites.net/specifications[O-RAN ALLIANCE Specifications].
-====
+include::snippets/ptp-amq-interconnect-eol.adoc[]
 
 .Overview of PTP fast events
 image::319_OpenShift_PTP_bare-metal_OCP_nodes_0323_4.13.png[Overview of PTP fast events]
 
---
-image:darkcircle-1.png[20,20]
+image:darkcircle-1.png[20,20] Event is generated on the cluster host::
 `linuxptp-daemon` in the PTP Operator-managed pod runs as a Kubernetes `DaemonSet` and manages the various `linuxptp` processes (`ptp4l`, `phc2sys`, and optionally for grandmaster clocks, `ts2phc`).
 The `linuxptp-daemon` passes the event to the UNIX domain socket.
 
-image:darkcircle-2.png[20,20]
+image:darkcircle-2.png[20,20] Event is passed to the cloud-event-proxy sidecar::
 The PTP plugin reads the event from the UNIX domain socket and passes it to the `cloud-event-proxy` sidecar in the PTP Operator-managed pod.
 `cloud-event-proxy` delivers the event from the Kubernetes infrastructure to Cloud-Native Network Functions (CNFs) with low latency.
 
-image:darkcircle-3.png[20,20]
-The `cloud-event-proxy` sidecar in the PTP Operator-managed pod persists the event and publishes the cloud native event by using a REST API.
+image:darkcircle-3.png[20,20] Event is persisted::
+The `cloud-event-proxy` sidecar in the PTP Operator-managed pod processes the event and publishes the cloud-native event by using a REST API.
++
+[NOTE]
+====
+When you use HTTP transport for events, you must persist the events subscription in the PTP Operator-managed pod by using a Persistent Volume (PV) resource or similar persistent storage mechanism.
+====
 
-image:darkcircle-4.png[20,20]
-The message is transported to the `cloud-event-proxy` sidecar in the application pod over HTTP or AMQP 1.0 QPID.
+image:darkcircle-4.png[20,20] Message is transported::
+The message transporter transports the event to the `cloud-event-proxy` sidecar in the application pod over HTTP or AMQP 1.0 QPID.
 
-image:darkcircle-5.png[20,20]
+image:darkcircle-5.png[20,20] Event is available from the REST API::
 The `cloud-event-proxy` sidecar in the Application pod processes the event and makes it available by using the REST API.
 
-image:darkcircle-6.png[20,20]
+image:darkcircle-6.png[20,20] Consumer application requests a subscription and receives the subscribed event::
 The consumer application sends an API request to the `cloud-event-proxy` sidecar in the application pod to create a PTP events subscription.
 The `cloud-event-proxy` sidecar creates an AMQ or HTTP messaging listener protocol for the resource specified in the subscription.
 
 The `cloud-event-proxy` sidecar in the application pod receives the event from the PTP Operator-managed pod, unwraps the cloud events object to retrieve the data, and posts the event to the consumer application.
 The consumer application listens to the address specified in the resource qualifier and receives and processes the PTP event.
---

modules/cnf-about-ptp-fast-event-notifications-framework.adoc

@@ -10,9 +10,20 @@ To start using PTP fast event notifications for a network interface in your clus
 
 .Prerequisites
 
-* Install the {product-title} CLI (`oc`).
-* Log in as a user with `cluster-admin` privileges.
-* Install the PTP Operator and AMQ Interconnect Operator.
+* You have installed the {product-title} CLI (`oc`).
+
+* You have logged in as a user with `cluster-admin` privileges.
+
+* You have installed the PTP Operator.
+
+* When you use HTTP events transport, configure dynamic volume provisioning in the cluster or manually create `StorageClass`, `LocalVolume`, and `PersistentVolume` resources to persist the events subscription.
++
+[NOTE]
+====
+When you enable dynamic volume provisioning in the cluster, a `PersistentVolume` resource is automatically created for the `PersistentVolumeClaim` that the PTP Operator deploys.
+
+For more information about manually creating persistent storage in the cluster, see "Persistent storage using local volumes".
+====
 
 .Procedure
 
@@ -32,10 +43,20 @@ spec:
     node-role.kubernetes.io/worker: ""
   ptpEventConfig:
     enableEventPublisher: true <1>
-    transportHost: amqp://<instance_name>.<namespace>.svc.cluster.local <2>
+    storageType: "example-storage-class" <2>
 ----
 <1> Set `enableEventPublisher` to `true` to enable PTP fast event notifications.
-<2> Set `transportHost` to the AMQ router that you configured where `<instance_name>` and `<namespace>` correspond to the AMQ Interconnect router instance name and namespace, for example, `amqp://amq-interconnect.amq-interconnect.svc.cluster.local`
+<2> Use the value you set for `storageType` to populate the `StorageClassName` field for the `PersistentVolumeClaim` (`PVC`) resource that the PTP Operator automatically deploys.
+The `PVC` resource is used to persist consumer event subscriptions.
++
+[NOTE]
+====
+In {product-title} 4.13 or later, you do not need to set the `spec.ptpEventConfig.transportHost` field in the `PtpOperatorConfig` resource when you use HTTP transport for PTP events.
+Set `transportHost` only when you use AMQP transport for PTP events.
+
+The value that you set for `.spec.storageType` in the `PtpOperatorConfig` CR must match the `storageClassName` that is set in the `PersistentVolume` CR.
+If `storageType` is not set and the `transportHost` uses HTTP, the PTP daemons are not deployed.
+====
 
 .. Update the `PtpOperatorConfig` CR:
 +
@@ -44,7 +65,8 @@ spec:
 $ oc apply -f ptp-operatorconfig.yaml
 ----
 
-. Create a `PtpConfig` custom resource (CR) for the PTP enabled interface, and set the required values for `ptpClockThreshold` and `ptp4lOpts`. The following YAML illustrates the required values that you must set in the `PtpConfig` CR:
+. Create a `PtpConfig` custom resource (CR) for the PTP enabled interface, and set the required values for `ptpClockThreshold` and `ptp4lOpts`.
+The following YAML illustrates the required values that you must set in the `PtpConfig` CR:
 +
 [source,yaml]
 ----
@@ -62,5 +84,5 @@ spec:
 ----
 <1> Append `--summary_interval -4` to use PTP fast events.
 <2> Required `phc2sysOpts` values. `-m` prints messages to `stdout`. The `linuxptp-daemon` `DaemonSet` parses the logs and generates Prometheus metrics.
-<3> Specify a string that contains the configuration to replace the default /etc/ptp4l.conf file. To use the default configuration, leave the field empty.
+<3> Specify a string that contains the configuration to replace the default `/etc/ptp4l.conf` file. To use the default configuration, leave the field empty.
 <4> Optional. If the `ptpClockThreshold` stanza is not present, default values are used for the `ptpClockThreshold` fields. The stanza shows default `ptpClockThreshold` values. The `ptpClockThreshold` values configure how long after the PTP master clock is disconnected before PTP events are triggered. `holdOverTimeout` is the time value in seconds before the PTP clock event state changes to `FREERUN` when the PTP master clock is disconnected. The `maxOffsetThreshold` and `minOffsetThreshold` settings configure offset values in nanoseconds that compare against the values for `CLOCK_REALTIME` (`phc2sys`) or master offset (`ptp4l`). When the `ptp4l` or `phc2sys` offset value is outside this range, the PTP clock state is set to `FREERUN`. When the offset value is within this range, the PTP clock state is set to `LOCKED`.

modules/cnf-configuring-the-ptp-fast-event-publisher.adoc

@@ -6,11 +6,15 @@
 [id="cnf-installing-amq-interconnect-messaging-bus_{context}"]
 = Installing the AMQ messaging bus
 
-To pass PTP fast event notifications between publisher and subscriber on a node, you must install and configure an AMQ messaging bus to run locally on the node. You do this by installing the AMQ Interconnect Operator for use in the cluster.
+To pass PTP fast event notifications between publisher and subscriber on a node, you can install and configure an AMQ messaging bus to run locally on the node.
+To use AMQ messaging, you must install the AMQ Interconnect Operator.
+
+include::snippets/ptp-amq-interconnect-eol.adoc[]
 
 .Prerequisites
 
 * Install the {product-title} CLI (`oc`).
+
 * Log in as a user with `cluster-admin` privileges.
 
 .Procedure
@@ -48,6 +52,3 @@ NAME                     READY   STATUS    RESTARTS       AGE
 linuxptp-daemon-2t78p    3/3     Running   0              12h
 linuxptp-daemon-k8n88    3/3     Running   0              12h
 ----
-
-
-

modules/cnf-installing-amq-interconnect-messaging-bus.adoc

@@ -0,0 +1,75 @@
+// Module included in the following assemblies:
+//
+// * monitoring/using-rfhe.adoc
+// * networking/using-ptp.adoc
+
+:_content-type: PROCEDURE
+[id="cnf-migrating-from-amqp-to-http-transport_{context}"]
+= Migrating consumer applications to use HTTP transport for PTP or bare-metal events
+
+If you have previously deployed PTP or bare-metal events consumer applications, you need to update the applications to use HTTP message transport.
+
+.Prerequisites
+
+* You have installed the OpenShift CLI (`oc`).
+
+* You have logged in as a user with `cluster-admin` privileges.
+
+* You have updated the PTP Operator or {redfish-operator} to version 4.13+ which uses HTTP transport by default.
+
+* Configure dynamic volume provisioning in the cluster or manually create `StorageClass`, `LocalVolume`, and `PersistentVolume` resources to persist the events subscription.
++
+[NOTE]
+====
+When dynamic volume provisioning is enabled, a `PersistentVolume` resource is automatically created for the `PersistentVolumeClaim` that the PTP Operator or {redfish-operator} deploys.
+====
+
+.Procedure
+
+. Update your events consumer application to use HTTP transport.
+Set the `http-event-publishers` variable for the cloud event sidecar deployment.
++
+For example, in a cluster with PTP events configured, the following YAML snippet illustrates a cloud event sidecar deployment:
++
+[source,yaml]
+----
+containers:
+  - name: cloud-event-sidecar
+    image: cloud-event-sidecar
+    args:
+      - "--metrics-addr=127.0.0.1:9091"
+      - "--store-path=/store"
+      - "--transport-host=consumer-events-subscription-service.cloud-events.svc.cluster.local:9043"
+      - "--http-event-publishers=ptp-event-publisher-service-NODE_NAME.openshift-ptp.svc.cluster.local:9043" <1>
+      - "--api-port=8089"
+----
+<1> The PTP Operator automatically resolves `NODE_NAME` to the host that is generating the PTP events.
+For example, `compute-1.example.com`.
++
+In a cluster with bare-metal events configured, set the `http-event-publishers` field to `hw-event-publisher-service.openshift-bare-metal-events.svc.cluster.local:9043` in the cloud event sidecar deployment CR.
+
+. Deploy the `consumer-events-subscription-service` service alongside the events consumer application.
+For example:
++
+[source,yaml]
+----
+apiVersion: v1
+kind: Service
+metadata:
+  annotations:
+    prometheus.io/scrape: "true"
+    service.alpha.openshift.io/serving-cert-secret-name: sidecar-consumer-secret
+  name: consumer-events-subscription-service
+  namespace: cloud-events
+  labels:
+    app: consumer-service
+spec:
+  ports:
+    - name: sub-port
+      port: 9043
+  selector:
+    app: consumer
+  clusterIP: None
+  sessionAffinity: None
+  type: ClusterIP
+----

modules/cnf-migrating-from-amqp-to-http-transport.adoc

@@ -0,0 +1,57 @@
+// Module included in the following assemblies:
+//
+// * networking/using-ptp.adoc
+
+:_content-type: PROCEDURE
+[id="cnf-monitoring-fast-events-metrics_{context}"]
+= Monitoring PTP fast event metrics
+
+You can monitor PTP fast events metrics from cluster nodes where the `linuxptp-daemon` is running.
+You can also monitor PTP fast event metrics in the {product-title} web console by using the pre-configured and self-updating Prometheus monitoring stack.
+
+.Prerequisites
+
+* Install the {product-title} CLI `oc`.
+
+* Log in as a user with `cluster-admin` privileges.
+
+* Install and configure the PTP Operator on a node with PTP-capable hardware.
+
+.Procedure
+
+. Check for exposed PTP metrics on any node where the `linuxptp-daemon` is running. For example, run the following command:
++
+[source,terminal]
+----
+$ curl http://<node_name>:9091/metrics
+----
++
+.Example output
+----
+# HELP openshift_ptp_clock_state 0 = FREERUN, 1 = LOCKED, 2 = HOLDOVER
+# TYPE openshift_ptp_clock_state gauge
+openshift_ptp_clock_state{iface="ens1fx",node="compute-1.example.com",process="ptp4l"} 1
+openshift_ptp_clock_state{iface="ens3fx",node="compute-1.example.com",process="ptp4l"} 1
+openshift_ptp_clock_state{iface="ens5fx",node="compute-1.example.com",process="ptp4l"} 1
+openshift_ptp_clock_state{iface="ens7fx",node="compute-1.example.com",process="ptp4l"} 1
+# HELP openshift_ptp_delay_ns
+# TYPE openshift_ptp_delay_ns gauge
+openshift_ptp_delay_ns{from="master",iface="ens1fx",node="compute-1.example.com",process="ptp4l"} 842
+openshift_ptp_delay_ns{from="master",iface="ens3fx",node="compute-1.example.com",process="ptp4l"} 480
+openshift_ptp_delay_ns{from="master",iface="ens5fx",node="compute-1.example.com",process="ptp4l"} 584
+openshift_ptp_delay_ns{from="master",iface="ens7fx",node="compute-1.example.com",process="ptp4l"} 482
+openshift_ptp_delay_ns{from="phc",iface="CLOCK_REALTIME",node="compute-1.example.com",process="phc2sys"} 547
+# HELP openshift_ptp_offset_ns
+# TYPE openshift_ptp_offset_ns gauge
+openshift_ptp_offset_ns{from="master",iface="ens1fx",node="compute-1.example.com",process="ptp4l"} -2
+openshift_ptp_offset_ns{from="master",iface="ens3fx",node="compute-1.example.com",process="ptp4l"} -44
+openshift_ptp_offset_ns{from="master",iface="ens5fx",node="compute-1.example.com",process="ptp4l"} -8
+openshift_ptp_offset_ns{from="master",iface="ens7fx",node="compute-1.example.com",process="ptp4l"} 3
+openshift_ptp_offset_ns{from="phc",iface="CLOCK_REALTIME",node="compute-1.example.com",process="phc2sys"} 12
+----
+
+. To view the PTP event in the {product-title} web console, copy the name of the PTP metric you want to query, for example, `openshift_ptp_offset_ns`.
+
+. In the {product-title} web console, click *Observe* -> *Metrics*.
+
+. Paste the PTP metric name into the *Expression* field, and click *Run queries*.