Skip to content

Commit b4c592d

Browse files
aireillyaireilly
authored
Merge pull request #74716 from aireilly/telcodocs-1676-tbc-dual-nic-ha
TELCODOCS-1676 - Dual NIC highly available PTP T-BC doc updates
2 parents 05c001e + 786a5bc commit b4c592d

12 files changed

+319
-17
lines changed

‎modules/cnf-getting-the-dpll-firmware-version-for-intel-800-series-nics.adoc

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * networking/ptp/configuring-ptp.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="cnf-getting-the-dpll-firmware-version-for-intel-800-series-nics_{context}"]
7+
= Getting the DPLL firmware version for the CGU in an Intel 800 series NIC
8+
9+
You can get the digital phase-locked loop (DPLL) firmware version for the Clock Generation Unit (CGU) in an Intel 800 series NIC by opening a debug shell to the cluster node and querying the NIC hardware.
10+
11+
.Prerequisites
12+
13+
* You have installed the OpenShift CLI (`oc`).
14+
15+
* You have logged in as a user with `cluster-admin` privileges.
16+
17+
* You have installed an Intel 800 series NIC in the cluster host.
18+
19+
* You have installed the PTP Operator on a bare-metal cluster with hosts that support PTP.
20+
21+
.Procedure
22+
23+
. Start a debug pod by running the following command:
24+
+
25+
[source,terminal]
26+
----
27+
$ oc debug node/<node_name>
28+
----
29+
+
30+
where:
31+
+
32+
--
33+
<node_name>:: Is the node where you have installed the Intel 800 series NIC.
34+
--
35+
36+
. Check the CGU firmware version in the NIC by using the `devlink` tool and the bus and device name where the NIC is installed.
37+
For example, run the following command:
38+
+
39+
[source,terminal]
40+
----
41+
sh-4.4# devlink dev info <bus_name>/<device_name> | grep cgu
42+
----
43+
+
44+
where:
45+
+
46+
--
47+
<bus_name> :: Is the bus where the NIC is installed. For example, `pci`.
48+
<device_name> :: Is the NIC device name. For example, `0000:51:00.0`.
49+
--
50+
+
51+
.Example output
52+
[source,terminal]
53+
----
54+
cgu.id 36 <1>
55+
fw.cgu 8032.16973825.6021 <2>
56+
----
57+
<1> CGU hardware revision number
58+
<2> The DPLL firmware version running in the CGU, where the DPLL firmware version is `6201`, and the DPLL model is `8032`.
59+
The string `16973825` is a shorthand representation of the binary version of the DPLL firmware version (`1.3.0.1`).
60+
61+
+
62+
[NOTE]
63+
====
64+
The firmware version has a leading nibble and 3 octets for each part of the version number.
65+
The number `16973825` in binary is `0001 0000 0011 0000 0000 0000 0001`.
66+
Use the binary value to decode the firmware version.
67+
For example:
68+
69+
.DPLL firmware version
70+
[cols="1,2", width="90%", options="header"]
71+
|====
72+
|Binary part
73+
|Decimal value
74+
75+
|`0001`
76+
|1
77+
78+
|`0000 0011`
79+
|3
80+
81+
|`0000 0000`
82+
|0
83+
84+
|`0000 0001`
85+
|1
86+
|====
87+
====

‎modules/nw-ptp-configuring-linuxptp-services-as-grandmaster-clock-dual-nic.adoc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,12 @@
88

99
You can configure the `linuxptp` services (`ptp4l`, `phc2sys`, `ts2phc`) as grandmaster clock (T-GM) for dual E810 Westport Channel NICs by creating a `PtpConfig` custom resource (CR) that configures the host NICs.
1010

11-
For distributed RAN (D-RAN) use cases, you can configure PTP for dual NICs as follows:
11+
For distributed RAN (D-RAN) use cases, you can configure PTP for dual-NICs as follows:
1212

1313
* NIC one is synced to the global navigation satellite system (GNSS) time source.
1414
* NIC two is synced to the 1PPS timing output provided by NIC one. This configuration is provided by the PTP hardware plugin in the `PtpConfig` CR.
1515
16-
The dual NIC PTP T-GM configuration uses a single instance of `ptp4l` and one `ts2phc` process reporting two `ts2phc` instances, one for each NIC.
16+
The dual-NIC PTP T-GM configuration uses a single instance of `ptp4l` and one `ts2phc` process reporting two `ts2phc` instances, one for each NIC.
1717
The host system clock is synchronized from the NIC that is connected to the GNSS time source.
1818

1919
[NOTE]

‎modules/nw-ptp-dual-wpc-hardware-config-reference.adoc

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,13 @@
88

99
Use this information to understand how to use the link:https://github.com/openshift/linuxptp-daemon/blob/release-4.14/addons/intel/e810.go[Intel E810-XXVDA4T hardware plugin] to configure a pair of E810 network interfaces as PTP grandmaster clock (T-GM).
1010

11-
Before you configure the dual NIC cluster host, you must connect the two NICs with an SMA1 cable using the 1PPS faceplace connections.
11+
Before you configure the dual-NIC cluster host, you must connect the two NICs with an SMA1 cable using the 1PPS faceplace connections.
1212

13-
When you configure a dual NIC T-GM, you need to compensate for the 1PPS signal delay that occurs when you connect the NICs using the SMA1 connection ports.
13+
When you configure a dual-NIC T-GM, you need to compensate for the 1PPS signal delay that occurs when you connect the NICs using the SMA1 connection ports.
1414
Various factors such as cable length, ambient temperature, and component and manufacturing tolerances can affect the signal delay.
1515
To compensate for the delay, you must calculate the specific value that you use to offset the signal delay.
1616

17-
.E810 dual NIC T-GM PtpConfig CR reference
17+
.E810 dual-NIC T-GM PtpConfig CR reference
1818
[cols="1,2" width="90%", options="header"]
1919
|====
2020
|PtpConfig field

‎modules/nw-ptp-installing-operator-cli.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
// Module included in the following assemblies:
22
//
3-
// * networking/ptp/about-ptp.adoc
3+
// * networking/ptp/configuring-ptp.adoc
44

55
:_mod-docs-content-type: PROCEDURE
66
[id="install-ptp-operator-cli_{context}"]

‎modules/nw-ptp-installing-operator-web-console.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
// Module included in the following assemblies:
22
//
3-
// * networking/ptp/about-ptp.adoc
3+
// * networking/ptp/configuring-ptp.adoc
44

55
:_mod-docs-content-type: PROCEDURE
66
[id="install-ptp-operator-web-console_{context}"]

‎modules/nw-ptp-operator-metrics-reference.adoc

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,4 +99,10 @@ Possible values are 0 (`UNAVAILABLE`) and 1 (`AVAILABLE`).
9999
GNSS provides satellite-based positioning, navigation, and timing services globally.
100100
Possible values are 0 (`NOFIX`), 1 (`DEAD RECKONING ONLY`), 2 (`2D-FIX`), 3 (`3D-FIX`), 4 (`GPS+DEAD RECKONING FIX`), 5, (`TIME ONLY FIX`).
101101
|`{from="gnss",iface="ens2fx",node="compute-1.example.com",process="gnss"} 3`
102+
103+
|`openshift_ptp_ha_profile_status`
104+
|Returns the current status of the highly available system clock when there are multiple time sources on different NICs.
105+
Possible values are 0 (`INACTIVE`), and 1 (`ACTIVE`).
106+
|`{node="node1",process="phc2sys",profile="profile1"} 1`
107+
`{node="node1",process="phc2sys",profile="profile2"} 0`
102108
|====

‎modules/ptp-configuring-linuxptp-services-as-boundary-clock-dual-nic.adoc

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,12 @@
44

55
:_mod-docs-content-type: PROCEDURE
66
[id="ptp-configuring-linuxptp-services-as-bc-for-dual-nic_{context}"]
7-
= Configuring linuxptp services as boundary clocks for dual NIC hardware
7+
= Configuring linuxptp services as boundary clocks for dual-NIC hardware
88

9-
:FeatureName: Precision Time Protocol (PTP) hardware with dual NIC configured as boundary clocks
9+
:FeatureName: Precision Time Protocol (PTP) hardware with dual-NIC configured as boundary clocks
1010
include::snippets/technology-preview.adoc[leveloffset=+1]
1111

12-
You can configure the `linuxptp` services (`ptp4l`, `phc2sys`) as boundary clocks for dual NIC hardware by creating a `PtpConfig` custom resource (CR) object for each NIC.
12+
You can configure the `linuxptp` services (`ptp4l`, `phc2sys`) as boundary clocks for dual-NIC hardware by creating a `PtpConfig` custom resource (CR) object for each NIC.
1313

1414
Dual NIC hardware allows you to connect each NIC to the same upstream leader clock with separate `ptp4l` instances for each NIC feeding the downstream clocks.
1515

@@ -76,7 +76,7 @@ spec:
7676
You must completely remove the `phc2sysOpts` field from the second `PtpConfig` CR to disable the `phc2sys` service on the second NIC.
7777
====
7878

79-
. Create the dual NIC `PtpConfig` CRs by running the following commands:
79+
. Create the dual-NIC `PtpConfig` CRs by running the following commands:
8080

8181
.. Create the CR that configures PTP for the first NIC:
8282
+
@@ -94,7 +94,7 @@ $ oc create -f boundary-clock-ptp-config-nic2.yaml
9494

9595
.Verification
9696

97-
* Check that the PTP Operator has applied the `PtpConfig` CRs for both NICs. Examine the logs for the `linuxptp` daemon corresponding to the node that has the dual NIC hardware installed. For example, run the following command:
97+
* Check that the PTP Operator has applied the `PtpConfig` CRs for both NICs. Examine the logs for the `linuxptp` daemon corresponding to the node that has the dual-NIC hardware installed. For example, run the following command:
9898
+
9999
[source,terminal]
100100
----

‎modules/ptp-configuring-linuxptp-services-as-ha-bc-for-dual-nic.adoc

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * networking/ptp/configuring-ptp.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="ptp-configuring-linuxptp-services-as-ha-bc-for-dual-nic_{context}"]
7+
= Configuring linuxptp as a highly available system clock for dual-NIC Intel E810 PTP boundary clocks
8+
9+
You can configure the `linuxptp` services `ptp4l` and `phc2sys` as a highly available (HA) system clock for dual PTP boundary clocks (T-BC).
10+
11+
The highly available system clock uses multiple time sources from dual-NIC Intel E810 Salem channel hardware configured as two boundary clocks.
12+
Two boundary clocks instances participate in the HA setup, each with its own configuration profile.
13+
You connect each NIC to the same upstream leader clock with separate `ptp4l` instances for each NIC feeding the downstream clocks.
14+
15+
Create two `PtpConfig` custom resource (CR) objects that configure the NICs as T-BC and a third `PtpConfig` CR that configures high availability between the two NICs.
16+
17+
[IMPORTANT]
18+
====
19+
When you create the `PtpConfig` CRs, ensure that the `phc2sysOpts` field is an empty string to prevent setting up the `phc2sys` processes on these two profiles.
20+
====
21+
22+
The third `PtpConfig` CR configures a highly available system clock service.
23+
The CR sets the `ptp4lOpts` field to an empty string to prevent the `ptp4l` process from running.
24+
The CR adds profiles for the `ptp4l` configurations under the `spec.profile.ptpSettings.haProfiles` key and passes the kernel socket path of those profiles to the `phc2sys` service.
25+
When a `ptp4l` failure occurs, the `phc2sys` service switches to the backup `ptp4l` configuration.
26+
When the primary profile becomes active again, the `phc2sys` service reverts to the original state.
27+
28+
.Prerequisites
29+
30+
* Install the {oc-first}.
31+
32+
* Log in as a user with `cluster-admin` privileges.
33+
34+
* Install the PTP Operator.
35+
36+
* Configure a cluster node with Intel E810 Salem channel dual-NIC.
37+
38+
.Procedure
39+
40+
. Create two separate `PtpConfig` CRs, one for each NIC, using the CRs in "Configuring linuxptp services as boundary clocks for dual-NIC hardware" as a reference for each CR.
41+
42+
.. Create the `ha-ptp-config-nic1.yaml` file, specifying an empty string for the `phc2sysOpts` field.
43+
For example:
44+
+
45+
[source,yaml]
46+
----
47+
apiVersion: ptp.openshift.io/v1
48+
kind: PtpConfig
49+
metadata:
50+
name: ha-ptp-config-nic1
51+
namespace: openshift-ptp
52+
spec:
53+
profile:
54+
- name: "ha-ptp-config-profile1"
55+
ptp4lOpts: "-2 --summary_interval -4"
56+
ptp4lConf: | <1>
57+
[ens5f1]
58+
masterOnly 1
59+
[ens5f0]
60+
masterOnly 0
61+
#...
62+
phc2sysOpts: "" <2>
63+
----
64+
<1> Specify the required interfaces to start `ptp4l` as a boundary clock. For example, `ens5f0` synchronizes from a grandmaster clock and `ens5f1` synchronizes connected devices.
65+
<2> Set `phc2sysOpts` with an empty string.
66+
These values are populated from the `spec.profile.ptpSettings.haProfiles` field of the `PtpConfig` CR that configures high availability.
67+
68+
.. Apply the `PtpConfig` CR for NIC 1 by running the following command:
69+
+
70+
[source,terminal]
71+
----
72+
$ oc create -f ha-ptp-config-nic1.yaml
73+
----
74+
75+
.. Create the `ha-ptp-config-nic2.yaml` file, specifying an empty string for the `phc2sysOpts` field.
76+
For example:
77+
+
78+
[source,yaml]
79+
----
80+
apiVersion: ptp.openshift.io/v1
81+
kind: PtpConfig
82+
metadata:
83+
name: ha-ptp-config-nic2
84+
namespace: openshift-ptp
85+
spec:
86+
profile:
87+
- name: "ha-ptp-config-profile2"
88+
ptp4lOpts: "-2 --summary_interval -4"
89+
ptp4lConf: |
90+
[ens7f1]
91+
masterOnly 1
92+
[ens7f0]
93+
masterOnly 0
94+
#...
95+
phc2sysOpts: ""
96+
----
97+
98+
.. Apply the `PtpConfig` CR for NIC 2 by running the following command:
99+
+
100+
[source,terminal]
101+
----
102+
$ oc create -f ha-ptp-config-nic2.yaml
103+
----
104+
105+
. Create the `PtpConfig` CR that configures the HA system clock.
106+
For example:
107+
108+
.. Create the `ptp-config-for-ha.yaml` file:
109+
+
110+
[source,yaml]
111+
----
112+
include::snippets/ztp_PtpConfigForHA.yaml[]
113+
----
114+
+
115+
[IMPORTANT]
116+
====
117+
Do not apply the high availability `PtpConfig` CR before the `PtpConfig` CRs that configure the individual NICs.
118+
====
119+
120+
.. Apply the HA `PtpConfig` CR by running the following command:
121+
+
122+
[source,terminal]
123+
----
124+
$ oc create -f ptp-config-for-ha.yaml
125+
----
126+
127+
.Verification
128+
129+
* Verify that the PTP Operator has applied the `PtpConfig` CRs correctly.
130+
Perform the following steps:
131+
132+
.. Get the list of pods in the `openshift-ptp` namespace by running the following command:
133+
+
134+
[source,terminal]
135+
----
136+
$ oc get pods -n openshift-ptp -o wide
137+
----
138+
+
139+
.Example output
140+
[source,terminal]
141+
----
142+
NAME READY STATUS RESTARTS AGE IP NODE
143+
linuxptp-daemon-4xkrb 1/1 Running 0 43m 10.1.196.24 compute-0.example.com
144+
ptp-operator-657bbq64c8-2f8sj 1/1 Running 0 43m 10.129.0.61 control-plane-1.example.com
145+
----
146+
+
147+
[NOTE]
148+
====
149+
There should be only one `linuxptp-daemon` pod.
150+
====
151+
152+
.. Check that the profile is correct by running the following command.
153+
Examine the logs of the `linuxptp` daemon that corresponds to the node you specified in the `PtpConfig` profile.
154+
+
155+
[source,terminal]
156+
----
157+
$ oc logs linuxptp-daemon-4xkrb -n openshift-ptp -c linuxptp-daemon-container
158+
----
159+
+
160+
.Example output
161+
[source,terminal]
162+
----
163+
I1115 09:41:17.117596 4143292 daemon.go:107] in applyNodePTPProfile
164+
I1115 09:41:17.117604 4143292 daemon.go:109] updating NodePTPProfile to:
165+
I1115 09:41:17.117607 4143292 daemon.go:110] ------------------------------------
166+
I1115 09:41:17.117612 4143292 daemon.go:102] Profile Name: ha-ptp-config-profile1
167+
I1115 09:41:17.117616 4143292 daemon.go:102] Interface:
168+
I1115 09:41:17.117620 4143292 daemon.go:102] Ptp4lOpts: -2
169+
I1115 09:41:17.117623 4143292 daemon.go:102] Phc2sysOpts: -a -r -n 24
170+
I1115 09:41:17.117626 4143292 daemon.go:116] ------------------------------------
171+
----

‎modules/ptp-dual-nics.adoc

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,12 @@
44

55
:_mod-docs-content-type: CONCEPT
66
[id="ptp-dual-nics_{context}"]
7-
= Using dual Intel E810 NIC hardware with PTP
7+
= Using dual-NIC Intel E810 hardware with PTP
88

9-
{product-title} supports single and dual NIC Intel E810 hardware for precision PTP timing in grandmaster clocks (T-GM) and boundary clocks (T-BC).
9+
{product-title} supports single and dual-NIC Intel E810 hardware for precision PTP timing in grandmaster clocks (T-GM) and boundary clocks (T-BC).
1010

1111
Dual NIC grandmaster clock::
12-
You can use a cluster host that has dual NIC hardware as PTP grandmaster clock.
12+
You can use a cluster host that has dual-NIC hardware as PTP grandmaster clock.
1313
One NIC receives timing information from the global navigation satellite system (GNSS).
1414
The second NIC receives the timing information from the first using the SMA1 Tx/Rx connections on the E810 NIC faceplate.
1515
The system clock on the cluster host is synchronized from the NIC that is connected to the GNSS satellite.
@@ -22,10 +22,22 @@ image::561_OpenShift_Using_PTP_network_0124.png[Dual NIC PTP grandmaster clock c
2222
+
2323
[NOTE]
2424
====
25-
In a dual NIC T-GM configuration, a single `ts2phc` process reports as two `ts2phc` instances in the system.
25+
In a dual-NIC T-GM configuration, a single `ts2phc` process reports as two `ts2phc` instances in the system.
2626
====
2727

2828
Dual NIC boundary clock::
2929
For 5G telco networks that deliver mid-band spectrum coverage, each virtual distributed unit (vDU) requires connections to 6 radio units (RUs). To make these connections, each vDU host requires 2 NICs configured as boundary clocks.
3030
+
3131
Dual NIC hardware allows you to connect each NIC to the same upstream leader clock with separate `ptp4l` instances for each NIC feeding the downstream clocks.
32+
33+
Highly available system clock with dual-NIC boundary clocks::
34+
You can configure Intel E810-XXVDA4 Salem channel dual-NIC hardware as dual PTP boundary clocks that provide timing for a highly available system clock.
35+
This is useful when you have multiple time sources on different NICs.
36+
High availability ensures that the node does not lose timing synchronisation if one of the two timing sources is lost or disconnected.
37+
+
38+
Each NIC is connected to the same upstream leader clock.
39+
Highly available boundary clocks use multiple PTP domains to synchronize with the target system clock.
40+
When a T-BC is highly available, the host system clock can maintain the correct offset even if one or more `ptp4l` instances syncing the NIC PHC clock fails.
41+
If any single SFP port or cable failure occurs, the boundary clock stays in sync with the leader clock.
42+
+
43+
Boundary clock leader source selection is done using the A-BMCA algorithm. For more information, see link:https://www.itu.int/rec/T-REC-G.8275.1/en[ITU-T recommendation G.8275.1].

0 commit comments

Comments
 (0)