Docs: order documentation in MTL (#1024)

Docs: order documentation in MTL (and also paralely in BCS and MCM repos
(as MTL will be used as sub-module there)).

Style of chapters change,
comas, points, colons, numbering of sections,
style of source code blocks consistent,
source code with commend lines to be paste'able for user,
isolate some parts of the text to be reused also in the documents of
others repositories.
Sphinx added generating the documentaries.
Added ST41 to the architecture diagram.
Added X540-AT2/X550T NICs to the architecture diagram.

Author: skolelis

.github/workflows/github_pages_update.yml

@@ -0,0 +1,57 @@
+name: documentation-build-and-publish
+on:
+  workflow_call:
+  workflow_dispatch:
+  push:
+    branches: [ "main" ]
+
+env:
+  DEBIAN_FRONTEND: noninteractive
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
+  cancel-in-progress: true
+
+jobs:
+  publishGitHubPages:
+    name: Publish GitHub Pages
+    permissions:
+      contents: read
+      id-token: write
+      pages: write
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    runs-on: 'ubuntu-22.04'
+    timeout-minutes: 20
+    steps:
+      - name: Secure the runner
+        uses: step-security/harden-runner@17d0e2bd7d51742c71671bd19fa12bdc9d40a3d6 # v2.8.1
+        with:
+          egress-policy: audit
+
+      - name: Checkout
+        uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
+
+      - name: Prepare operating system for documentation build
+        run: |
+          sudo apt-get update -y && \
+          sudo apt-get install -y --no-install-recommends make python3 python3-pip python3-sphinx
+
+      - name: Prepare environment for documentation build
+        run: python3 -m pip install sphinx_book_theme myst_parser sphinxcontrib.mermaid sphinx-copybutton
+
+      - name: Build documentation
+        run:  make -C doc/sphinx html
+
+      - name: Upload GitHub Pages artifact
+        uses: actions/[email protected]
+        with:
+          path: ./doc/_build/html
+
+      - name: Publish to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v4

.gitignore

@@ -82,6 +82,9 @@ mtl_system_status_*
 *.pcm
 *.wav
 
+#Generated documentation
+doc/_build
+
 # Gpu direct files
 gpu_direct/tests/fff.h
 gpu_direct/subprojects/*

.markdown-lint.yml

@@ -26,7 +26,7 @@ MD013:
 MD026:
   punctuation: ".,;:!。，；:"  # List of not allowed
 MD029: false                  # Ordered list item prefix
-MD033: true                  # Allow inline HTML
+MD033: true                   # Allow inline HTML
 MD034: true                   # no-bare-urls
 MD036: false                  # Emphasis used instead of a heading
 

CHANGELOG.md

@@ -7,7 +7,7 @@
 * st2110/30: add force NUMA option support on session level, see ST30_TX_FLAG_FORCE_NUMA/ST30_RX_FLAG_FORCE_NUMA
 * ffmpeg: fix RX side dropping frames at the beginning of the session with st20/st22/st30.
 * st22: fix last frame dropping in TX. Ensure that last frame status changed to FREE.
-* dpdk: optimizing memory pool size.
+* DPDK: optimizing memory pool size.
 * manager: fix docker build.
 * ffmpeg: improve unicast initialization, reduce amount of dropping frames in the beginning of the session.
 * ixgbe: add driver support. Tested on 10-Gigabit X540-AT2 (1528) and Intel 10G X550T (1563).
@@ -25,7 +25,7 @@
 
 ## Changelog for 24.06
 
-* dpdk: upgrade dpdk version to 23.11.
+* DPDK: upgrade DPDK version to 23.11.
 * st22: add interlaced support.
 * log: add custom log printer, see mtl_set_log_printer.
 * rx/timing_parser: add timing_parser stat report for RX video, `--rx_timing_parser` in RxTxApp to enable.
@@ -48,11 +48,11 @@
 ## Changelog for 23.12
 
 * log: add log to file support, see mtl_openlog_stream.
-* dpdk: upgrade dpdk version to 23.07.
+* DPDK: upgrade DPDK version to 23.07.
 * virtio_user: add virtio_user support for exception path, deprecate kni.
 * st22p/tx: add external frame support, see ST22P_TX_FLAG_EXT_FRAME.
 * backend: add kernel socket based backend, see doc/kernel_socket.md.
-* dpdk pmd: add AF_PACKET PMD support, see doc/experimental/af_packet.md.
+* DPDK pmd: add AF_PACKET PMD support, see doc/experimental/af_packet.md.
 * st22p/rx: add external frame support, see ST22P_RX_FLAG_EXT_FRAME.
 * ptp: add user callback for ptp sync message. See ptp_sync_notify in struct mtl_init_params.
 * api: add arp timeout parameter support for st2110 unicast address. See arp_timeout_s in struct mtl_init_params.
@@ -82,14 +82,14 @@
 * tx/st30: separate build and pacing stage.
 * tx/pacing: add epoch drop/onward stat.
 * tx/st2110: add epoch information in stxx_tx_frame_meta for get_next_frame callback.
-* dpdk: upgrade dpdk version to latest 23.03
+* DPDK: upgrade DPDK version to latest 23.03
 * Windows: add MSYS2 build guide and CI
 * vm: add Windows guest OS support, see vm_WIN.md.
 * rx/video: add fpt(first packet time to epoch) in struct st20_rx_frame_meta.
 * st20p: add interlace format support, refer to tests/script/loop_json/st20p_2v_1080i50.json
 * CI: add ossf scorecard support.
 * gtest: support run with AWS ena driver.
-* dhcp: set proto to dhcp in interfaces segment of json to enable DHCP.
+* dhcp: set proto to dhcp in interfaces segment of JSON to enable DHCP.
 * rx/video: support out of sequence for both first and last(marker) packet.
 * udp: add gso(general segment offload) for tx
 * udp: add reuse port support.
@@ -102,7 +102,7 @@
 
 ## Changelog for 23.04
 
-* dpdk: upgrade dpdk version to latest 22.11 for both windows and linux.
+* DPDK: upgrade DPDK version to latest 22.11 for both Windows and Linux.
 * ptp: add pi controller and software frequency adjust to improve the accuracy to ~100ns.
 * mtl_init_params: add gateway and netmask support for wan.
 * udp: introduce a highly efficient udp stack support, see mudp_api.h
@@ -149,7 +149,7 @@
 ## Changelog for 22.09
 
 * License: update to BSD-3
-* dpdk: update DPDK to v22.07
+* DPDK: update DPDK to v22.07
 * ice: update driver to 1.9.11
 * vf: add SRIOV based virtual NIC support, see vf.md.
 * vm: add SRIOV+KVM based virtualization support, see vm.md.

README.md

@@ -17,7 +17,7 @@ The Media Transport Library solves the strict timing challenges of transporting
 
 If you find value in our project, please consider giving it a star. Your support helps us grow and reach more people in the open-source community. Every star counts and is greatly appreciated.
 
-### 1.1 Features
+### 1.1. Features
 
 * Supported data path backend: DPDK PMD, native kernel socket, and AF_XDP with eBPF filter.
 * The User-space LibOS UDP stack features a POSIX socket compatible API.
@@ -28,7 +28,7 @@ If you find value in our project, please consider giving it a star. Your support
 * FFMPEG plugin, OBS(Open Broadcaster Software) plugin, and Intel® Media SDK support.
 * In addition to the native C/C++ API, it also offers bindings for [Python](python/README.md) and [Rust](rust/README.md).
 
-#### 1.1.1 ST2110 features
+#### 1.1.1. ST2110 features
 
 * Narrow and wide pacing. Please see [compliance](doc/compliance.md) page for the ST2110 narrow report on our software solution.
 * ST2110-10, ST2110-20, ST2110-21, ST2110-30, ST2110-40, ST2022-7.
@@ -40,7 +40,7 @@ If you find value in our project, please consider giving it a star. Your support
 * ST2022-6 by RTP passthrough interface.
 * ST2110-20 RX timing compliance parser with hardware RX timestamp offload.
 
-### 1.2 Architecture
+### 1.2. Architecture
 
 The Media Transport Library leverages DPDK (Data Plane Development Kit) EAL (Environment Abstraction Layer including the memory and core management) to implement a highly efficient, real-time, and low-latency media transport solution. This software-based media transport stack enables deployment on edge and cloud environments using COTS hardware.
 
@@ -56,13 +56,13 @@ Additionally, the packet pacing module offers support for various pacing algorit
 
 MTL also incorporates SIMD (Single Instruction, Multiple Data) for CSC (Color Space Format Conversion) of the big-endian and little-endian, DMA (Direct Memory Access), and plugin interfaces, enabling the creation of a comprehensive video production ecosystem.
 
-For the detail design, please refer to [design guide](doc/design.md).
+For the detail design, please refer to [Design Guide](doc/design.md).
 
 <div align="center">
 <img src="doc/png/arch.svg" align="center" alt="/Overall Architecture">
 </div>
 
-### 1.3 Ethernet supported
+### 1.3. Ethernet supported
 
 MTL offers versatile Ethernet support, thanks to its compatibility with DPDK PMD, kernel socket, and AF_XDP backends.
 
@@ -76,25 +76,25 @@ An important point to note is that narrow pacing of TX is only supported for the
 
 ## 2. Build
 
-Please refer to [build guide](doc/build.md) for instructions on how to build DPDK, the library, and the sample application.
+Please refer to [Build Guide](doc/build.md) for instructions on how to build DPDK, the library, and the sample application.
 
-For Windows, please refer to the [Win build guide](doc/build_WIN.md) for instructions on how to build.
+For Windows, please refer to the [Windows Build Guide](doc/build_WIN.md) for instructions on how to build.
 
 ## 3. Run ST2110
 
-Please refer to [run guide](doc/run.md) for instructions on how to set up and run the demo pipeline application based on DPDK PMD backend.
+Please refer to [Run Guide](doc/run.md) for instructions on how to set up and run the demo pipeline application based on DPDK PMD backend.
 
-For Windows, please refer to [Windows run guide](doc/run_WIN.md).
+For Windows, please refer to [Run Guide on Windows](doc/run_WIN.md).
 
-Additionally, please refer to the [VM guide](doc/vm.md) and [Windows VM guide](doc/vm_WIN.md) for instructions on setting up Linux and Windows guest VMs based on VF passthrough.
+Additionally, please refer to the [VM Guide](doc/vm.md) and [Windows VM Guide](doc/vm_WIN.md) for instructions on setting up Linux and Windows guest VMs based on VF passthrough.
 
-For AWS (cloud environment), please refer to [AWS run guide](doc/aws.md) for instructions on how to set up and run the demo.
+For AWS (cloud environment), please refer to [AWS Run Guide](doc/aws.md) for instructions on how to set up and run the demo.
 
 To run this library on the kernel network stack with the built-in kernel NIC driver, please follow the instructions provided in the [kernel socket guide](doc/kernel_socket.md).
 
 ## 4. ST2110 Programmers guide
 
-To quickly develop applications based on the Media Transport Library, please refer to `## 6. ST2110 API` from [design guide](doc/design.md).
+To quickly develop applications based on the Media Transport Library, please refer to section ["ST2110 API" in Design Guide](doc/design.md#6-st2110-api).
 
 ## 5. User space LibOS UDP stack guide
 
@@ -115,7 +115,7 @@ Whitepaper: Open Source Library Enables Real-Time Media over IP Networks. <https
 
 We welcome community contributions to the Media Transport Library project. If you have any ideas or issues, please share them with us by using GitHub issues or opening a pull request.
 
-### 7.1 Fork this repository
+### 7.1. Fork this repository
 
 Before opening a pull request, please follow these steps:
 
@@ -127,19 +127,19 @@ Before opening a pull request, please follow these steps:
 
 If you do not want the main branch automatically synced to the upstream, please go to `Actions` and disable the `Upstream Sync` workflow.
 
-### 7.2 Coding style
+### 7.2. Coding style
 
 We use the super-linter action for style checks.
 
-#### 7.2.1 C/C++
+#### 7.2.1. C/C++
 
 For C/C++ coding, you can run the following command to quickly fix the style:
 
 ```bash
 ./format-coding.sh
 ```
 
-#### 7.2.2 Python
+#### 7.2.2. Python
 
 For Python, `black` and `isort` formatter is used.
 
@@ -155,7 +155,7 @@ isort python/
 find python/example/ -name "*.py" -exec pylint {} \;
 ```
 
-#### 7.2.3 Others
+#### 7.2.3. Others
 
 For other languages, please check with the following example command inside the Docker container:
 

app/sample/msvc/README.md

@@ -6,7 +6,7 @@ The is sample code for how to develop MSVC application quickly based on Media Tr
 
 ## 2. Steps
 
-### 2.1 Prepare latest .lib file
+### 2.1. Prepare latest .lib file
 
 * Build MTL library in MSYS2, see [WIN build guide](../../../doc/build_WIN.md), also need to add MSYS2 binary PATH to system environment variables.
 
@@ -20,7 +20,7 @@ The is sample code for how to develop MSVC application quickly based on Media Tr
 
 * Copy `libmtl.lib` to project folder.
 
-### 2.2 Set project properties
+### 2.2. Set project properties
 
 * Double-click `imtl_sample.sln` to launch VS or create a console app in VS with the `imtl_sample.cpp`.
 
@@ -32,7 +32,7 @@ The is sample code for how to develop MSVC application quickly based on Media Tr
 
 * Click `OK` or `Apply` to save properties.
 
-### 2.3 Build and run
+### 2.3. Build and run
 
 * Inside IDE: Click Run button (Local Windows Debugger/ Start Without Debugging) to build and run the application.
 

app/udp/README.md

@@ -11,7 +11,7 @@ The dir include the simple sample code for how to develop application quickly ba
 [ufd_server_sample.c](ufd_server_sample.c): A udp client(tx) application based on POSIX socket compatible(file descriptor) UDP API.<br>
 [ufd_server.json](ufd_server.json): The sample configuration file for server sample.<br>
 
-### 2.1 Run
+### 2.1. Run
 
 Customize the p_tx_ip args and the ufd_client.json/ufd_server.json as your setup
 

app/v4l2_to_ip/README.md

@@ -29,7 +29,7 @@ sudo media-ctl -v -l "\"Intel IPU6 CSI2 BE SOC\":16 -> \"Intel IPU6 BE SOC captu
 sudo ./build/app/V4l2toIPApp /dev/video51 --log-status --ptp --tsn
 ```
 ## Expected Output
-```bash
+```text
 MT: * *    M T    D E V   S T A T E   * *
 MT: DEV(0): Avr rate, tx: 1743.150045 Mb/s, rx: 0.017576 Mb/s, pkts, tx: 1646228, rx: 245
 MT: CNI(0): eth_rx_rate 0 Mb/s, eth_rx_cnt 245
@@ -43,3 +43,4 @@ MT: PTP(0): rx time error 0, tx time error 0, delta result error 58
 MT: TX_VIDEO_SESSION(0,0:v4l2_st20_tx): fps 49.991044, frame 500 pkts 1646200:1646199 inflight 407057:411550
 MT: TX_VIDEO_SESSION(0,0): throughput 1742.887329 Mb/s: 0.000000 Mb/s, cpu busy 0.000000
 MT: * *    E N D    S T A T E   * *
+```

doc/asan.md

@@ -29,5 +29,5 @@ ST_BUILD_ENABLE_ASAN=true ./build.sh
 ## 3. Run the application to check for any memory issues
 
 ```bash
-./build/app/RxTxApp --config_file tests/script/loop_json/1080p59_1v.json
+./tests/tools/RxTxApp/build/RxTxApp --config_file tests/script/loop_json/1080p59_1v.json
 ```