Script to generate keywords added

ATBD and PUM templates updated

Show Version field in HTML doc

Author: mckeea

.github/workflows/render-quarto.yaml

@@ -36,6 +36,17 @@ jobs:
           sudo apt install -y libreoffice libreoffice-common libreoffice-writer fonts-dejavu
           sudo apt install -y coreutils procps
 
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+
+      - name: Generate keywords for documents
+        run: |
+          python -m pip install --upgrade pip
+          pip install keybert ruamel.yaml pyyaml
+          python scripts/generate_keywords.py
+
       - name: Remove cached version of _site
         run: |
           rm -rf _site

metadata/default-before-body.html

@@ -0,0 +1,23 @@
+
+<script>
+  document.addEventListener("DOMContentLoaded", function() {
+
+    // Remove link to .docx file
+    //
+    document.querySelectorAll("a[href$='.docx']").forEach(link => link.style.display = 'none');
+
+    // Add Version to the title's metadata
+    //
+    const titleMetaEl = document.querySelector(".quarto-title-meta");
+    const wrapperDiv = document.createElement('div');
+    wrapperDiv.innerHTML = `
+      <div class="quarto-title-meta-heading">Version</div>
+      <div class="quarto-title-meta-contents">
+        <p>{{< meta version >}}</p>
+      </div>
+    `;
+    titleMetaEl.appendChild(wrapperDiv);
+
+    });
+
+</script>

metadata/default.yml

@@ -6,10 +6,12 @@ number-sections: true
 sitemap: true                       # Enables sitemap generation for web crawlers
 tbl-colwidths: auto
 license: "EUPL (>= 1.2)"
+keywords: "{keywords_str}"
 format:
   html:
     include-in-header:
       - ../../metadata/json-ld.html
+    include-before-body: ../../metadata/default-before-body.html
     code-fold: true                 # Allow code blocks to be foldable
     self-contained: true            # Embed resources like CSS and images
   docx:

scripts/generate_keywords.py

@@ -0,0 +1,71 @@
+import yaml
+import re
+import glob
+from pathlib import Path
+from keybert import KeyBERT
+from io import StringIO
+from ruamel.yaml import YAML
+from ruamel.yaml.comments import CommentedSeq
+from ruamel.yaml.scalarstring import DoubleQuotedScalarString
+
+
+QUARTO_CONFIG="_quarto.yaml"
+
+kw_model = KeyBERT('paraphrase-mpnet-base-v2')
+
+def get_render_files(quarto_yml_path=QUARTO_CONFIG):
+    with open(quarto_yml_path, 'r') as f:
+        config = yaml.safe_load(f)
+
+    render_list = config.get('project', {}).get('render', [])
+    matched_files = []
+
+    for pattern in render_list:
+        matched = glob.glob(pattern, recursive=True)
+        matched_files.extend([Path(f) for f in matched if f.endswith('.qmd')])
+
+    # Exclude files named 'index.qmd'
+    return [f for f in matched_files if f.name.lower() != 'index.qmd']
+
+def inject_keywords(file_path: Path):
+    with open(file_path, 'r', encoding='utf-8') as f:
+        content = f.read()
+
+    parts = re.split(r'^---\s*$', content, maxsplit=2, flags=re.MULTILINE)
+    if len(parts) < 3:
+        return
+
+    _, yaml_block, body = parts
+    
+    yaml = YAML()
+    yaml.preserve_quotes = True
+    yaml_data = yaml.load(StringIO(yaml_block))
+
+    keywords = kw_model.extract_keywords(body, 
+                                         keyphrase_ngram_range=(1, 2), 
+                                         stop_words='english', 
+                                         use_mmr=True, 
+                                         diversity=0.2,
+                                         top_n=10)
+
+    # Update keywords field
+    keywords_list = CommentedSeq([DoubleQuotedScalarString(kw) for kw, _ in keywords])
+    keywords_list.fa.set_flow_style() 
+    yaml_data['keywords'] = keywords_list
+
+    # Reconstruct YAML as string
+    yaml_out = StringIO()
+    yaml.dump(yaml_data, yaml_out)
+    final_yaml = yaml_out.getvalue().strip()
+
+    # Combine back
+    updated_content = f"---\n{final_yaml}\n---\n{body}"
+
+    with open(file_path, 'w', encoding='utf-8') as f:
+        f.write(updated_content)
+        
+
+if __name__ == "__main__":
+    files = get_render_files()
+    for file in files:
+        inject_keywords(file)

src/guidelines/IT_Architecture_Principles_and_Implementation_Guidelines.qmd

@@ -1,33 +1,23 @@
 ---
 title: "IT Architecture Principles and Implementation Guidelines"
 subtitle: "Copernicus Land Monitoring Service"
-author: "European Environment Agency (EEA)"  
+author: "European Environment Agency (EEA)"
 version: "1.4a"
 date: "2025-03-06"
 product-name: IT Architecture Principles and Implementation Guidelines
 description: "IT Architecture Principles and Implementation Guidelines"
-keywords: ["Copernicus Land Monitoring Service, CLMS IT Architecture, European Environment Agency, IT Principles and Guidelines, IT Ecosystem, IT Security, EUPL Licensing, Reproducibility, Reusability, Transparency, Scalability, Maintainability, Resilient IT Solutions, Modular IT Architecture, Continuous Integration"]
-
 
 metadata-files:
-  - ../../metadata/default.yml
-  
+- ../../metadata/default.yml
+
 format:
-  pdf: default
-  html: 
+  html:
     css: ../styles/styles.css
   docx:
-    reference-doc: ../styles/template-guideline.docx  
-
+    reference-doc: ../styles/template-guideline.docx
+  pdf: default  
 ---
 
-```{=html}
-<script>
-  document.addEventListener("DOMContentLoaded", function() {
-    document.querySelectorAll("a[href$='.docx']").forEach(link => link.style.display = 'none');
-  });
-</script>
-```
 
 # Preface {#preface}
 

src/styles/template-atbd.docx

@@ -2,27 +2,19 @@
 title: "Product SHORT NAME ALGORITHM THEORETICAL BASIS DOCUMENT (ATBD)"
 subtitle: "ATBD Copernicus Land Monitoring Service -- Product full name"
 date: "2022-10-06"
-version: Issue x.y (“(x) version of the document” + “.”+ “(y) version  of the document update”)
+version: Issue x.y (“(x) version of the document” + “.”+ “(y) version  of the document
+  update”)
 product-name: Product Name
 description: "Product DESCRIPTION"
-    
 metadata-files:
-  - ../../metadata/default.yml
-  
+- ../../metadata/default.yml
+
 format:
   docx:
     reference-doc: ../styles/template-atbd.docx
     hidden: true
 ---
 
-```{=html}
-<script>
-  document.addEventListener("DOMContentLoaded", function() {
-    document.querySelectorAll("a[href$='.docx']").forEach(link => link.style.display = 'none');
-  });
-</script>
-```
-
 {{< pagebreak >}}
 
 ::: {.subtitle custom-style="subtitle"}
@@ -221,7 +213,7 @@ In detail, the document is structured as follows:
 # Methodology [(mandatory chapter)]{.yellow-box custom-style="yellow-box-line"}
 
 :::{.yellow-box custom-style="yellow-box"}
-Describes the workflow, input datasets, production methodology including data pre-processing, processing, post-processing and quality control and internal validation.
+Describes the workflow, input datasets, production methodology including data pre-processing, processing, post-processing and quality control and product verification.
 
 In the "Methodology Description" chapter, some of the subchapters are classified as mandatory, but only if they are relevant to the methodology being described. Optional subchapters can be included or omitted, allowing flexibility to adapt the structure to the specific characteristics of the product.
 
@@ -260,18 +252,24 @@ Provide an overview table of methodology. Follow the example below:
       <td>Example: 10m spatial resolution</td>
     </tr>
     <tr>
-      <td>Validation and Accuracy</td>
-      <td>Example: Validation using ground-truth data, accuracy assessment with RMSE calculations.</td>
+      <td>Verification and Accuracy</td>
+      <td>Example: Product verification using ground-based reference, accuracy assessment with RMSE calculations.</td>
     </tr>
   </tbody>
 </table>
 ```
 {{< pagebreak >}}
 
+## Theoretical Background [(optional subchapter)]{.yellow-box custom-style="yellow-box-line"}
+
+:::{.yellow-box custom-style="yellow-box"}
+Provides an overview of concepts, common definitions and assumptions of the product and links it to related previous studies and work to allow an understanding of the product and applied methodology.
+:::
+
 ## Methodology and workflow [(mandatory subchapter)]{.yellow-box custom-style="yellow-box-line"}
 
 :::{.yellow-box custom-style="yellow-box"}
-Provides an overview of the product generation, highlighting the workflow, the input datasets, key processes involved, and algorithms used.
+Provides an overview of the product generation, highlighting the workflow, the input datasets, key processes involved, and algorithms used. Ideally, the description is complemented by a traceability diagram that allows users to understand the uncertainty budget of the product as a result of its inputs and workflow (if needed, add subchapter).
 
 Below is a generic example that should be adapted to the specific needs of the product to provide an overview of the production process workflow.
 :::
@@ -286,7 +284,7 @@ Below is a generic example that should be adapted to the specific needs of the p
 Explain which datasets (e.g., Sentinel-x, DEM, or other CLMS products) serve as input for the product. Clarify internal dependencies within the product, such as primary data used to generate secondary outputs. Reference external thematic data where applicable and ensure the workflow diagram illustrates the connections between different datasets.
 :::
 
-## Pre-processing [(mandatory subchapter)]{.yellow-box custom-style="yellow-box-line"}
+## Pre-processing [(optional subchapter)]{.yellow-box custom-style="yellow-box-line"}
 
 :::{.yellow-box custom-style="yellow-box"}
 Details the initial steps taken to prepare the data for further processing, including corrections and adjustments to raw data. Sub-chapter structure optional.
@@ -348,23 +346,23 @@ Outlines the workflow and methods used during the processing stage, detailing ea
 Provides detailed descriptions of the algorithms used in the processing stage.
 :::
 
-## Post-processing [(mandatory subchapter)]{.yellow-box custom-style="yellow-box-line"}
+## Post-processing [(optional subchapter)]{.yellow-box custom-style="yellow-box-line"}
 
 :::{.yellow-box custom-style="yellow-box"}
-Presents tasks typically involved in the refinement, validation, and finalization of the datasets such has manual steps, filtering and smoothing, integration of ancillary data, reprojecting, resampling or merging, classification improvements and preparing data for end-users. A subchapter structure as suggested with previous sections may be introduced if necessary.
+Presents tasks typically involved in the refinement, quality assurance and control, and finalization of the datasets such has manual steps, filtering and smoothing, integration of ancillary data, reprojecting, resampling or merging, classification improvements and preparing data for end-users. A subchapter structure as suggested with previous sections may be introduced if necessary.
 :::
 
 ## Output products [(mandatory subchapter)]{.yellow-box custom-style="yellow-box-line"}
 
 :::{.yellow-box custom-style="yellow-box"}
-Specifies the types of output products generated (e.g., status layer / change, aggregated, expert, reference layers / and other datasets).
+Specifies the types of output products generated (e.g., status layer/change/auxiliary layer, reference layer, near-real time).
 :::
 
 {{< pagebreak >}}
 # Quality control and production verification [(mandatory subchapter)]{.yellow-box custom-style="yellow-box-line"}
 
 :::{.yellow-box custom-style="yellow-box"}
-Discusses the methods and criteria used to assess the internal validation process (i.e. production verification), the quality and accuracy of the products, ensuring they meet the required standards and specifications.
+Discusses the methods and criteria used to assess the internal quality assurance and control process as well as production verification, the quality and accuracy of the products, ensuring they meet the required standards and specifications.
 :::
 
 {{< pagebreak >}}
@@ -381,7 +379,6 @@ Discusses any challenges or limitations of the current methodology, algorithm, i
 Presents information on the terms of use, citation guidelines, and technical support for the products.
 :::
 
-{{< pagebreak >}}
 ## Terms of use [(mandatory subchapter)]{.yellow-box custom-style="yellow-box-line"}
 
 :::{.yellow-box custom-style="yellow-box"}
@@ -391,17 +388,18 @@ Here an example of a standard text.
 :::
 
 :::{.grey-box custom-style="grey-box"}
-The Terms of Use for the product(s) described in this document, acknowledge that:
-
-"Free, full and open access to the products and services of the Copernicus Land Monitoring Service is made on the conditions that:
+The Terms of Use for the product(s) described in this document, acknowledge the following:  
+Free, full and open access to the products and services of the Copernicus Land Monitoring Service is made on the conditions that:
 
 1.  When distributing or communicating Copernicus Land Monitoring Service products and services (data, software scripts, web services, user and methodological documentation and similar) to the public, users shall inform the public of the source of these products and services and shall acknowledge that the Copernicus Land Monitoring Service products and services were produced "with funding by the European Union".
 
 2.  Where the Copernicus Land Monitoring Service products and services have been adapted or modified by the user, the user shall clearly state this.
 
 3.  Users shall make sure not to convey the impression to the public that the user\'s activities are officially endorsed by the European Union.
 
-The user has all intellectual property rights to the products he/she has created based on the Copernicus Land Monitoring Service products and services." [^1]
+The user has all intellectual property rights to the products he/she has created based on the Copernicus Land Monitoring Service products and services. 
+
+[Consult Data policy — Copernicus Land Monitoring Service for further details](https://land.copernicus.eu/en/data-policy)
 :::
 
 ## Citation [(mandatory subchapter)]{.yellow-box custom-style="yellow-box-line"}
@@ -413,17 +411,20 @@ Here are examples of standard text to be used according to the situation:
 :::
 
 :::{.grey-box custom-style="grey-box"}
-When **planning to publish a publication (scientific, commercial, etc.)**, it should be explicitly mentioned:
+When **planning to publish a publication (scientific, commercial, etc.)**, it shall explicitly mention:
+
+> "This publication has been prepared using European Union\'s Copernicus Land Monitoring Service information; \<insert all relevant DOI links here, if applicable\>"
 
-> "This publication has been prepared using European Union\'s Copernicus Land Monitoring Service information; \<insert all relevant DOI links here, if applicable\>"^1^
+When developing a **product or service using the products or services of the Copernicus Land Monitoring Service**, it shall explicitly mention:
 
-When developing a **product or service using the products or services of the Copernicus Land Monitoring Service**, it should explicitly mentioned:
+> "Generated using European Union\'s Copernicus Land Monitoring Service information; \<insert all relevant DOI links here, if applicable\>"
 
-> "Generated using European Union\'s Copernicus Land Monitoring Service information; \<insert all relevant DOI links here, if applicable\>"^1^
+When **redistributing a part of the Copernicus Land Monitoring Service (product, dataset, documentation, picture, web service, etc.)**, it shall explicitly mention:
 
-When **redistributing a part of the Copernicus Land Monitoring Service (product, dataset, documentation, picture, web service, etc.)**, it should explicitly mentioned:
+> "European Union\'s Copernicus Land Monitoring Service information; \<insert all relevant DOI links here, if applicable\>"
+
+[Consult Data policy — Copernicus Land Monitoring Service for further details](https://land.copernicus.eu/en/data-policy)
 
-> "European Union\'s Copernicus Land Monitoring Service information; \<insert all relevant DOI links here, if applicable\>"^1^
 :::
 
 ## Product technical support [(mandatory subchapter)]{.yellow-box custom-style="yellow-box-line"}
@@ -435,8 +436,7 @@ Here an example of a standard text.
 :::
 
 :::{.grey-box custom-style="grey-box"}
-Product technical support is provided by the product custodian through Copernicus Land Monitoring Service desk[^2]. Product technical support does not include software specific user support or general GIS or remote sensing support.
-
+Product technical support is provided by the product custodian through [Copernicus Land Monitoring Service -- Service desk](https://land.copernicus.eu/en/data-policy). Product technical support does not include software specific user support or general GIS or remote sensing support.  
 More information on the products can be found on the Copernicus Land Monitoring Service website (https://land.copernicus.eu/)
 :::
 
@@ -472,7 +472,3 @@ Lists references cited throughout the document.
 :::{.yellow-box custom-style="yellow-box"}
 Annexes should be kept to a minimum but can be placed as needed to present technical details, data and information repository, resources on product development.
 :::
-
-[^1]: [Copernicus Land Monitoring Service - Data policy](https://land.copernicus.eu/en/data-policy)
-
-[^2]: [Copernicus Land Monitoring Service -- Service desk](https://land.copernicus.eu/en/data-policy)