🚧 start to explore adding subsections to homesection (#125)

* 🚧 start to explore adding subsections to homesection

* ✨ revert to homepage layout as before, add new section

- use the homepage layout as it was before: title, descrpiton, logo and graphical abstract
- if there are components in the main folder, trigger to add section with main folder name if the configuration is built using the directory based method.
  - a new section with the components is added
  - if configs are created manually, the first section is treated as every other section

* 🐛 fix syntax error (two closing parantheses)

* 🐛 Make excel df paths relative insetad of absolute

* 🚧 add current state of qmd notebooks

- move html report file to correct place: update rel. paths
- add others as of now

* 🐛 ignore description.md and empty sections

- do not use description as component in main folder
- the description is added to the key description

* 🐛 resolve file path first

* 🐛 fix issue with non-directory generated config files, add test

- do not treat first section special
- add example of how to run this config - check in cdci workflow

* 🐛 fix path to tests/report_examples

* 🐛 add correct file ending..

* ✅ test all report types based on quarto (update static path setting)

* 🐛 bash for loop needs to be close using done

* 🐛 get rid of absolute path (missed in #136)

- update tests
- make altair plot path relative to qmd file

* 🎨 clean-up comments/prints and improve logic

* 📝 Update example config files for basic and EMP case studies and update execution section of README

* 💚 Correct paths relative to the docs folder to pass CI tests

* 📝 Update README with details about running vuegen using the cofig exmaple files

* 🐛 commit change to test from svg to png logo

---------

Co-authored-by: sayalaruano <[email protected]>

Author: enryH

.github/workflows/cdci.yml

@@ -109,6 +109,15 @@ jobs:
           cd docs
           vuegen -dir example_data/Earth_microbiome_vuegen_demo_notebook -rt jupyter
           vuegen -c example_data/Earth_microbiome_vuegen_demo_notebook/Earth_microbiome_vuegen_demo_notebook_config.yaml -rt jupyter
+      - name: quarto html report based on predefined config file
+        run: |
+          cd docs
+          vuegen -c example_config_files/Basic_example_vuegen_demo_notebook_config.yaml -output_dir ../tests/report_examples/Basic_example_vuegen_demo_notebook_cfg/html -rt html
+          # Check for changes
+          if git diff ../tests/report_examples | grep .; then
+            echo "Error: One or more protected files have been modified."
+            exit 1
+          fi
       - name: check for changes in report files
         run: |
           # write streamlit report to test folder 
@@ -118,7 +127,18 @@ jobs:
             echo "Error: One or more protected files have been modified."
             exit 1
           fi
-
+      - name: check for changes in quarto report files 
+        run: |
+          # write quarto based report to test folder
+          for format in html pdf docx odt revealjs pptx jupyter; do
+            vuegen -dir docs/example_data/Basic_example_vuegen_demo_notebook -output_dir tests/report_examples/Basic_example_vuegen_demo_notebook/$format  -rt $format 
+            # Check for changes
+            if git diff tests/report_examples | grep .; then
+              echo Failed for report: $format
+              echo "Error: One or more protected files have been modified."
+              exit 1
+            fi
+          done
 
   publish:
     name: Publish package to PyPI

README.md

@@ -126,6 +126,9 @@ vuegen --directory docs/example_data/Earth_microbiome_vuegen_demo_notebook --rep
 
 > [!NOTE]
 > By default, the `streamlit_autorun` argument is set to False, but you can use it in case you want to automatically run the streamlit app.
+> You can also specify the output directory with the `--output_directory` argumument, which defaults to the current working directory.
+> See all available arguments with the `--help` option.
+
 
 ### Folder structure
 
@@ -154,6 +157,8 @@ report_folder/
 The titles for sections, subsections, and components are extracted from the corresponding folder and file names, and afterward, users can add descriptions, captions, and other details to the configuration file. Component types are inferred from the file extensions and names.
 The order of sections, subsections, and components can be defined using numerical suffixes in folder and file names.
 
+### Configuration file
+
 It's also possible to provide a configuration file instead of a directory:
 
 ```bash
@@ -162,6 +167,8 @@ vuegen --config docs/example_config_files/Earth_microbiome_vuegen_demo_notebook.
 
 If a configuration file is given, users can specify titles and descriptions for sections and subsections, as well as component paths and required attributes, such as file format and delimiter for dataframes, plot types, and other details.
 
+The component paths in the configuration file can be absolute or relative to the execution directory. In the examples, we assume that the working directory is the `docs` folder, so the paths are relative to it. If you run VueGen from another directory, you need to adjust the paths accordingly.
+
 The current report types supported by VueGen are:
 
 - Streamlit

docs/example_config_files/Basic_example_vuegen_demo_notebook_config.yaml

@@ -1,8 +1,8 @@
 report:
   title: Basic Example Vuegen Demo Notebook
   description: A general description of the report.
-  graphical_abstract: https://raw.githubusercontent.com/Multiomics-Analytics-Group/vuegen/main/docs/images/vuegen_logo.svg
-  logo: https://raw.githubusercontent.com/Multiomics-Analytics-Group/vuegen/main/docs/images/vuegen_logo.svg
+  graphical_abstract: https://raw.githubusercontent.com/Multiomics-Analytics-Group/vuegen/main/docs/images/vuegen_logo.png
+  logo: https://raw.githubusercontent.com/Multiomics-Analytics-Group/vuegen/main/docs/images/vuegen_logo.png
 sections:
 - title: Plots
   description: This section contains example plots.

docs/example_config_files/Earth_microbiome_vuegen_demo_notebook_config.yaml

@@ -123,8 +123,7 @@ sections:
       component_type: PLOT
       plot_type: STATIC
   - title: Shanon entropy analysis
-    description: This subsection contains the Shannon entropy analysis of the EMP
-    dataset.
+    description: This subsection contains the Shannon entropy analysis of the EMP dataset.
     components:
     - title: Specificity of sequences and higher taxonomic groups for environment
       file_path: https://raw.githubusercontent.com/biocore/emp/master/methods/images/figure4_entropy.png

src/vuegen/config_manager.py

@@ -316,6 +316,7 @@ def create_yamlconfig_fromdir(
         # Generate the YAML config
         yaml_config = {
             "report": {
+                # This will be used for the home section of a report
                 "title": self._create_title_fromdir(base_dir_path.name),
                 "description": self._read_description_file(base_dir_path),
                 "graphical_abstract": "",
@@ -328,11 +329,10 @@ def create_yamlconfig_fromdir(
         sorted_sections = self._sort_paths_by_numprefix(list(base_dir_path.iterdir()))
 
         main_section_config = {
-            "title": "",
-            "description": "Components added to homepage.",
+            "title": self._create_title_fromdir(base_dir_path.name),
+            "description": "",
             "components": [],
         }
-        yaml_config["sections"].append(main_section_config)
 
         # Generate sections and subsections config
         for section_dir in sorted_sections:
@@ -343,12 +343,20 @@ def create_yamlconfig_fromdir(
             # could be single plots?
             else:
                 file_in_main_section_dir = section_dir
+                if file_in_main_section_dir.name.lower() == "description.md":
+                    continue  # Skip description files in the main section
                 component_config = self._create_component_config_fromfile(
                     file_in_main_section_dir
                 )
                 if component_config is not None:
                     main_section_config["components"].append(component_config)
 
+        if main_section_config["components"]:
+            # If components were added to the main section, i.e. there were components
+            # found in the main report directory, add it to the first position of the
+            # list of sections
+            yaml_config["sections"].insert(0, main_section_config)
+
         return yaml_config, base_dir_path
 
     def initialize_report(self, config: dict) -> tuple[r.Report, dict]:

src/vuegen/quarto_reportview.py

@@ -85,7 +85,7 @@ def generate_report(self, output_dir: Optional[Path] = None) -> None:
             Will overwrite value set on initialization of QuartoReportView.
         """
         if output_dir is not None:
-            self.output_dir = Path(output_dir).resolve().absolute()
+            self.output_dir = Path(output_dir).resolve()
 
         self.report.logger.debug(
             f"Generating '{self.report_type}' report in directory: '{self.output_dir}'"
@@ -130,35 +130,10 @@ def generate_report(self, output_dir: Optional[Path] = None) -> None:
                 qmd_content.append(
                     self._generate_image_content(self.report.graphical_abstract)
                 )
-            # ? Do we need to handle overview separately?
-            main_section = self.report.sections[0]
-
-            # ! description can be a Markdown component, but it is treated differently
-            # ! It won't be added to the section content.
-            if main_section.components:
-                self.report.logger.debug(
-                    "Adding components of main section folder to the report as overall overview."
-                )
-                section_content, section_imports = self._combine_components(
-                    main_section.components
-                )
-                if section_content:
-                    qmd_content.append("# General Overview")
-
-                    if is_report_revealjs:
-                        # Add tabset for revealjs
-                        section_content = [
-                            "::: {.panel-tabset}\n",
-                            *section_content,
-                            ":::",
-                        ]
-                    qmd_content.extend(section_content)
-
-                report_imports.extend(section_imports)
 
             # Add the sections and subsections to the report
             self.report.logger.info("Starting to generate sections for the report.")
-            for section in self.report.sections[1:]:
+            for section in self.report.sections:
                 self.report.logger.debug(
                     f"Processing section: '{section.title}' - {len(section.subsections)} subsection(s)"
                 )
@@ -568,7 +543,7 @@ def _generate_plot_content(self, plot) -> List[str]:
             # ? should that be in the output folder
             static_plot_path = (
                 Path(self.static_dir) / f"{plot.title.replace(' ', '_')}.png"
-            ).absolute()
+            ).resolve()
             self.report.logger.debug(f"Static plot path: {static_plot_path}")
         else:
             html_plot_file = (
@@ -594,7 +569,7 @@ def _generate_plot_content(self, plot) -> List[str]:
                 plot_content.append(self._generate_plot_code(plot))
                 if self.is_report_static:
                     plot_content.append(
-                        f"""fig_altair.save("{static_plot_path.as_posix()}")\n```\n"""
+                        f"""fig_altair.save("{static_plot_path.relative_to(self.output_dir).as_posix()}")\n```\n"""
                     )
                     plot_content.append(self._generate_image_content(static_plot_path))
                 else:

src/vuegen/streamlit_reportview.py

@@ -150,14 +150,12 @@ def generate_report(self, output_dir: str = SECTIONS_DIR) -> None:
             report_manag_content.append("\nsections_pages = {}")
 
             # Generate the home page and update the report manager content
-            # ! top level files (compontents) are added to the home page
             self._generate_home_section(
                 output_dir=output_dir,
                 report_manag_content=report_manag_content,
-                home_section=self.report.sections[0],
             )
 
-            for section in self.report.sections[1:]:  # skip home section components
+            for section in self.report.sections:
                 # Create a folder for each section
                 subsection_page_vars = []
                 section_name_var = make_valid_identifier(
@@ -354,7 +352,6 @@ def _generate_home_section(
         self,
         output_dir: str,
         report_manag_content: list,
-        home_section: r.Section,
     ) -> None:
         """
         Generates the homepage for the report and updates the report manager content.
@@ -367,13 +364,7 @@ def _generate_home_section(
             A list to store the content that will be written to the report manager file.
         """
         self.report.logger.debug("Processing home section.")
-        all_components = []
-        subsection_imports = []
-        if home_section.components:
-            # some assert on title?
-            all_components, subsection_imports, _ = self._combine_components(
-                home_section.components
-            )
+
         try:
             # Create folder for the home page
             home_dir_path = Path(output_dir) / "Home"
@@ -386,11 +377,7 @@ def _generate_home_section(
 
             # Create the home page content
             home_content = []
-            subsection_imports.append("import streamlit as st")
-            subsection_imports = set(subsection_imports)
-            subsection_imports, _ = sort_imports(subsection_imports)
-
-            home_content.extend(subsection_imports)
+            home_content.append("import streamlit as st")
             if self.report.description:
                 home_content.append(
                     self._format_text(text=self.report.description, type="paragraph")
@@ -401,8 +388,6 @@ def _generate_home_section(
                 )
 
             # add components content to page (if any)
-            if all_components:
-                home_content.extend(all_components)
 
             # Define the footer variable and add it to the home page content
             home_content.append("footer = '''" + generate_footer() + "'''\n")
@@ -434,13 +419,11 @@ def _generate_sections(self, output_dir: str) -> None:
             The folder where section files will be saved.
         """
         self.report.logger.info("Starting to generate sections for the report.")
-
         try:
-            for section in self.report.sections[1:]:
+            for section in self.report.sections:
                 self.report.logger.debug(
                     f"Processing section '{section.id}': '{section.title}' - {len(section.subsections)} subsection(s)"
                 )
-
                 if section.components:
                     # add an section overview page
                     section_content, section_imports, _ = self._combine_components(
@@ -456,9 +439,8 @@ def _generate_sections(self, output_dir: str) -> None:
                     )
 
                 if not section.subsections:
-                    self.report.logger.warning(
-                        f"No subsections found in section: '{section.title}'. "
-                        "To show content in the report, add subsections to the section."
+                    self.report.logger.debug(
+                        f"No subsections found in section: '{section.title}'."
                     )
                     continue
 
@@ -631,7 +613,7 @@ def _generate_plot_content(self, plot) -> List[str]:
                 else:
                     plot_content.append(
                         f"""
-with open('{Path(html_plot_file).relative_to(Path.cwd())}', 'r') as html_file:
+with open('{Path(html_plot_file).resolve().relative_to(Path.cwd())}', 'r') as html_file:
     html_content = html_file.read()\n"""
                     )