Merge pull request #309 from softwarepub/feature/init-command

Feature/init command

Author: poikilotherm

docs/source/tutorials/writing-a-plugin-for-hermes.md

@@ -1,12 +1,14 @@
 <!--
-SPDX-FileCopyrightText: 2024 German Aerospace Center (DLR)
+SPDX-FileCopyrightText: 2025 German Aerospace Center (DLR), Forschungszentrum Jülich GmbH
 
 SPDX-License-Identifier: CC-BY-SA-4.0
 -->
 
 <!--
 SPDX-FileContributor: Michael Meinel
 SPDX-FileContributor: Sophie Kernchen
+SPDX-FileContributor: Nitai Heeb
+SPDX-FileContributor: Oliver Bertuch
 -->
 
 # Write a plugin for HERMES
@@ -26,7 +28,7 @@ If you never used HERMES before, you might want to check the tutorial: [Automate
 
 HERMES uses a plugin architecture. Therefore, users are invited to contribute own features.
 The structure for every plugin follows the same schema.
-There is a top-level base class for every plugin. In this `HermesPlugin` class there is one abstract method `__ call __` which needs to be overwritten.
+There is a top-level base class for every plugin. In this `HermesPlugin` class there is one abstract method `__call__` which needs to be overwritten.
 Furthermore, the `HermesCommand` class provides all needs for writing a plugin used in a HERMES command.
 So the `HermesPlugin`s call method gets an instance of the `HermesCommand` that triggered this plugin to run.
 In our case this will be the `HermesHarvestCommand` which calls all harvest plugins.
@@ -38,6 +40,7 @@ To overwrite a parameter from command line, use the `-O` command line option fol
 E.g., you can set your authentication token for InvenioRDM by adding the following options to your call to `hermes deposit`:
 ```shell
 hermes deposit -O invenio_rdm.auth_token YourSecretAuthToken
+```
 
 ## Set Up Plugin
 To write a new plugin, it is important to follow the given structure.
@@ -64,15 +67,21 @@ class GitHarvestPlugin(HermesHarvestPlugin):
         return {}, {}
 ```
 
-The code uses the `HermesHarvestPlugin` as base class and pydantics Basemodel for the settings. In the `GitHarvestSettings` you
-can see that an additional parameter is defined. The Parameter `from_branch` is specific for this plugin and can be accessed inside the plugin using `self.settings.harvest.git.git_branch` as long as our plugin will be named git.
+The code uses the `HermesHarvestPlugin` as base class and pydantic's base model for the settings.
+In the `GitHarvestSettings` you can see that an additional parameter is defined.
+The Parameter `from_branch` is specific for this plugin and can be accessed inside the plugin using `self.settings.harvest.git.from_branch` as long as our plugin will be named `git`.
 In the `hermes.toml` this would be achieved by [harvest.{plugin_name}].
-The `GitHarvestSettings` are associated with the `GitHarvestPlugin`. In the plugin you need to overwrite the `__ call __` method.
-For now a simple Hello World will do. The method returns two dictionaries. These will contain the harvested data in CodeMeta (JSON-LD) and additional information, e.g., to provide provenance information.
+The `GitHarvestSettings` are associated with the `GitHarvestPlugin`.
+In the plugin you need to overwrite the `__call__` method.
+For now a simple "Hello World" will do. The method returns two dictionaries.
+These will contain the harvested data in CodeMeta (JSON-LD) and additional information, e.g., to provide provenance information.
 That is the basic structure for the plugins source code.
 
-To integrate this code, you have to register it as a plugin in the `pyproject.toml`. To learn more about the `pyproject.toml` check https://python-poetry.org/docs/pyproject/ or refer to [PEP621](https://peps.python.org/pep-0621/).
-We will just look at the important places for this plugin. There are two ways to integrate this plugin. First we will show how to use the plugin environment as the running base with HERMES as a dependency.
+To integrate this code, you have to register it as a plugin in the `pyproject.toml`.
+To learn more about the `pyproject.toml` check https://python-poetry.org/docs/pyproject/ or refer to [PEP621](https://peps.python.org/pep-0621/).
+We will just look at the important places for this plugin.
+There are two ways to integrate this plugin.
+First we will show how to use the plugin environment as the running base with HERMES as a dependency.
 Then we say how to integrate this plugin in HERMES itself.
 
 ### Include HERMES as Dependency
@@ -124,10 +133,10 @@ Note that this differs with the accessibility and your wishes, check [Explicit P
 
 The second thing to adapt is to declare the access point for the plugin.
 You can do that with `git = "hermes_plugin_git.harvest:GitHarvestPlugin"`.
-This expression makes the GitHarvestPlugin from the hermes_plugin_git package, a hermes.harvest plugin named git.
+This expression makes the `GitHarvestPlugin` from the `hermes_plugin_git` package, a `hermes.harvest` plugin named `git`.
 So you need to configure this line with your plugin properties.
 
-Now you just need to add the plugin to the hermes.toml and reinstall the adapted poetry package.
+Now you just need to add the plugin to the `hermes.toml` and reinstall the adapted poetry package.
 
 ### Configure hermes.toml
 To use the plugin, you have to activate it in the `hermes.toml`.

src/hermes/commands/init/__init__.py

@@ -1,2 +1,2 @@
-# SPDX-FileCopyrightText: 2024 Forschungszentrum Jülich
+# SPDX-FileCopyrightText: 2024 Forschungszentrum Jülich GmbH
 # SPDX-License-Identifier: Apache-2.0

src/hermes/commands/init/base.py

@@ -1,27 +1,30 @@
-# SPDX-FileCopyrightText: 2024 Forschungszentrum Jülich
+# SPDX-FileCopyrightText: 2024 Forschungszentrum Jülich GmbH
 # SPDX-License-Identifier: Apache-2.0
 # SPDX-FileContributor: Nitai Heeb
 
 import argparse
+import logging
 import os
+import re
 import subprocess
 import sys
-import requests
-import toml
-import re
+from dataclasses import dataclass
 from enum import Enum, auto
-from urllib.parse import urlparse, urljoin
+from importlib import metadata
 from pathlib import Path
+from urllib.parse import urljoin, urlparse
+
+import requests
+import toml
 from pydantic import BaseModel
-from dataclasses import dataclass
-from hermes.commands.base import HermesCommand
-import hermes.commands.init.connect_github as connect_github
-import hermes.commands.init.connect_gitlab as connect_gitlab
-import hermes.commands.init.connect_zenodo as connect_zenodo
-import hermes.commands.init.slim_click as sc
+from requests import HTTPError
 
+import hermes.commands.init.util.slim_click as sc
+from hermes.commands.base import HermesCommand, HermesPlugin
+from hermes.commands.init.util import (connect_github, connect_gitlab,
+                                       connect_zenodo)
 
-TUTORIAL_URL = "https://docs.software-metadata.pub/en/latest/tutorials/automated-publication-with-ci.html"
+TUTORIAL_URL = "https://hermes.software-metadata.pub/en/latest/tutorials/automated-publication-with-ci.html"
 
 
 class GitHoster(Enum):
@@ -52,7 +55,7 @@ class DepositPlatform(Enum):
 class HermesInitFolderInfo:
     def __init__(self):
         self.absolute_path: str = ""
-        self.has_git: bool = False
+        self.has_git_folder: bool = False
         self.git_remote_url: str = ""
         self.git_base_url: str = ""
         self.used_git_hoster: GitHoster = GitHoster.Empty
@@ -75,31 +78,50 @@ def is_git_installed():
 
 
 def scout_current_folder() -> HermesInitFolderInfo:
+    """
+    This method looks at the current directory and collects all init relevant data.
+
+    @return: HermesInitFolderInfo object containing the gathered knowledge
+    """
     info = HermesInitFolderInfo()
     current_dir = os.getcwd()
     info.current_dir = current_dir
     info.absolute_path = str(current_dir)
-    info.has_git = os.path.isdir(os.path.join(current_dir, ".git"))
-    if info.has_git:
+
+    info.has_git_folder = os.path.isdir(os.path.join(current_dir, ".git"))
+    # git-enabled project
+    if info.has_git_folder:
+        # Get remote name for next command
+        # TODO: missing case of multiple configured remotes (should we make the user choose?)
         git_remote = str(subprocess.run(['git', 'remote'], capture_output=True, text=True).stdout).strip()
         sc.debug_info(f"git remote = {git_remote}")
-        # Get remote url
+
+        # Get remote url via Git CLI and convert it to an HTTP link in case it's an SSH remote.
+        # TODO: unchecked usage of empty remote name
         info.git_remote_url = convert_remote_url(
             str(subprocess.run(['git', 'remote', 'get-url', git_remote], capture_output=True, text=True).stdout)
         )
+        # TODO: missing an "else" part!
+        # TODO: can't these three pieces be stitched together to only execute when there is a remote?
+        # TODO: is the url parsing necessary of it's hosted on github?
+        if info.git_remote_url:
+            parsed_url = urlparse(info.git_remote_url)
+            info.git_base_url = f"{parsed_url.scheme}://{parsed_url.netloc}/"
+            sc.debug_info(f"git base url = {info.git_base_url}")
+        if "github.com" in info.git_remote_url:
+            info.used_git_hoster = GitHoster.GitHub
+        elif connect_gitlab.is_url_gitlab(info.git_base_url):
+            info.used_git_hoster = GitHoster.GitLab
+
+        # Extract current branch name information by parsing Git output
+        # TODO: no exception or handling in case branch is empty (e.g. detached HEAD)
+        # TODO: why not use git rev-parse --abbrev-ref HEAD ?
         branch_info = str(subprocess.run(['git', 'branch'], capture_output=True, text=True).stdout)
         for line in branch_info.splitlines():
             if line.startswith("*"):
                 info.current_branch = line.split()[1].strip()
                 break
-        if info.git_remote_url:
-            parsed_url = urlparse(info.git_remote_url)
-            info.git_base_url = f"{parsed_url.scheme}://{parsed_url.netloc}/"
-            sc.debug_info(f"git base url = {info.git_base_url}")
-    if "github.com" in info.git_remote_url:
-        info.used_git_hoster = GitHoster.GitHub
-    elif connect_gitlab.is_url_gitlab(info.git_base_url):
-        info.used_git_hoster = GitHoster.GitLab
+
     info.has_hermes_toml = os.path.isfile(os.path.join(current_dir, "hermes.toml"))
     info.has_gitignore = os.path.isfile(os.path.join(current_dir, ".gitignore"))
     info.has_citation_cff = os.path.isfile(os.path.join(current_dir, "CITATION.cff"))
@@ -110,15 +132,19 @@ def scout_current_folder() -> HermesInitFolderInfo:
         if os.path.isdir(os.path.join(current_dir, f))
         and not f.startswith(".")
     ]
+
     return info
 
 
 def download_file_from_url(url, filepath, append: bool = False):
-    with requests.get(url, stream=True) as r:
-        r.raise_for_status()
-        with open(filepath, 'ab' if append else 'wb') as f:
-            for chunk in r.iter_content(chunk_size=8192):
-                f.write(chunk)
+    try:
+        with requests.get(url, stream=True) as r:
+            r.raise_for_status()
+            with open(filepath, 'ab' if append else 'wb') as f:
+                for chunk in r.iter_content(chunk_size=8192):
+                    f.write(chunk)
+    except HTTPError:
+        sc.echo(f"No file found at {url}.", formatting=sc.Formats.FAIL)
 
 
 def string_in_file(file_path, search_string: str) -> bool:
@@ -139,6 +165,28 @@ def convert_remote_url(url: str) -> str:
     return url
 
 
+def get_builtin_plugins(plugin_commands: list[str]) -> dict[str: HermesPlugin]:
+    plugins = {}
+    for plugin_command_name in plugin_commands:
+        entry_point_group = f"hermes.{plugin_command_name}"
+        group_plugins = {
+            entry_point.name: entry_point.load()
+            for entry_point in metadata.entry_points(group=entry_point_group)
+        }
+        plugins.update(group_plugins)
+    return plugins
+
+
+def get_handler_by_name(name: str) -> logging.Handler:
+    """Own implementation of logging.getHandlerByName so that we don't require Python 3.12"""
+    for logger_name in logging.root.manager.loggerDict:
+        logger = logging.getLogger(logger_name)
+        for handler in logger.handlers:
+            if handler.get_name() == name:
+                return handler
+    return None
+
+
 class _HermesInitSettings(BaseModel):
     """Configuration of the ``init`` command."""
     pass
@@ -166,6 +214,8 @@ def __init__(self, parser: argparse.ArgumentParser):
             "deposit_extra_files": "",
             "push_branch": "main"
         }
+        self.plugin_relevant_commands = ["harvest", "deposit"]
+        self.builtin_plugins: dict[str: HermesPlugin] = get_builtin_plugins(self.plugin_relevant_commands)
 
     def init_command_parser(self, command_parser: argparse.ArgumentParser) -> None:
         command_parser.add_argument('--template-branch', nargs=1, default="",
@@ -175,11 +225,24 @@ def load_settings(self, args: argparse.Namespace):
         pass
 
     def refresh_folder_info(self):
-        sc.echo("Scanning folder...", debug=True)
+        sc.debug_info("Scanning folder...")
         self.folder_info = scout_current_folder()
-        sc.echo("Scan complete.", debug=True)
+        sc.debug_info("Scan complete.")
+
+    def setup_file_logging(self):
+        # Silence old StreamHandler
+        handler = get_handler_by_name("terminal")
+        if handler:
+            handler.setLevel(logging.CRITICAL)
+        # Set file logger level
+        self.log.setLevel(level=logging.INFO)
+        # Connect logger with sc
+        sc.default_file_logger = self.log
 
     def __call__(self, args: argparse.Namespace) -> None:
+        # Setup logging
+        self.setup_file_logging()
+
         # Save command parameter (template branch)
         if hasattr(args, "template_branch"):
             if args.template_branch != "":
@@ -230,7 +293,7 @@ def test_initialization(self):
         self.refresh_folder_info()
 
         # Abort if there is no git
-        if not self.folder_info.has_git:
+        if not self.folder_info.has_git_folder:
             sc.echo("The current directory has no `.git` subdirectory. "
                     "Please execute `hermes init` in the root directory of your git project.",
                     formatting=sc.Formats.WARNING)
@@ -317,6 +380,7 @@ def update_gitignore(self):
                 with open(".gitignore", "a") as file:
                     file.write("# Ignoring all HERMES cache files\n")
                     file.write(".hermes/\n")
+                    file.write("hermes.log\n")
                 sc.echo("Added `.hermes/` to the `.gitignore` file.", formatting=sc.Formats.OKGREEN)
 
     def get_template_url(self, filename: str) -> str:
@@ -384,12 +448,6 @@ def get_zenodo_token(self):
         # Deactivated Zenodo OAuth as long as the refresh token bug is not fixed.
         if self.setup_method == "a":
             sc.echo("Doing OAuth with Zenodo is currently not available.")
-        #     self.tokens[self.deposit_platform] = "REFRESH_TOKEN:" + connect_zenodo.get_refresh_token()
-        #     if self.tokens[self.deposit_platform]:
-        #         sc.echo("OAuth at Zenodo was successful.")
-        #         sc.echo(self.tokens[self.deposit_platform], debug=True)
-        #     else:
-        #         sc.echo("Something went wrong while doing OAuth. You'll have to do it manually instead.")
         if self.setup_method == "m" or self.tokens[self.deposit_platform] == '':
             zenodo_token_url = urljoin(DepositPlatformUrls[self.deposit_platform],
                                        "account/settings/applications/tokens/new/")
@@ -400,7 +458,18 @@ def get_zenodo_token(self):
             if self.setup_method == "m":
                 sc.press_enter_to_continue()
             else:
-                self.tokens[self.deposit_platform] = sc.answer("Then enter the token here: ")
+                while True:
+                    self.tokens[self.deposit_platform] = sc.answer("Enter the token here: ")
+                    valid = connect_zenodo.test_if_token_is_valid(self.tokens[self.deposit_platform])
+                    if valid:
+                        sc.echo(f"The token was validated by {connect_zenodo.name}.",
+                                formatting=sc.Formats.OKGREEN)
+                        break
+                    else:
+                        sc.echo(f"The token could not be validated by {connect_zenodo.name}. "
+                                "Make sure to enter the complete token.\n"
+                                "(If this error persists, you should try switching to the manual setup mode.)",
+                                formatting=sc.Formats.WARNING)
 
     def configure_git_project(self):
         """Adding the token to the git secrets & changing action workflow settings"""