|
| 1 | +Migrating Legacy Projects |
| 2 | +========================= |
| 3 | + |
| 4 | +Migrating old projects to a new project setup and the :code:`python-toolbox` can be tedious and annoying. This guide will try to provide some basic steps and guidance to simplify this process. |
| 5 | + |
| 6 | +.. _before_you_migrate: |
| 7 | + |
| 8 | +What a Project Should Have Before You Migrate |
| 9 | ++++++++++++++++++++++++++++++++++++++++++++++ |
| 10 | + |
| 11 | +* The project configuration should be :code:`pyproject.toml` based. **[Required]** |
| 12 | +* The project should be :code:`poetry` based. **[Required]** |
| 13 | +* Dependencies should point only to officially published dependencies on PyPI (no git references or similar) **[Required]**. |
| 14 | +* The project documentation should be :code:`sphinx` based **[Required]**. |
| 15 | +* Automated tasks within the project should be available as Python code or `Nox`_ tasks. **[Helpful]** |
| 16 | + |
| 17 | +.. hint:: |
| 18 | + |
| 19 | + If you are interested or want to get a good idea of what a bare standard project based on the toolbox should have, excluding the workflows, it is always good to have a look at the `cookiecutter template <https://github.com/exasol/python-toolbox/tree/main/project-template>`_ provided by the toolbox. If the template parts are confusing, you can generate an "empty" project based on it. For details, refer to the :ref:`new project` section. |
| 20 | + |
| 21 | + |
| 22 | +Iterative Migration Guide |
| 23 | +++++++++++++++++++++++++++ |
| 24 | + |
| 25 | +Ensure you comply with the :ref:`basic requirements <before_you_migrate>` for your project. Follow these steps for a smooth migration process. |
| 26 | + |
| 27 | +1. Introduce Nox |
| 28 | +---------------- |
| 29 | +As a first step, it is generally advisable to introduce `Nox`_ as a task runner if it is not already in use. Since the :code:`python-toolbox` uses `Nox`_ as its foundation, this simplifies the migration to the toolbox and enhances the user's understanding of `Nox`_. |
| 30 | + |
| 31 | +2. Introduce the Standard Nox Tasks |
| 32 | +----------------------------------- |
| 33 | +This can be done incrementally for different checks of your project, such as *project*, *test*, *linting*, *documentation*, and *other* tasks. |
| 34 | +As the tasks can be split into roughly five groups, it likely makes sense to integrate at least one group at a time. |
| 35 | +For more details on the current tasks and groups, refer to: :ref:`toolbox tasks`. |
| 36 | + |
| 37 | + |
| 38 | +Overloading a Task |
| 39 | +__________________ |
| 40 | + |
| 41 | +In certain cases, it may not be trivial to use the nox task defined within the `python-toolbox`_. In those cases, overloads can be used to provide a smoother or faster transition or to cope with project-specific constraints or needs. |
| 42 | + |
| 43 | +For example, if test execution isn't performed in the standard way (e.g., :code:`pytest test/unit`, :code:`pytest test/integration`, :code:`pytest test`). |
| 44 | + |
| 45 | +.. warning:: |
| 46 | + |
| 47 | + While overwriting can be handy and seem like the simplest way to integrate with the toolbox, please take care when making this decision. In the long run, we want to harmonize projects, their builds, execution, testing, and reporting, etc. If you create an overwrite, you are likely to run into troubles later on, e.g., when reporting is required, because you may lack specific artifacts a specific task was creating, etc. |
| 48 | + |
| 49 | + So while overwriting can be a good temporary solution, for the long term, it should be considered where to add or use a configuration point within the toolbox to make the standard task work for *all* projects. |
| 50 | + |
| 51 | + **Potential options here are:** |
| 52 | + |
| 53 | + * Add additional :ref:`plugin extension <plugin support>` points to the toolbox |
| 54 | + * Implement functionality for an :ref:`existing extension point <plugins>` in the `python-toolbox`_ |
| 55 | + * Add additional configuration parameters in :code:`noxconfig.py` or :code:`pyproject.toml` |
| 56 | + * Add additional parameterization to tasks |
| 57 | + * :code:`pytest plugins` can take care of a specific test preparation need, e.g., `pytest-backend <https://github.com/exasol/pytest-plugins/tree/main/pytest-backend>`_ |
| 58 | + |
| 59 | + |
| 60 | +.. code-block:: python |
| 61 | +
|
| 62 | + import nox |
| 63 | +
|
| 64 | + # imports all nox task provided by the toolbox |
| 65 | + from exasol.toolbox.nox.tasks import * # pylint: disable=wildcard-import disable=unused-wildcard-import |
| 66 | +
|
| 67 | + # default actions to be run if nothing is explicitly specified with the -s option |
| 68 | + nox.options.sessions = ["project:fix"] |
| 69 | +
|
| 70 | + @nox.session(name="project:fix", python=False) |
| 71 | + def my_project_fix_overwrite(session) -> None: |
| 72 | + """Runs all automated fixes on the code base""" |
| 73 | +
|
| 74 | + # ATTENTION: |
| 75 | + # In cases where it is reasonable to use "internal" functions, please do those imports |
| 76 | + # within the function to keep them isolated and simplify future removal or replacement. |
| 77 | + from exasol.toolbox.nox._shared import python_files |
| 78 | +
|
| 79 | + py_files = [f"{file}" for file in python_files(PROJECT_CONFIG.root)] |
| 80 | + print("The original 'project:fix' task has been taken hostage by this overwrite") |
| 81 | + print("Files:\n{files}".format(files="\n".join(py_files)) |
| 82 | +
|
| 83 | +
|
| 84 | +3. Establish a Baseline |
| 85 | +----------------------- |
| 86 | +Configure code quality settings in the :code:`pyproject.toml` file to establish a baseline for your project. If necessary, create tickets for further improvements, especially if major parts of your code require suppression, e.g., in the mypy configuration. |
| 87 | +
|
| 88 | +**:code:`pyproject.toml` sections to include/consider:** |
| 89 | +
|
| 90 | +* [tool.coverage.run] |
| 91 | +* [tool.coverage.report] |
| 92 | +* [tool.black] |
| 93 | +* [tool.isort] |
| 94 | +* [tool.pylint.format] |
| 95 | +* [[tool.mypy.overrides]] |
| 96 | +
|
| 97 | +Example |
| 98 | +_______ |
| 99 | +
|
| 100 | +.. code-block:: toml |
| 101 | +
|
| 102 | + # Adjust this section if you want fine-grained control |
| 103 | + # over what is considered for code coverage |
| 104 | + [tool.coverage.run] |
| 105 | + relative_files = true |
| 106 | + source = [ |
| 107 | + "exasol", |
| 108 | + ] |
| 109 | +
|
| 110 | + # Adjust this section to define the minimum required |
| 111 | + # code coverage for your project |
| 112 | + [tool.coverage.report] |
| 113 | + fail_under = 15 |
| 114 | +
|
| 115 | +
|
| 116 | + # Adjust control maximum line length in your project |
| 117 | + # |
| 118 | + # NOTE: |
| 119 | + # As a rule of thumb, you should not exceed 120 characters, |
| 120 | + # because overly long lines usually accompany higher cyclomatic complexity, |
| 121 | + # as complex functions tend to shift right. |
| 122 | + [tool.black] |
| 123 | + line-length = 88 |
| 124 | + include = "\\.pyi?$" |
| 125 | +
|
| 126 | +
|
| 127 | + # Adjust to modify the behavior of import sorting |
| 128 | + [tool.isort] |
| 129 | + profile = "black" |
| 130 | + force_grid_wrap = 2 |
| 131 | +
|
| 132 | +
|
| 133 | + # Adjust to define the minimum linting score considered acceptable for your project |
| 134 | + [tool.pylint.master] |
| 135 | + fail-under = 7.5 |
| 136 | +
|
| 137 | + # Maximum line length should match what is configured for black. |
| 138 | + # Additionally, a maximum module size can be defined here. |
| 139 | + [tool.pylint.format] |
| 140 | + max-line-length = 88 |
| 141 | + max-module-lines = 800 |
| 142 | +
|
| 143 | +
|
| 144 | + # Configure exceptions for the type checker |
| 145 | + [[tool.mypy.overrides]] |
| 146 | + module = [ |
| 147 | + "test.unit.*", |
| 148 | + "test.integration.*", |
| 149 | + ] |
| 150 | + ignore_errors = true |
| 151 | +
|
| 152 | +
|
| 153 | +4. Introduce GitHub Workflows |
| 154 | +----------------------------- |
| 155 | +Install the GitHub workflows provided by the :code:`python-toolbox` for futher details refer to the section :ref:`GitHub Workflows`. |
| 156 | +
|
| 157 | +.. attention:: |
| 158 | + This is just guidance. If you have a good understanding of the standard project setup, technologies, and tools used, feel free to diverge at any point or exercise your own judgment. |
| 159 | +
|
| 160 | +
|
| 161 | +Migration Progess |
| 162 | ++++++++++++++++++ |
| 163 | +
|
| 164 | +Could be tracked in a format and based on the information listed in the real life example bellow. |
| 165 | +
|
| 166 | +.. hint:: |
| 167 | +
|
| 168 | + This table does not provide any information about the specific `python-toolbox`_ used in the respective projects. |
| 169 | +
|
| 170 | +.. list-table:: Migration Progress |
| 171 | + :widths: 20 15 15 15 15 15 15 15 |
| 172 | + :header-rows: 1 |
| 173 | +
|
| 174 | + * - Project |
| 175 | + - pyproject.toml |
| 176 | + - poetry |
| 177 | + - PyPI |
| 178 | + - Sphinx Docs |
| 179 | + - nox |
| 180 | + - toolbox-tasks |
| 181 | + - toolbox-workflows |
| 182 | + * - `python-toolbox`_ |
| 183 | + - ✓ |
| 184 | + - ✓ |
| 185 | + - ✓ |
| 186 | + - ✓ |
| 187 | + - ✓ |
| 188 | + - ✓ |
| 189 | + - ✓ |
| 190 | + * - `error-reporting-python <https://github.com/exasol/error-reporting-python>`_ |
| 191 | + - ✓ |
| 192 | + - ✓ |
| 193 | + - ✓ |
| 194 | + - ✓ |
| 195 | + - ✓ |
| 196 | + - ✓ |
| 197 | + - ✓ |
| 198 | + * - `pyexasol <https://github.com/exasol/pyexasol>`_ |
| 199 | + - ✓ |
| 200 | + - ✓ |
| 201 | + - ✓ |
| 202 | + - ✗ |
| 203 | + - ✓ |
| 204 | + - ✗ |
| 205 | + - ✗ |
| 206 | + * - `sqlalchemy-exasol <https://github.com/exasol/sqlalchemy-exasol>`_ |
| 207 | + - ✓ |
| 208 | + - ✓ |
| 209 | + - ✓ |
| 210 | + - ✓ |
| 211 | + - ✓ |
| 212 | + - ✗ |
| 213 | + - ✗ |
| 214 | + * - `bucketfs-python <https://github.com/exasol/bucketfs-python/tree/main>`_ |
| 215 | + - ✓ |
| 216 | + - ✓ |
| 217 | + - ✓ |
| 218 | + - ✓ |
| 219 | + - ✓ |
| 220 | + - ✓ |
| 221 | + - ✓/✗ partialy |
| 222 | + * - `ITDE <https://github.com/exasol/integration-test-docker-environment>`_ |
| 223 | + - ✓ |
| 224 | + - ✓ |
| 225 | + - ✓ |
| 226 | + - ✓ |
| 227 | + - ✓ |
| 228 | + - ✓/✗ partialy |
| 229 | + - ✓/✗ partialy |
| 230 | + * - `schemas <https://github.com/exasol/schemas>`_ |
| 231 | + - ✓ |
| 232 | + - ✓ |
| 233 | + - ✗ |
| 234 | + - ✗ |
| 235 | + - ✗ |
| 236 | + - ✗ |
| 237 | + - ✗ |
| 238 | + * - `pytest-plugins <https://github.com/exasol/pytest-plugins>`_ |
| 239 | + - ✓ |
| 240 | + - ✓ |
| 241 | + - ✓ |
| 242 | + - ✓/✗ partialy |
| 243 | + - ✓ |
| 244 | + - ✓/✗ partialy |
| 245 | + - ✗ |
| 246 | + * - `harlequin-exasol <https://github.com/Nicoretti/harlequin-exasol>`_ |
| 247 | + - ✓ |
| 248 | + - ✓ |
| 249 | + - ✓ |
| 250 | + - ✓ |
| 251 | + - ✓ |
| 252 | + - ✓ |
| 253 | + - ✗ |
| 254 | +
|
| 255 | +
|
| 256 | +.. list-table:: Legend |
| 257 | + :widths: 20 80 |
| 258 | + :header-rows: 1 |
| 259 | +
|
| 260 | + * - Column |
| 261 | + - Description |
| 262 | + * - Project |
| 263 | + - Name of the project |
| 264 | + * - pyproject.toml |
| 265 | + - Project configuration and setup is `pyproject.toml`_ based |
| 266 | + * - poetry |
| 267 | + - Project configuration and build is `Poetry`_ based |
| 268 | + * - PYPI |
| 269 | + - Project can be build and published to `PyPi`_ |
| 270 | + * - Sphinx Docs |
| 271 | + - The project doumentation is `Sphinx`_ based |
| 272 | + * - nox |
| 273 | + - The projects automated tasks are executed using the `Nox`_ task runner |
| 274 | + * - toolbox-tasks |
| 275 | + - All nox tasks provided by the `python-toolbox`_ are available and fully functional |
| 276 | + * - toolbox-workflows |
| 277 | + - All :ref:`GitHub Workflows` provided by the `python-toolbox`_ are available and fully functional |
| 278 | +
|
| 279 | +.. _pyproject.toml: https://peps.python.org/pep-0621/ |
| 280 | +.. _Nox: https://nox.thea.codes/en/stable/ |
| 281 | +.. _Poetry: https://python-poetry.org/ |
| 282 | +.. _PyPi: https://pypi.org/ |
| 283 | +.. _Sphinx: https://www.sphinx-doc.org/en/master/ |
| 284 | +.. _python-toolbox: https://github.com/exasol/python-toolbox |
0 commit comments