Setup process (#2)

* Added image.py and associated files

* Add pre-commit-config.yaml

* Pre-commit corrections
- Removed unused variables
~ Modified line width to be 79 characters or less

* Modified typing, location of isatty() check
+ Added Union typing to clarify argument types for output files
~ Moved isatty() check to the run() method to cover instances where the stdout changes over the course of usage

* Bugfixes from prior commit

* Exceptions & Command Availability Check

+ Added a CommandNotFoundOnImageError exception that is raised when an Image attempts to run a command that does not exist on the docker image.
+ Added a method to Image that performs checks on a docker image to see which commands from a given list are present on the image.
+ Added pytest tests to check the above
+ Corrected how interactive calls are made on Image.run()
+ Added a pytest test to check that interactive Image.run() calls proceed correctly
+ Added .gitignore

* Image.tags str -> list

~ Image.tags modified to return a list of strings rather than a string representation of a list.
~ Modified associated pytests (tags and repr) to correctly reflect this change

* File structure reorganization

~ Moved package files into docker_cls folder, test files into test folder
+ added __init__.py to docker_cls folder
+ added conftest.py to control the pytest working directory and clean up testing files
~ renamed some dockerfiles to be more indicative of their use and test that the system can use dockerfiles not named Dockerfile

* PR Fixes

+ Added "Self" Type
+ Added return values to all methods
~ Modified some values to be keyword-only
~ Modified several argument types to be more idiomatic and expressive
~ Modified the value of the "network" command in build and run methods to be passed by argument instead of hardcoded
~ Changed parsing of the command argument from building and then splitting a string to constructing a list
~ Other small modifications

* _run_process method added

+ Added the _run_process() method which governs how the Image class as a whole activates commands. This function can print, return, and write to files all of the outputs of a given command, and can also input an arbitrary string to stdin for that command.
- Removed typing_extensions, changed to using typing.TypeVar
+ Added print_output arguments to run commands in order to allow them to print to the screen as they process.
~ Modified all command calling except for _inspect() to use _run_process()
~ Various small changes
~ Updated test_image run() tests to account for the fact that some stderr messages also output

* Documentation update

+ Added to the docstring of run() to describe the CommandNotFoundOnImageError
- Removed an extra testing file that was not intended for inclusion in the prior commit

* Build overrides, Run redefined

~ build() redefined from build_from_dockerfile and build_from_string using overrides to allow users to define whether they want to build using a dockerfile string without having to call a separate function - this will allow for maintenance of only one method rather than two
~ run() modified to use subprocess.run() and flexibly handle the need for returning strings or writing to files
~ Tests modified and updated to fit new use cases

* Update image.py

Removed an accidental debug print statement from image.py

* No-cache option added to image.build() methods

+ Added no-cache to image.build() methods to enable testing of image building in a more complete manner

* PR updates

+ Added a ValueError check in get_image_id for name_or_id
~ Streamlined and/or clarified many docstrings in image.py and image_test.py
~ Renamed alpine_functional and alpine_broken to have ".dockerfile" extension
~ Modified methods with file outputs to accept stout and stderr arguments separately
~ Modified Image.build() to raise a ValueError if both dockerfile and dockerfile_string commands are given.
~ Modified image.build() to default to building a dockerfile in the context root folder if neither dockerfile_string nor dockerfile are given.
~ Moved the bash warning to the run command.
~ Updated pytest tests to fit new interfaces

* Additional docstring changes

~ Added some further simplifications to the docstrings.
~ Modified Image.build such that `context` is optional and defaults to use "."

* Update image.py

~ Listed optional arguments as optional in docstrings

* Update image.py

* Update image.py

* Update image.py

* Update docker_cli/image.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Update docker_cli/image.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Update docker_cli/image.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Update docker_cli/image.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Update test/test_image.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Update to use checkout from  docker_build_tree

~ Moved exceptions.py and image.py to private files
~ Moved test_image_id fixture to conftest.py
~ Modified pytest_sessionstart to allow pytests from anywhere
~ Added a tox.ini file to handle additional changes
~ Added has_command to Image, adjusted check command to use it
~ Added drop-in to Image
~ Changes to documentation

* Test upgrades

- Removed pytest_sessionfinish from conftest.py
~ Changed test_image_id to image_id
~ Used yield to have image_id clean up after itself
~ Used try/finally blocks to have build tests clean up after themselves
~ Used temporary files instead of filenames
~ Improved some documentation in _image.py

* More PR fixes

~ Changed "cat dockerfile" lines to use Path.read_text()
~ Changed Image.has_command to use try/except/else

* Removed extra "Raises" commentary in docstrings

* Fixed import issue

* Update conftest.py

Added comments to explain the purpose of the lines in pytest_sessionstart

* Exception improvements

~ Changed CommandNotFoundError to subclass from Exception instead of CalledProcessError
~ Specified when ImageNotFoundError should be raised more clearly
~ Fixed some tests and documentation in test_image.py

* Changed "test" image to "isce3_pytest_temp"

~ Changed other temporary images to "isce3_pytest_temp_2"
~ Removed some hanging commas in test parameters.

* Proper capitalization of "Dockerfile"

* Added image_tag fixture

~ Added an image_tag fixture to simplify modifying the test tags when needed.

* Update docker_cli/_exceptions.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Update docker_cli/_image.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Update test/test_image.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Update docker_cli/_image.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Update docker_cli/_image.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Update docker_cli/_image.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Update docker_cli/_image.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Update docker_cli/_image.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Update docker_cli/_image.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Update docker_cli/_image.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Update docker_cli/_image.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Update docker_cli/_image.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Update docker_cli/_image.py

Co-authored-by: Geoffrey Gunter <[email protected]>

* Exception overloads, dropin change, documentation

~ Exceptions modified to use method overloads for __init__
~ Exceptions modified to call super.__init__ instead of defining __str__
~ image.drop_in modified to just call image.run()
~ Some documentation changes and shuffling of arguments

* Fixed DockerBuildError raises in Image.build

* Added utils.py

+ added utils.py file
~ Moved image_tag, image_id fixtures to utils.py
~ Updated tests and pre-commit hooks for importing fixtures

* More robust fixture usage, use of test classes and marks

+ Began using marks to enable later test segregation
+ Began using test classes for simpler marking
+ Simplified flake8 per-file-ignores in tox.ini
~ Renamed utils.py to fixtures.py
(utils.py will later be used for helper functions)

* Image fixture scope, parallelism

+ Added scope to fixtures in fixtures.py
+ Added test/utils.py, with a universal function for removing images to avoid syntax mistakes
+ Added docker_cli/utils.py, which has a codename generator to allow parallel tests to run without race conditions

* Dynamic scope

+ Added a dynamic scope function for pytest-xdist rescoping

* A docstring for determine_scope

* PR Suggestions, pytest-cov compatibility

* Dedent escape character corrected

* minor

* Added some documentation to determine_scope

* fixed a rebase bug

* Command line setup phase and auxiliary commands

+ Added a command line interface for the module
+ Added init, Micromamba, and CUDA build pathway
+ Added drop-in, image removal, and lockfile generation
+ Added associated tests

* Removed "Dockerfile"

+ Pulled docker_cli/__init__.py, test/fixtures.py, and test/utils.py from a more current build
~ Modified the Image.build() overloads to appease the pernicious and angry mypy gods
- Removed the Dockerfile data structure from the codebase
- Removed some unnecessary functions from _utils.py

* Removed NotImplementedErrors

~ Also, changed some indentations in _shell_cmds.py to be more consistent

* Added docker_cli setup env add command

+ Added functionality to add a handful of given packages to an image's environment.

* "docker" to "Docker"

* black formatting

* Ran pre-commit with black (not sure why it didn't run automatically)

* PR updates to BindMount, exceptions, some CLI

* Removed json from gitignore

* Set subcommand arguments to required

~ Changed subcommand checking from being handled in main to being handled by argparse
- Removed insufficient_subcommands_message
~ Small change to __init__.py

* Added a comment describing the reason for the help formatter

* Commented on the options for different specfile file formats.

* Removed case insensitivity for subcommands

* Switched to using ArgumentDefaultsHelpFormatter

* Set up the basis for unique defaults in the setup conda runtime and dev commands

* Mamba build fix, environment lockfiles

+ Added lockfiles under env_files folder
~ Changed env file defaults on the CLI
~ moved in fixes in _docker_mamba.py that exist in downstream branches

* Default file paths relative to module and not pwd

* Set the module location to be one file deeper

* Intermediate work on CLI refactor

* PR responses & fixes

* tiny docs change

* additional doc fixes

* _url_reader_check -> two functions

~ Split _url_reader_check into two functions
~ Changed some functions unnecessarily marked private to be marked public

* PR changes to package_manager_check and url_reader_check

* Modified random string generation to work per PR recommendation

* CLI submodule

* Made command script naming convention more consistent

* Some command PR fixes

* Capitalized Dockerfile and CUDA

* docker_cuda PR fixes and backport of automatic test image deletion

+ Added automatic deletion of test images from later work

* Removed unnecessary error logic from remove command.

* split image_command_check into a checker and a temporary image generator.

* Documenting temp_image reasoning

* Switched from some list based implementations of dockerfiles to string-based.

* Removed unnecessary CUDA version environment variable

* Removed check_command_availability

* (forgot to commit this in last commit)

* Switched remove quiet argument with verbose

* Removed cli_command.py

* PR fixes

* Removed non-functioning "conda add" command

* Changed nomenclature from "spec file" to "requirements file."

~ Also changed some references to os.PathLike to Path

* docker_mamba PR fixes

* defaults.py PR fixes

* _utils.py PR fixes

* Root user fix in docker_cuda

+ Also modified the setup_conda_dev and setup_conda_runtime methods to enforce requirements file paths as relative to the cwd

* Contexts for setup_conda_dev, setup_conda_runtime

+ Added context argument and input sanitization for requirement file paths

* WIP fixing CUDA install with root issue

* updating lockfiles

* Fixed wrong default dev reqs file on setup all

* Fixed context issue with requirement files in conda image setup

* Documentation & path resolution fixes

~ Fixed some inaccurate docstrings.
- Removed some unnecessary `resolve()` calls on absolute paths.

---------

Co-authored-by: Tyler Hudson <[email protected]>
Co-authored-by: Geoffrey Gunter <[email protected]>

Author: Tyler-g-hudson

.pre-commit-config.yaml

@@ -18,11 +18,11 @@ repos:
       - id: mixed-line-ending
       - id: trailing-whitespace
 
-  - repo: https://github.com/pre-commit/mirrors-autopep8
-    rev: "v2.0.1"
+  - repo: https://github.com/psf/black
+    rev: 23.3.0
     hooks:
-      - id: autopep8
-
+      - id: black
+        language_version: python3.10
   - repo: https://github.com/PyCQA/flake8
     rev: "6.0.0"
     hooks:
@@ -32,6 +32,7 @@ repos:
     rev: "5.12.0"
     hooks:
       - id: isort
+        args: ["--profile", "black"]
 
   - repo: https://github.com/pre-commit/mirrors-mypy
     rev: "v1.0.0"

docker_cli/__init__.py

@@ -1,3 +1,11 @@
-from ._exceptions import (CommandNotFoundError, DockerBuildError,
-                          ImageNotFoundError)
+from ._bind_mount import BindMount
+from ._docker_cuda import CUDADockerfileGenerator, get_cuda_dockerfile_generator
+from ._docker_mamba import mamba_install_dockerfile
+from ._exceptions import CommandNotFoundError, DockerBuildError, ImageNotFoundError
 from ._image import Image, get_image_id
+from ._package_manager import (
+    PackageManager,
+    get_package_manager,
+    get_supported_package_managers,
+)
+from ._url_reader import URLReader, get_supported_url_readers, get_url_reader

docker_cli/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+main()

docker_cli/_bind_mount.py

@@ -0,0 +1,34 @@
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass
+
+
+@dataclass
+class BindMount:
+    """A Docker bind mount."""
+
+    src: str | os.PathLike[str]
+    """
+    str or os.PathLike[str] :
+        The path to the file or directory on the Docker daemon host.
+    """
+
+    dst: str | os.PathLike[str]
+    """
+    str or os.PathLike[str] :
+        The path where the file or directory is mounted in the container.
+    """
+
+    permissions: str = "rw"
+    """str : The bind mount permissions -- 'ro' for readonly, 'rw' for read/write."""
+
+    def __post_init__(self):
+        if self.permissions not in ("ro", "rw"):
+            raise ValueError(
+                f"permissions must be 'ro' or 'rw', got {self.permissions!r}"
+            )
+
+    def mount_string(self) -> str:
+        """Returns a string describing the mount."""
+        return f"{self.src}:{self.dst}:{self.permissions}"