Expand ruff configurations to enforce comprehensive code quality standards (#23)

## Summary

Enhances our `ruff` configurations to enforce comprehensive standards
across various aspects of code quality, including style, formatting,
structure, complexity, security, bug prevention, and documentation.

## Details

### Changes Implemented:

1. **Code Style and Formatting:**
- Enforced PEP 8 conventions, including spacing, indentation, and line
length.
    - Included `isort` for consistent import sorting.
- Added rules for trailing commas, commented-out code, import
conventions, naming conventions, and quote usage.
2. **Code Structure and Complexity:**
    - Simplified complex expressions and improved readability.
    - Applied best practices for list, set, and dict comprehensions.
    - Checked for overly complex code using cyclomatic complexity.
3. **Code Security and Bug Prevention:**
- Detected unused function arguments, boolean traps, and implicit string
concatenations.
    - Prevented common programming mistakes and potential bugs.
    - Enforced correct exception handling and detected security issues.
4. **Code Documentation:**
    - Ensured adherence to documentation standards.
    - Detected unresolved FIXMEs and other temporary comments.

### Files Modified:

- `.pre-commit-config.yaml`: Updated configurations to include new ruff
rules and standards.
- `pyproject.toml`: Extended ruff linting rules to cover additional
checks for various code quality aspects.
- Source and test files: Applied necessary changes to comply with the
new ruff configurations, including refactoring code, updating
documentation, and adding comments.

### Test Plan:

- **Automated Testing:**
    - Automated tests updated to run the latest changes
- **Manual Testing:**
- Manually reviewed changes to ensure they align with the updated
linting standards.
    - Manually ran tests and checks

Author: markurtz

.pre-commit-config.yaml

@@ -20,10 +20,10 @@ repos:
         datasets,
         loguru,
         numpy,
+        openai,
         pydantic,
         pydantic_settings,
         pyyaml,
-        openai,
         requests,
         transformers,
 

docs/__init__.py

@@ -99,7 +99,80 @@ ignore_missing_imports=true
 [tool.ruff]
 line-length = 88
 exclude = ["build", "dist", "env", ".venv"]
-lint.select = ["E", "F", "W"]
+lint.ignore = [
+    "PLR0913",
+    "TCH001",
+    "COM812",
+    "ISC001",
+    "TCH002",
+]
+lint.select = [
+    # Rules reference: https://docs.astral.sh/ruff/rules/
+
+    # Code Style / Formatting
+    "E", # pycodestyle: checks adherence to PEP 8 conventions including spacing, indentation, and line length
+    "W", # pycodestyle: checks adherence to PEP 8 conventions including spacing, indentation, and line length
+    "A", # flake8-builtins: prevents shadowing of Python built-in names
+    "C", # Convention: ensures code adheres to specific style and formatting conventions
+    "COM", # flake8-commas: enforces the correct use of trailing commas
+    "ERA", # eradicate: detects commented-out code that should be removed
+    "I", # isort: ensures imports are sorted in a consistent manner
+    "ICN", # flake8-import-conventions: enforces import conventions for better readability
+    "N", # pep8-naming: enforces PEP 8 naming conventions for classes, functions, and variables
+    "NPY", # NumPy: enforces best practices for using the NumPy library
+    "PD", # pandas-vet: enforces best practices for using the pandas library
+    "PT", # flake8-pytest-style: enforces best practices and style conventions for pytest tests
+    "PTH", # flake8-use-pathlib: encourages the use of pathlib over os.path for file system operations
+    "Q", # flake8-quotes: enforces consistent use of single or double quotes
+    "TCH", # flake8-type-checking: enforces type checking practices and standards
+    "TID", # flake8-tidy-imports: enforces tidy and well-organized imports
+
+    # Code Structure / Complexity
+    "C4", # flake8-comprehensions: improves readability and performance of list, set, and dict comprehensions
+    "C90", # mccabe: checks for overly complex code using cyclomatic complexity
+    "FBT", # flake8-boolean-trap: prevents the use of boolean traps in function arguments and calls
+    "ISC", # flake8-implicit-str-concat: prevents implicit string concatenation
+    "PIE", # flake8-pie: identifies and corrects common code inefficiencies and mistakes
+    "R", # Refactor: suggests improvements to code structure and readability
+    "SIM", # flake8-simplify: simplifies complex expressions and improves code readability
+
+    # Code Security / Bug Prevention
+    "ARG", # flake8-unused-arguments: detects unused function and method arguments
+    "ASYNC", # flake8-async: identifies incorrect or inefficient usage patterns in asynchronous code
+    "B", # flake8-bugbear: detects common programming mistakes and potential bugs
+    "BLE", # flake8-blind-except: prevents blind exceptions that catch all exceptions without handling
+    "E", # Error: detects and reports errors in the code
+    "F", # Pyflakes: detects unused imports, shadowed imports, undefined variables, and various formatting errors in string operations
+    "INP", # flake8-no-pep420: prevents implicit namespace packages by requiring __init__.py
+    "PGH", # pygrep-hooks: detects deprecated and dangerous code patterns
+    "PL", # Pylint: comprehensive source code analyzer for enforcing coding standards and detecting errors
+    "RSE", # flake8-raise: ensures exceptions are raised correctly
+    "S", # flake8-bandit: detects security issues and vulnerabilities in the code
+    "SLF", # flake8-self: prevents incorrect usage of the self argument in class methods
+    "T10", # flake8-debugger: detects the presence of debugging tools such as pdb
+    "T20", # flake8-print: detects print statements left in the code
+    "UP", # pyupgrade: automatically upgrades syntax for newer versions of Python
+    "W", # Warning: provides warnings about potential issues in the code
+    "YTT", # flake8-2020: identifies code that will break with future Python releases
+
+    # Code Documentation
+    "FIX", # flake8-fixme: detects FIXMEs and other temporary comments that should be resolved
+]
+
+[tool.ruff.lint.extend-per-file-ignores]
+"tests/**/*.py" = [
+    "S101", # asserts allowed in tests
+    "ARG", # Unused function args allowed in tests
+    "FBT", # Booleans as positional arguments in tests, e.g. via @pytest.mark.parametrize()
+    "PLR2004", # Magic value used in comparison
+    "TCH002", # No import only type checking in tests
+    "SLF001", # enable private member access in tests
+    "S105", # allow hardcoded passwords in tests
+    "S311", # allow standard pseudo-random generators in tests
+    "PT011", # allow generic exceptions in tests
+    "N806", # allow uppercase variable names in tests
+    "PGH003", # allow general ignores in tests
+]
 
 
 [tool.pytest.ini_options]

pyproject.toml

@@ -15,7 +15,6 @@ class LoggingSettings(BaseModel):
 
 
 class OpenAISettings(BaseModel):
-
     # OpenAI API key.
     api_key: str = "invalid"
 

src/config/__init__.py

@@ -1,3 +1,4 @@
+import functools
 from abc import ABC, abstractmethod
 from dataclasses import dataclass
 from enum import Enum
@@ -79,6 +80,16 @@ def create(cls, backend_type: BackendEngine, **kwargs) -> "Backend":
 
         return Backend._registry[backend_type](**kwargs)
 
+    @property
+    def default_model(self) -> str:
+        """
+        Get the default model for the backend.
+
+        :return: The default model.
+        :rtype: str
+        """
+        return _cachable_default_model(self)
+
     def submit(self, request: TextGenerationRequest) -> TextGenerationResult:
         """
         Submit a result request and populate the BenchmarkResult.
@@ -92,7 +103,7 @@ def submit(self, request: TextGenerationRequest) -> TextGenerationResult:
         logger.info(f"Submitting request with prompt: {request.prompt}")
 
         result = TextGenerationResult(
-            request=TextGenerationRequest(prompt=request.prompt)
+            request=TextGenerationRequest(prompt=request.prompt),
         )
         result.start(request.prompt)
 
@@ -111,7 +122,8 @@ def submit(self, request: TextGenerationRequest) -> TextGenerationResult:
 
     @abstractmethod
     def make_request(
-        self, request: TextGenerationRequest
+        self,
+        request: TextGenerationRequest,
     ) -> Iterator[GenerativeResponse]:
         """
         Abstract method to make a request to the backend.
@@ -121,8 +133,7 @@ def make_request(
         :return: An iterator over the generative responses.
         :rtype: Iterator[GenerativeResponse]
         """
-
-        pass
+        raise NotImplementedError
 
     @abstractmethod
     def available_models(self) -> List[str]:
@@ -132,20 +143,7 @@ def available_models(self) -> List[str]:
         :return: A list of available models.
         :rtype: List[str]
         """
-
-        pass
-
-    @property
-    @abstractmethod
-    def default_model(self) -> str:
-        """
-        Abstract method to get the default model for the backend.
-
-        :return: The default model.
-        :rtype: str
-        """
-
-        pass
+        raise NotImplementedError
 
     @abstractmethod
     def model_tokenizer(self, model: str) -> Optional[str]:
@@ -157,5 +155,14 @@ def model_tokenizer(self, model: str) -> Optional[str]:
         :return: The tokenizer for the model, or None if it cannot be created.
         :rtype: Optional[str]
         """
+        raise NotImplementedError
+
+
+@functools.lru_cache(maxsize=1)
+def _cachable_default_model(backend: Backend) -> str:
+    if models := backend.available_models():
+        logger.debug(f"Default model: {models[0]}")
+        return models[0]
 
-        pass
+    logger.error("No models available.")
+    raise ValueError("No models available.")

src/guidellm/backend/base.py

@@ -1,12 +1,12 @@
 from typing import Any, Dict, Generator, List, Optional
 
 import openai
+from config import settings
 from loguru import logger
 from openai import OpenAI, Stream
 from openai.types import Completion
 from transformers import AutoTokenizer
 
-from config import settings
 from guidellm.backend import Backend, BackendEngine, GenerativeResponse
 from guidellm.core import TextGenerationRequest
 
@@ -53,7 +53,7 @@ def __init__(
             raise ValueError(
                 "`GUIDELLM__OPENAI__API_KEY` environment variable "
                 "or --openai-api-key CLI parameter "
-                "must be specify for the OpenAI backend"
+                "must be specify for the OpenAI backend",
             )
 
         if target is not None:
@@ -65,16 +65,17 @@ def __init__(
         else:
             raise ValueError(
                 "`GUIDELLM__OPENAI__BASE_URL` environment variable "
-                "or --target CLI parameter must be specify for the OpenAI backend."
+                "or --target CLI parameter must be specified for the OpenAI backend."
             )
 
         self.openai_client = OpenAI(api_key=_api_key, base_url=base_url)
         self.model = model or self.default_model
 
-        logger.info(f"OpenAI {self.model} Backend listening on {target}")
+        logger.info("OpenAI {} Backend listening on {}", self.model, target)
 
     def make_request(
-        self, request: TextGenerationRequest
+        self,
+        request: TextGenerationRequest,
     ) -> Generator[GenerativeResponse, None, None]:
         """
         Make a request to the OpenAI backend.
@@ -143,22 +144,6 @@ def available_models(self) -> List[str]:
             logger.info(f"Available models: {models}")
             return models
 
-    @property
-    def default_model(self) -> str:
-        """
-        Get the default model for the backend.
-
-        :return: The default model.
-        :rtype: str
-        """
-
-        if models := self.available_models():
-            logger.info(f"Default model: {models[0]}")
-            return models[0]
-
-        logger.error("No models available.")
-        raise ValueError("No models available.")
-
     def model_tokenizer(self, model: str) -> Optional[Any]:
         """
         Get the tokenizer for a model.
@@ -172,8 +157,8 @@ def model_tokenizer(self, model: str) -> Optional[Any]:
             tokenizer = AutoTokenizer.from_pretrained(model)
             logger.info(f"Tokenizer created for model: {model}")
             return tokenizer
-        except Exception as e:
-            logger.warning(f"Could not create tokenizer for model {model}: {e}")
+        except Exception as err:  # noqa: BLE001
+            logger.warning(f"Could not create tokenizer for model {model}: {err}")
             return None
 
     def _token_count(self, text: str) -> int:

src/guidellm/backend/openai.py

@@ -16,7 +16,8 @@ class Distribution(Serializable):
     """
 
     data: Sequence[float] = Field(
-        default_factory=list, description="The data points of the distribution."
+        default_factory=list,
+        description="The data points of the distribution.",
     )
 
     def __str__(self):
@@ -160,7 +161,7 @@ def describe(self) -> dict:
             "std_deviation": self.std_deviation,
             "percentile_indices": [10, 20, 30, 40, 50, 60, 70, 80, 90, 95, 99],
             "percentile_values": self.percentiles(
-                [10, 20, 30, 40, 50, 60, 70, 80, 90, 95, 99]
+                [10, 20, 30, 40, 50, 60, 70, 80, 90, 95, 99],
             ),
             "min": self.min,
             "max": self.max,

src/guidellm/core/distribution.py

@@ -17,10 +17,12 @@ class TextGenerationRequest(Serializable):
     )
     prompt: str = Field(description="The input prompt for the text generation.")
     prompt_token_count: Optional[int] = Field(
-        default=None, description="The number of tokens in the input prompt."
+        default=None,
+        description="The number of tokens in the input prompt.",
     )
     generate_token_count: Optional[int] = Field(
-        default=None, description="The number of tokens to generate."
+        default=None,
+        description="The number of tokens to generate.",
     )
     params: Dict[str, Any] = Field(
         default_factory=dict,