feature: Native support for src-layout projects #5537
Description
Feature request
Add native support for Python projects using the src-layout pattern, where the package code resides in a src/ subdirectory rather than at the project root. BentoML should automatically detect projects with pyproject.toml that specify [tool.setuptools.packages.find] where = ["src"] and install them as editable packages during the Docker build process.
Motivation
The src-layout is a [widely recommended best practice] in the Python packaging community because it:
- Prevents accidental imports from the development directory
- Ensures tests run against the installed package, not local files
- Provides better isolation between development and installed code
Currently, BentoML requires workarounds to support src-layout projects:
Problem: When using src-layout, the service module path is src/package_name/service.py, but BentoML includes it as src/src/package_name/**/*.py in the bento. At runtime, Python cannot import package_name because:
- The package isn't installed in site-packages
- Setting PYTHONPATH=/home/bentoml/bento/src/src conflicts with OpenTelemetry auto-instrumentation and other tools that modify PYTHONPATH
Current Workarounds:
- Custom Dockerfile templates with manual uv pip install -e commands
- PYTHONPATH manipulation (breaks with OTEL and other instrumentation)
- Restructuring projects to use flat layout (abandons best practices)
Other
build-ctx doesn't work as often you have some files to include outside of the src/ folder like .env or some artifacts.