Skip to content

Commit 68826ac

Browse files
authored
ci/cd: add nightly build and CI for flashinfer-python, flashinfer-jit-cache, flashinfer-cubin (#1872)
<!-- .github/pull_request_template.md --> ## 📌 Description Duplicate of #1867, created from flashinfer/nightly to get write permission. ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. -->
1 parent 9c6c265 commit 68826ac

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

54 files changed

+1667
-505
lines changed

.github/workflows/nightly-release.yml

Lines changed: 422 additions & 0 deletions
Large diffs are not rendered by default.

README.md

Lines changed: 50 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -42,50 +42,86 @@ FlashInfer supports PyTorch, TVM and C++ (header-only) APIs, and can be easily i
4242

4343
Using our PyTorch API is the easiest way to get started:
4444

45-
### Install from PIP
45+
### Install from PyPI
4646

47-
FlashInfer is available as a Python package for Linux on PyPI. You can install it with the following command:
47+
FlashInfer is available as a Python package for Linux. Install the core package with:
4848

4949
```bash
5050
pip install flashinfer-python
5151
```
5252

53+
**Package Options:**
54+
- **flashinfer-python**: Core package that compiles/downloads kernels on first use
55+
- **flashinfer-cubin**: Pre-compiled kernel binaries for all supported GPU architectures
56+
- **flashinfer-jit-cache**: Pre-built kernel cache for specific CUDA versions
57+
58+
**For faster initialization and offline usage**, install the optional packages to have most kernels pre-compiled:
59+
```bash
60+
pip install flashinfer-python flashinfer-cubin
61+
pip install flashinfer-jit-cache --index-url https://flashinfer.ai/whl/
62+
```
63+
64+
This eliminates compilation and downloading overhead at runtime.
65+
5366
### Install from Source
5467

55-
Alternatively, build FlashInfer from source:
68+
Build the core package from source:
5669

5770
```bash
5871
git clone https://github.com/flashinfer-ai/flashinfer.git --recursive
5972
cd flashinfer
6073
python -m pip install -v .
74+
```
6175

62-
# for development & contribution, install in editable mode
76+
**For development**, install in editable mode:
77+
```bash
6378
python -m pip install --no-build-isolation -e . -v
6479
```
6580

66-
`flashinfer-python` is a source-only package and by default it will JIT compile/download kernels on-the-fly.
67-
For fully offline deployment, we also provide two additional packages `flashinfer-jit-cache` and `flashinfer-cubin`, to pre-compile and download cubins ahead-of-time.
68-
69-
#### flashinfer-cubin
81+
**Build optional packages:**
7082

71-
To build `flashinfer-cubin` package from source:
83+
`flashinfer-cubin`:
7284
```bash
7385
cd flashinfer-cubin
7486
python -m build --no-isolation --wheel
7587
python -m pip install dist/*.whl
7688
```
7789

78-
#### flashinfer-jit-cache
79-
80-
To build `flashinfer-jit-cache` package from source:
90+
`flashinfer-jit-cache` (customize `FLASHINFER_CUDA_ARCH_LIST` for your target GPUs):
8191
```bash
82-
export FLASHINFER_CUDA_ARCH_LIST="7.5 8.0 8.9 10.0a 10.3a 12.0a" # user can shrink the list to specific architectures
92+
export FLASHINFER_CUDA_ARCH_LIST="7.5 8.0 8.9 10.0a 10.3a 12.0a"
8393
cd flashinfer-jit-cache
8494
python -m build --no-isolation --wheel
8595
python -m pip install dist/*.whl
8696
```
8797

88-
For more details, refer to the [Install from Source documentation](https://docs.flashinfer.ai/installation.html#install-from-source).
98+
For more details, see the [Install from Source documentation](https://docs.flashinfer.ai/installation.html#install-from-source).
99+
100+
### Install Nightly Build
101+
102+
Nightly builds are available for testing the latest features:
103+
104+
```bash
105+
# Core and cubin packages
106+
pip install -U --pre flashinfer-python --extra-index-url https://flashinfer.ai/whl/nightly/
107+
pip install -U --pre flashinfer-cubin --index-url https://flashinfer.ai/whl/nightly/
108+
# JIT cache package (replace cu129 with your CUDA version: cu128, cu129, or cu130)
109+
pip install -U --pre flashinfer-jit-cache --index-url https://flashinfer.ai/whl/nightly/cu129
110+
```
111+
112+
### Verify Installation
113+
114+
After installation, verify that FlashInfer is correctly installed and configured:
115+
116+
```bash
117+
flashinfer show-config
118+
```
119+
120+
This command displays:
121+
- FlashInfer version and installed packages (flashinfer-python, flashinfer-cubin, flashinfer-jit-cache)
122+
- PyTorch and CUDA version information
123+
- Environment variables and artifact paths
124+
- Downloaded cubin status and module compilation status
89125

90126
### Trying it out
91127

build_backend.py

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
"""
2+
Copyright (c) 2023 by FlashInfer team.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
"""
16+
17+
import os
18+
import shutil
19+
from pathlib import Path
20+
21+
from setuptools import build_meta as orig
22+
from build_utils import get_git_version
23+
24+
_root = Path(__file__).parent.resolve()
25+
_data_dir = _root / "flashinfer" / "data"
26+
27+
28+
def _create_build_metadata():
29+
"""Create build metadata file with version information."""
30+
version_file = _root / "version.txt"
31+
if version_file.exists():
32+
with open(version_file, "r") as f:
33+
version = f.read().strip()
34+
else:
35+
version = "0.0.0+unknown"
36+
37+
# Add dev suffix if specified
38+
dev_suffix = os.environ.get("FLASHINFER_DEV_RELEASE_SUFFIX", "")
39+
if dev_suffix:
40+
version = f"{version}.dev{dev_suffix}"
41+
42+
# Get git version
43+
git_version = get_git_version(cwd=_root)
44+
45+
# Create build metadata in the source tree
46+
package_dir = Path(__file__).parent / "flashinfer"
47+
build_meta_file = package_dir / "_build_meta.py"
48+
49+
# Check if we're in a git repository
50+
git_dir = Path(__file__).parent / ".git"
51+
in_git_repo = git_dir.exists()
52+
53+
# If file exists and not in git repo (installing from sdist), keep existing file
54+
if build_meta_file.exists() and not in_git_repo:
55+
print("Build metadata file already exists (not in git repo), keeping it")
56+
return version
57+
58+
# In git repo (editable) or file doesn't exist, create/update it
59+
with open(build_meta_file, "w") as f:
60+
f.write('"""Build metadata for flashinfer package."""\n')
61+
f.write(f'__version__ = "{version}"\n')
62+
f.write(f'__git_version__ = "{git_version}"\n')
63+
64+
print(f"Created build metadata file with version {version}")
65+
return version
66+
67+
68+
# Create build metadata as soon as this module is imported
69+
_create_build_metadata()
70+
71+
72+
def write_if_different(path: Path, content: str) -> None:
73+
if path.exists() and path.read_text() == content:
74+
return
75+
path.parent.mkdir(parents=True, exist_ok=True)
76+
path.write_text(content)
77+
78+
79+
def _create_data_dir(use_symlinks=True):
80+
_data_dir.mkdir(parents=True, exist_ok=True)
81+
82+
def ln(source: str, target: str) -> None:
83+
src = _root / source
84+
dst = _data_dir / target
85+
if dst.exists():
86+
if dst.is_symlink():
87+
dst.unlink()
88+
elif dst.is_dir():
89+
shutil.rmtree(dst)
90+
else:
91+
dst.unlink()
92+
93+
if use_symlinks:
94+
dst.symlink_to(src, target_is_directory=True)
95+
else:
96+
# For wheel/sdist, copy actual files instead of symlinks
97+
if src.exists():
98+
shutil.copytree(src, dst, symlinks=False, dirs_exist_ok=True)
99+
100+
ln("3rdparty/cutlass", "cutlass")
101+
ln("3rdparty/spdlog", "spdlog")
102+
ln("csrc", "csrc")
103+
ln("include", "include")
104+
105+
106+
def _prepare_for_wheel():
107+
# For wheel, copy actual files instead of symlinks so they are included in the wheel
108+
if _data_dir.exists():
109+
shutil.rmtree(_data_dir)
110+
_create_data_dir(use_symlinks=False)
111+
112+
113+
def _prepare_for_editable():
114+
# For editable install, use symlinks so changes are reflected immediately
115+
if _data_dir.exists():
116+
shutil.rmtree(_data_dir)
117+
_create_data_dir(use_symlinks=True)
118+
119+
120+
def _prepare_for_sdist():
121+
# For sdist, copy actual files instead of symlinks so they are included in the tarball
122+
if _data_dir.exists():
123+
shutil.rmtree(_data_dir)
124+
_create_data_dir(use_symlinks=False)
125+
126+
127+
def get_requires_for_build_wheel(config_settings=None):
128+
_prepare_for_wheel()
129+
return []
130+
131+
132+
def get_requires_for_build_sdist(config_settings=None):
133+
_prepare_for_sdist()
134+
return []
135+
136+
137+
def get_requires_for_build_editable(config_settings=None):
138+
_prepare_for_editable()
139+
return []
140+
141+
142+
def prepare_metadata_for_build_wheel(metadata_directory, config_settings=None):
143+
_prepare_for_wheel()
144+
return orig.prepare_metadata_for_build_wheel(metadata_directory, config_settings)
145+
146+
147+
def prepare_metadata_for_build_editable(metadata_directory, config_settings=None):
148+
_prepare_for_editable()
149+
return orig.prepare_metadata_for_build_editable(metadata_directory, config_settings)
150+
151+
152+
def build_editable(wheel_directory, config_settings=None, metadata_directory=None):
153+
_prepare_for_editable()
154+
return orig.build_editable(wheel_directory, config_settings, metadata_directory)
155+
156+
157+
def build_sdist(sdist_directory, config_settings=None):
158+
_prepare_for_sdist()
159+
return orig.build_sdist(sdist_directory, config_settings)
160+
161+
162+
def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
163+
_prepare_for_wheel()
164+
return orig.build_wheel(wheel_directory, config_settings, metadata_directory)

build_utils.py

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
"""
2+
Copyright (c) 2025 by FlashInfer team.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
"""
16+
17+
"""Shared build utilities for flashinfer packages."""
18+
19+
import subprocess
20+
from pathlib import Path
21+
from typing import Optional
22+
23+
24+
def get_git_version(cwd: Optional[Path] = None) -> str:
25+
"""
26+
Get git commit hash.
27+
28+
Args:
29+
cwd: Working directory for git command. If None, uses current directory.
30+
31+
Returns:
32+
Git commit hash or "unknown" if git is not available.
33+
"""
34+
try:
35+
git_version = (
36+
subprocess.check_output(
37+
["git", "rev-parse", "HEAD"],
38+
cwd=cwd,
39+
stderr=subprocess.DEVNULL,
40+
)
41+
.decode("ascii")
42+
.strip()
43+
)
44+
return git_version
45+
except Exception:
46+
return "unknown"

custom_backend.py

Lines changed: 0 additions & 80 deletions
This file was deleted.

0 commit comments

Comments
 (0)