Skip to content

V0.1.0 flp trt #216

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pschroedl wants to merge 12 commits into from
Closed

V0.1.0 flp trt #216

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
10578c4
refactor comfystream to export server and pipeline (#114)
eliteprox Apr 15, 2025
48ffba4
don't use GPU runner for comfyui-base workflow (#133)
pwilczynskiclearcode Apr 16, 2025
f184251
Revert "don't use GPU runner for comfyui-base workflow (#133)" (#134)
pwilczynskiclearcode Apr 16, 2025
fe4d733
exclude app.py from comfystream package (#131)
eliteprox Apr 16, 2025
9bddee0
pin comfyui version to 0.3.27 (#147)
eliteprox Apr 28, 2025
f0f7d7e
update ver to 0.1.0 (#151)
eliteprox Apr 28, 2025
69eb2c3
add ComfyUI-FasterLivePortrait, models
pschroedl Apr 23, 2025
a54fb01
whitespace
pschroedl Apr 23, 2025
53cb80e
chore: update FasterLivePortrait node branch
rickstaa Apr 25, 2025
8597a6e
fix: fix broken FasterLivePortrait branch name
rickstaa Apr 25, 2025
16fefab
include onnx for liveportrait trt, update build deps
pschroedl Apr 29, 2025
4c91b23
add liveportrait trt build deps
pschroedl Apr 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions configs/models.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,3 +73,77 @@ models:
url: "https://huggingface.co/Lykon/dreamshaper-8/resolve/main/text_encoder/model.fp16.safetensors"
path: "text_encoders/CLIPText/model.fp16.safetensors"
type: "text_encoder"

# JoyVASA models for ComfyUI-FasterLivePortrait
joyvasa_motion_generator:
name: "JoyVASA Motion Generator"
url: "https://huggingface.co/KwaiVGI/LivePortrait/resolve/main/joyvasa_models/motion_generator_hubert_chinese.pt"
path: "checkpoints/joyvasa_models/motion_generator_hubert_chinese.pt"
type: "torch"

joyvasa_audio_model:
name: "JoyVASA Hubert Chinese"
url: "https://huggingface.co/KwaiVGI/LivePortrait/resolve/main/joyvasa_models/chinese-hubert-base.zip"
path: "checkpoints/joyvasa_models/chinese-hubert-base.zip"
type: "archive"

joyvasa_motion_template:
name: "JoyVASA Motion Template"
url: "https://huggingface.co/KwaiVGI/LivePortrait/resolve/main/joyvasa_models/motion_template.pkl"
path: "checkpoints/joyvasa_models/motion_template.pkl"
type: "pickle"

# LivePortrait ONNX models - only necessary to build TRT engines
warping_spade:
name: "WarpingSpadeModel"
url: "https://huggingface.co/KwaiVGI/LivePortrait/resolve/main/liveportrait_onnx/warping_spade-fix.onnx"
path: "checkpoints/liveportrait_onnx/warping_spade-fix.onnx"
type: "onnx"

motion_extractor:
name: "MotionExtractorModel"
url: "https://huggingface.co/KwaiVGI/LivePortrait/resolve/main/liveportrait_onnx/motion_extractor.onnx"
path: "checkpoints/liveportrait_onnx/motion_extractor.onnx"
type: "onnx"

landmark:
name: "LandmarkModel"
url: "https://huggingface.co/KwaiVGI/LivePortrait/resolve/main/liveportrait_onnx/landmark.onnx"
path: "checkpoints/liveportrait_onnx/landmark.onnx"
type: "onnx"

face_analysis_retinaface:
name: "FaceAnalysisModel - RetinaFace"
url: "https://huggingface.co/KwaiVGI/LivePortrait/resolve/main/liveportrait_onnx/retinaface_det_static.onnx"
path: "checkpoints/liveportrait_onnx/retinaface_det_static.onnx"
type: "onnx"

face_analysis_2dpose:
name: "FaceAnalysisModel - 2DPose"
url: "https://huggingface.co/KwaiVGI/LivePortrait/resolve/main/liveportrait_onnx/face_2dpose_106_static.onnx"
path: "checkpoints/liveportrait_onnx/face_2dpose_106_static.onnx"
type: "onnx"

appearance_feature_extractor:
name: "AppearanceFeatureExtractorModel"
url: "https://huggingface.co/KwaiVGI/LivePortrait/resolve/main/liveportrait_onnx/appearance_feature_extractor.onnx"
path: "checkpoints/liveportrait_onnx/appearance_feature_extractor.onnx"
type: "onnx"

stitching:
name: "StitchingModel"
url: "https://huggingface.co/KwaiVGI/LivePortrait/resolve/main/liveportrait_onnx/stitching.onnx"
path: "checkpoints/liveportrait_onnx/stitching.onnx"
type: "onnx"

stitching_eye_retarget:
name: "StitchingModel (Eye Retargeting)"
url: "https://huggingface.co/KwaiVGI/LivePortrait/resolve/main/liveportrait_onnx/stitching_eye.onnx"
path: "checkpoints/liveportrait_onnx/stitching_eye.onnx"
type: "onnx"

stitching_lip_retarget:
name: "StitchingModel (Lip Retargeting)"
url: "https://huggingface.co/KwaiVGI/LivePortrait/resolve/main/liveportrait_onnx/stitching_lip.onnx"
path: "checkpoints/liveportrait_onnx/stitching_lip.onnx"
type: "onnx"
6 changes: 6 additions & 0 deletions configs/nodes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,12 @@ nodes:
branch: "feature/add-export-trt-args"
type: "tensorrt"

comfyui-fasterliveportrait:
name: "ComfyUI FasterLivePortrait"
url: "https://github.com/pschroedl/ComfyUI-FasterLivePortrait.git"
branch: "master"
type: "tensorrt"

# Ryan's nodes
comfyui-ryanontheinside:
name: "ComfyUI RyanOnTheInside"
Expand Down
32 changes: 27 additions & 5 deletions docker/Dockerfile.base
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,20 +7,27 @@ FROM "${BASE_IMAGE}"
ARG CONDA_VERSION \
PYTHON_VERSION

ENV DEBIAN_FRONTEND=noninteractive \
ENV TensorRT_ROOT=/opt/TensorRT-10.9.0.34 \
DEBIAN_FRONTEND=noninteractive \
CONDA_VERSION="${CONDA_VERSION}" \
PATH="/workspace/miniconda3/bin:${PATH}" \
PYTHON_VERSION="${PYTHON_VERSION}"

# System dependencies
RUN apt update && apt install -yqq \
RUN apt update && apt install -yqq --no-install-recommends \
git \
wget \
nano \
socat \
libsndfile1 \
build-essential llvm tk-dev && \
rm -rf /var/lib/apt/lists/*
build-essential \
llvm \
tk-dev \
cmake \
swig \
libprotobuf-dev \
protobuf-compiler \
&& rm -rf /var/lib/apt/lists/*

# Conda setup
RUN mkdir -p /workspace/comfystream && \
Expand All @@ -30,8 +37,23 @@ RUN mkdir -p /workspace/comfystream && \
conda create -n comfystream python="${PYTHON_VERSION}" ffmpeg=6 -c conda-forge -y && \
rm /tmp/miniconda.sh && echo 'export LD_LIBRARY_PATH=/workspace/miniconda3/envs/comfystream/lib:$LD_LIBRARY_PATH' >> ~/.bashrc

# TensorRT SDK
WORKDIR /opt
RUN wget --progress=dot:giga \
https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/10.9.0/tars/TensorRT-10.9.0.34.Linux.x86_64-gnu.cuda-12.8.tar.gz \
&& tar -xzf TensorRT-10.9.0.34.Linux.x86_64-gnu.cuda-12.8.tar.gz \
&& rm TensorRT-10.9.0.34.Linux.x86_64-gnu.cuda-12.8.tar.gz

# Link libraries and update linker cache
RUN echo "${TensorRT_ROOT}/lib" > /etc/ld.so.conf.d/tensorrt.conf \
&& ldconfig

# Install matching TensorRT Python bindings for CPython 3.11
RUN conda run -n comfystream pip install --no-cache-dir \
${TensorRT_ROOT}/python/tensorrt-10.9.0.34-cp311-none-linux_x86_64.whl

# Clone ComfyUI
RUN git clone https://github.com/comfyanonymous/ComfyUI.git /workspace/ComfyUI
RUN git clone --branch v0.3.27 --depth 1 https://github.com/comfyanonymous/ComfyUI.git /workspace/ComfyUI

# Copy only files needed for setup
COPY ./src/comfystream/scripts /workspace/comfystream/src/comfystream/scripts
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "comfystream"
description = "Build Live AI Video with ComfyUI"
version = "0.0.5"
version = "0.1.0"
license = { file = "LICENSE" }
dependencies = [
"asyncio",
Expand Down
8 changes: 3 additions & 5 deletions server/app.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,12 @@
import logging
import os
import sys

import torch

# Initialize CUDA before any other imports to prevent core dump.
if torch.cuda.is_available():
torch.cuda.init()


from aiohttp import web
from aiortc import (
MediaStreamTrack,
Expand All @@ -22,10 +20,10 @@
)
from aiortc.codecs import h264
from aiortc.rtcrtpsender import RTCRtpSender
from pipeline import Pipeline
from comfystream.pipeline import Pipeline
from twilio.rest import Client
from utils import patch_loop_datagram, add_prefix_to_app_routes, FPSMeter
from metrics import MetricsManager, StreamStatsManager
from comfystream.server.utils import patch_loop_datagram, add_prefix_to_app_routes, FPSMeter
from comfystream.server.metrics import MetricsManager, StreamStatsManager
import time

logger = logging.getLogger(__name__)
Expand Down
14 changes: 14 additions & 0 deletions src/comfystream/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
from .client import ComfyStreamClient
from .pipeline import Pipeline
from .server.utils import temporary_log_level
from .server.utils import FPSMeter
from .server.metrics import MetricsManager, StreamStatsManager

__all__ = [
'ComfyStreamClient',
'Pipeline',
'temporary_log_level',
'FPSMeter',
'MetricsManager',
'StreamStatsManager'
]
Loading