Mcv binary cache

[maryamtahhan]

Enable vllm binary cache support for MCV

fixes: #147

[maryamtahhan]

TODO - add torch_inductor dir

[maryamtahhan]

No precache

(EngineCore_DP0 pid=22) INFO 02-23 01:35:37 [backends.py:812] Using cache directory: /root/.cache/vllm/torch_compile_cache/8d0a361fbc/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=22) INFO 02-23 01:35:37 [backends.py:872] Dynamo bytecode transform time: 28.30 s
(EngineCore_DP0 pid=22) [rank0]:W0223 01:35:45.613000 22 torch/_inductor/utils.py:1613] Not enough SMs to use max_autotune_gemm mode
(EngineCore_DP0 pid=22) INFO 02-23 01:35:55 [backends.py:302] Cache the graph of compile range (1, 2048) for later use
(EngineCore_DP0 pid=22) INFO 02-23 01:36:01 [backends.py:319] Compiling a graph for compile range (1, 2048) takes 18.20 s
(EngineCore_DP0 pid=22) INFO 02-23 01:36:01 [monitor.py:34] torch.compile takes 46.50 s in total

with pre-cache:

(EngineCore_DP0 pid=22) INFO 02-23 03:12:47 [backends.py:812] Using cache directory: /root/.cache/vllm/torch_compile_cache/8d0a361fbc/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=22) INFO 02-23 03:12:47 [backends.py:872] Dynamo bytecode transform time: 7.85 s
(EngineCore_DP0 pid=22) INFO 02-23 03:12:54 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 2048) from the cache, took 1.273 s
(EngineCore_DP0 pid=22) INFO 02-23 03:12:54 [monitor.py:34] torch.compile takes 9.12 s in total

[Billy99]

Looks good, only minor comments.
One part I was sure about is the GPU detection. I feel like there is a limited (small) number of GPUs that we detect, or am I missing something?

[maryamtahhan]

Looks good, only minor comments.
One part I was sure about is the GPU detection. I feel like there is a limited (small) number of GPUs that we detect, or am I missing something?

ATM it's using CUDA or ROCM - so should detect all NVIDIA or AMD GPUs?

this is something I plan on changing and doing through kube moving forward