Discussion: Cortex.cpp Model Loading and Inference Errors

[freelerobot]

At the moment we fail silently and users have to send us logs. "model failed to load"

Can we get a handle on all the potential reasons why their model failed to load, and discuss how to handle each issue?

Goal:

1. Graceful failures
2. Predefined errors
3. Though there are endless errors, lets adopt the Pareto Rule, as 80% of our bugs are due to 20% common model loading challenges

Examples

1. Model won't fit in RAM/VRAM
2. Another model is running... other edge cases & race conditions
3. Wrong model format (i.e. unsupported runtime)
4. Version conflicts (in trt-llm engine scneario)
5. Missing model.yaml, template, key input/configs
6. Corrupted or missing model binaries
7. Incompat hardware. See

Questions:

1. What are the other common issues?
2. We support various engines, but should we standardize failure modes? This allows us to offer better dx/ux down the road.
3. What are the various ways that llamacpp, trtllm, directml currently handle errors? Do they have a predefined, neat list we can adopt?

Related issues:

• feat: Add Exception Codes for OpenAI compatibility #130
• epic: Better Error Handing  jan#2556
• feat: Graceful failure for model loading (not Something's Amiss Error) jan#3517

[dan-menlo]

@0xSage I recommend we expand this to

• Error handling for Model Loading
• Error handling for Model Running

[freelerobot]

Example:

Model Loading

Error Code	Error Message	Failover (if any)
InsufficientMemory	"The model is too big for your (V)RAM"	-

Model Running

Error Code	Error Message	Failover (if any)
ContextExceeded	"Your input exceeded the model context window of tokens"	-

[dan-menlo]

@0xSage @vansangpfiev I am renaming this discussion to "Model Loading and Inference Errors"

[dan-menlo]

This Bug report could be more informative, with better logs from cortex.cpp:
janhq/jan#3552